Heterogeneity: Helpful or Harmful?

Some studies may conclude that A is better than B. Some others may claim that B is better than A. Even other studies may report that there is no difference between A and B. Even among studies that report A is better than B, some may say A is much better than B. When such a situation happens, it is usually caused by heterogeneity. But what is heterogeneity? Is it a beauty or bullshit? How can one manage heterogeneity between studies?

Heterogeneity is a variation between studies. Under ‘ideal’ conditions, two ‘identical’ studies are expected to reach at same results. But identical studies are rare and unnecessary. Frequently, studies addressing the same construct yield different findings as they may differ in several ways. The differences between studies can occur at any point in the life cycle of a study i.e. from its inception to completion. The following are the 10 common sources of heterogeneity.

  1. Purpose: If the purposes of two studies differ, their results may differ.
  2. Population: If the characteristics of study population differ, results may differ.
  3. Sample size: Differences in sample size may result in different results.
  4.  Instrument: Differences in research instruments may affect the results.
  5. Procedure: Data collection/measurement Procedures may affect data outputs.
  6. Variable: Differences in the definition of the study variable could affect the findings.
  7. Setting: Differences in the study setting (time, place and context) affects the findings.
  8. Observer: Differences in the person measuring, reading and/or recording the data may contribute to heterogeneity.
  9. Management: Management of data (i.e. analysis and reporting) can cause differences
  10. Perceiver: Differences in comprehension among readers of studies could also result in heterogeneity.

Studies are conducted to add some ‘new’ value to the existing body of knowledge. In doing so, they may use a different element. Thus, heterogeneity is an inherent character among studies. Different studies can explore different perspectives of a phenomenon or situation. When considered altogether, heterogeneous studies can describe a bigger picture of a phenomenon from various perspectives.  For those who are interested in exploration of the different perspectives of a phenomenon, heterogeneity is a beauty, a state of balance among different perspectives.

However, for those who want to sum up everything in to ‘averages,’ heterogeneity is a noise, an irregularity among studies that makes aggregation of results challenging. They have to account for it to reach at a reliable summary figure using meta-regression. They may see heterogeneity as a noise that has to be controlled! Here, it should be noted that, though an ‘average’ figure may be more generalizable, such ‘context-free’ evidence is less applicable. One can ‘average’ numbers but not context.

5 responses to “Heterogeneity: Helpful or Harmful?

  1. All is true. We can not have homogeneous studies in a heterogeneous world. To come with findings of value to the heterogeneous world, heterogeneity of studies is the rule, not the exception.

  2. Well presented as usual. I am not sure how the statistical measure of heterogeneity in meta-analysis (chi-square) factors in the different sources of heterogeneity.

  3. Excellent points Ayalew and Senay. Ayalew, could you please comment on the point Senay raised? You are among the most appropriate ones to do so.

  4. Dear P2A,

    I don’t think I’m most appropriate but I can forward what I know. Coming to Senay’s point, when we do meta-analysis of heterogeneous studies, if we want to know what brought about the heterogeneity, in our meta-analysis dataset, we can include study level covariates (e.g., setting–hospital-based vs. community-based selection of cases, case ascertainment method, etc). Then through the use of meta-analysis regression (meta-regression) technique, we can investigate which of the study-level covariates can explain the heterogeneity in effect size.

    We use different statistics for our assessment. Modified I-squared statistic is used to quantify the percentage of variation accounted for by between-study heterogeneity. We use adjusted R-squared to explain the proportion of residual variation accounted for by the study-level covariates. There is also a kind of F-test used to test for the joint significance of all the covariates in meta-regression. The importance of each covariate in explaining the variability is assessed using a statistic such as t-test.

    In Stata, meta-regression is run using the command metareg.

    I hope this furnishes a piece of help.


  5. Many thanks Ayalew. I’m gaining a lot by the day from you guys !!!!!

Join this discussion

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s