Heterogeneity: Helpful or Harmful?

Some studies may conclude that A is better than B. Some others may claim that B is better than A. Even other studies may report that there is no difference between A and B. Even among studies that report A is better than B, some may say A is much better than B. When such a situation happens, it is usually caused by heterogeneity. But what is heterogeneity? Is it a beauty or bullshit? How can one manage heterogeneity between studies?

Heterogeneity is a variation between studies. Under ‘ideal’ conditions, two ‘identical’ studies are expected to reach at same results. But identical studies are rare and unnecessary. Frequently, studies addressing the same construct yield different findings as they may differ in several ways. The differences between studies can occur at any point in the life cycle of a study i.e. from its inception to completion. The following are the 10 common sources of heterogeneity.

  1. Purpose: If the purposes of two studies differ, their results may differ.
  2. Population: If the characteristics of study population differ, results may differ.
  3. Sample size: Differences in sample size may result in different results.
  4.  Instrument: Differences in research instruments may affect the results.
  5. Procedure: Data collection/measurement Procedures may affect data outputs.
  6. Variable: Differences in the definition of the study variable could affect the findings.
  7. Setting: Differences in the study setting (time, place and context) affects the findings.
  8. Observer: Differences in the person measuring, reading and/or recording the data may contribute to heterogeneity.
  9. Management: Management of data (i.e. analysis and reporting) can cause differences
  10. Perceiver: Differences in comprehension among readers of studies could also result in heterogeneity.

Studies are conducted to add some ‘new’ value to the existing body of knowledge. In doing so, they may use a different element. Thus, heterogeneity is an inherent character among studies. Different studies can explore different perspectives of a phenomenon or situation. When considered altogether, heterogeneous studies can describe a bigger picture of a phenomenon from various perspectives.  For those who are interested in exploration of the different perspectives of a phenomenon, heterogeneity is a beauty, a state of balance among different perspectives.

However, for those who want to sum up everything in to ‘averages,’ heterogeneity is a noise, an irregularity among studies that makes aggregation of results challenging. They have to account for it to reach at a reliable summary figure using meta-regression. They may see heterogeneity as a noise that has to be controlled! Here, it should be noted that, though an ‘average’ figure may be more generalizable, such ‘context-free’ evidence is less applicable. One can ‘average’ numbers but not context.