Bias in meta-analysis

The meta-analysis of clinical trials of homeopathy and allopathy by Shang et al (The Lancet 2005; 366: 726–732) does not support the headline of ‘the end of homoeopathy’. The study itself, while attempting to eliminate false-positive bias in randomized placebo-controlled trials of homeopathy (RCT) introduces bias by failing to assess false-negative bias. False-negative biases are omnipresent in RCTs but we believe that they are more likely in trials of homoeopathy.

• For example, in a paediatric RCT on respiratory infections, homoeopathy was provided versus placebo in addition to standard antibiotic treatment and tonsillectomy.[ ¹ ]

  Homoeopathy had to prove benefit additional to conventional therapy, a heavy burden of proof. If homoeopathy was effective, control patients would need more antibiotics and surgery, and did so in this study. Such surplus of conventional therapies in control patients can easily mask homoeopathy effects in verum patients and create false-negative results.

• False negatives may be induced when the basic simile principle of homoeopathy is neglected and an identical single remedy given to all patients, making RCTs easier to perform. For example, an RCT on Rhus tox. with individualized simile matching produced a positive result,[ ² ] an RCT of the same medicine neglecting the similie gave a negative result.[ ³ ]

• Randomized trials have important limitations in complex interventions that require particular skills;[ ⁴ ] and homoeopathy, especially classical homoeopathy, is highly skill dependent. Finding the correct homoeopathic simillimum depends on in-depth anamnesis in an atmosphere of trust, which is disrupted by randomization. Skilled practitioners with positive treatment experience are, for ethical reasons, less likely to participate in RCTs.

Other false-negative factors are: drop-outs and non-compliers; contamination; informed consent; submissive answers; insensitive questionnaires, group assimilation, conditioning, cognitive interactions, etc.[ ⁵ ] Several of these may be present in a single study. Assessing trial quality according to randomization, blinding and size does not weed out trials with false-negative bias: ‘Orthodoxy always invokes the danger of Type One errors to ensure the occurrence of Type Two errors!’ (Eysenck, 1993). As Woods demonstrated, the logistics of large trials often demand simplified protocols that easily lead to false-negative results.[ ⁶ ] Conditions necessary for quality homeopathy treatment, especially classical homeopathy are less likely to be provided in well randomized, well blinded and large trials. Unfortunately, the authors refused our requests to identify the decisive 14 ‘larger trials of higher reported methodological quality.’ This makes it impossible to assess if these larger trials allowed for optimal treatment conditions or if simplifications put homeopathy at disadvantage.

Shang et al interpreted asymmetric funnel plots as publication bias but this requires further proof: the 1997 meta-analysis on homoeopathy had dismissed publication bias after an extensive inquiry with manufacturers, researchers and practitioners.[ ⁷ ] And concerning more pronounced between-trial heterogeneity in conventional medicine, its greater diversity of treatment methods also has to be taken in account.

In conclusion, this meta-analysis is far from confirmative and false-negative bias seems to have been the blind spot.

We declare that we have no conflict of interest.

• References

• 1 De Lange de Klerk E., Blommers J., Kuik D., Bezemer P., Feenstra L. Effect of homeopathic medicines on daily burden of symptoms in children with upper respiratory tract infections. Br Med J 1994; 309: 1392-1402.
• 2 Fisher P., Greenwood A., Huskisson E.C., Turner P., Belon P. Effect of homoeopathic treatment on fibrositis (primary fibromyalgia). Br Med J 1989; 299: 365-366.
• 3 Shipley M., Berry H., Broster G., Jenkins M., Clover A., Williams I. Controlled trial of homoeopathic treatment of osteoarthritis. Lancet 1983; 1: 97-98.
• 4 Kotaska A. Inappropriate use of randomised trials to evaluate complex phenomena: case study of vaginal breech delivery. Br Med J 2004; 329: 1036-1039.
• 5 Kiene H. Komplementäre Methodenlehre der klinischen Forschung. Berlin: Springer; 2001.
• 6 Woods K.L. Mega-trials and management of acute myocardial infarction. Lancet 1995; 346: 611-614.
• 7 Linde K. et al. Are the clinical effects of homeopathy placebo effects. Lancet 1997; 350: 834-843.