Meta-analyses to evaluate analgesic interventions: A systematic qualitative review of their methodology

  • Alejandro R. Jadad
    Footnotes
    Affiliations
    Oxford Regional Pain Relief Unit, Churchill Hospital, University of Oxford, Oxford, United Kingdom

    Nuffield Department of Anaesthetics, University of Oxford, Oxford, United Kingdom
    Search for articles by this author
  • Henry J. McQuay
    Correspondence
    Please address correspondence to: Henry J. McQuay, Oxford Regional Pain Relief Unit, Churchill Hospital, Oxford, OX3 7LJ, United Kingdom.
    Affiliations
    Oxford Regional Pain Relief Unit, Churchill Hospital, University of Oxford, Oxford, United Kingdom

    Nuffield Department of Anaesthetics, University of Oxford, Oxford, United Kingdom
    Search for articles by this author
  • Author Footnotes
    † Current address: Department of Clinical Epidemiology and Biostatistics, McMaster University, 1200 Main Street West, Hamilton, Canada L8N 325.
      This paper is only available as a PDF. To read, Please Download here.

      Abstract

      A systematic search of the literature was performed to identify the maximum possible number of meta-analyses that evaluated analgesic interventions. Seventy-four reports were identified and retrieved and the scientific quality of 80 separate meta-analyses was assessed under blind conditions by 2 judges using Oxman and Guyatt's index. Most of the meta-analyses evaluated pharmacological interventions for chronic pain conditions and two-thirds were published since 1990. Ninety percent of the meta-analyses had methodological flaws that could limit their validity. The main deficiencies were lack of information on methods to retrieve and to assess the validity of primary studies and lack of data on the design of the primary studies. Meta-analyses of low quality produced significantly more positive conclusions.
      For several topics, different meta-analyses evaluating the same intervention produced conflicting results. The need to resolve these contradictions is highlighted.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic and Personal

      Subscribe:

      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Sacks HS
        • Berrier J
        • Reitman D
        • Ancona-Berk VA
        • Chalmers TC
        Meta-analyses of randomized controlled trials.
        N Engl J Med. 1987; 316: 450-455
        • Gerbarg ZB
        • Horwitz RI
        Resolving conflicting clinical trials: Guidelines for meta-analysis.
        J Clin Epidemiol. 1988; 41: 503-509
        • Kassirer JP
        Clinical trials and meta-analysis—what do they do for us?.
        N Engl J Med. 1992; 327: 273-274
        • Goldman L
        • Feinstein AR
        Anticoagulants and myocardial infarction—the problems of pooling, drowning, and floating.
        Ann Intern Med. 1979; 90: 92-94
        • Fleiss JL
        • Gross AJ
        Meta-analysis in epidemiology, with special reference to studies of the association between exposure to environmental tobacco smoke and lung cancer: A critique.
        J Clin Epidemiol. 1991; 44: 127-139
        • Thomson SG
        • Pocock SJ
        Can meta-analysis be trusted?.
        Lancet. 1991; 338: 1127-1130
        • Felson D
        Bias in meta-analysis.
        J Clin Epidemiol. 1992; 45: 885-892
        • Oxman AD
        • Guyatt GH
        Validation of an index of the quality of review articles.
        J Clin Epidemiol. 1991; 44: 1271-1278
        • Oxman AD
        • Guyatt GH
        • Singer J
        • Goldsmith CH
        • Hutchison BG
        • Milner RA
        • Streiner DL
        Agreement among reviewers of review articles.
        J Clin Epidemiol. 1991; 44: 91-98
        • Jadad AR
        • McQuay HJ
        A high-yield strategy to identify randomized controlled trials for systematic reviews.
        Online J Curr Clin Trials. 1993; (Doc No. 33)
        • Shrout PE
        • Fleiss JL
        Intraclass correlations: Uses in assessing rater reliability.
        Psychol Bull. 1979; 86: 420-428
        • Duckro PN
        • Cantwell-Simmonds E
        A review of studies evaluating bio-feedback and relaxation training in the management of pediatric headache.
        Headache. 1989; 29: 428-433
        • Amery WK
        Prophylactic and curative treatment of migraine with calcium antagonists.
        Drug Desi Delivery. 1989; 4: 197-203
        • Abenhaim L
        • Bergeron AM
        Twenty years of randomized clinical trials of manipulative therapy for back pain: A review.
        Clin Invest Med. 1992; 15: 527-535
        • Gabriel SE
        • Jaakkimainen L
        • Bombardier C
        The cost-effectiveness of misoprostol for nonsteroidal antiinflammatory drug-associated adverse gastrointestinal events.
        Arthr Rheum. 1993; 36: 447-459
        • Simkin P
        Non-pharmacological methods of pain relief during labour.
        in: Chalmers I Enkin M Keirse MJNC Effective Care in Pregnancy and Childbirth. Oxford University Press, Oxford1989: 893-911
        • Dickersin K
        Pharmacological control of pain during labour.
        in: Chalmers I Enkin M Keirse MJNC Effective Care in Pregnancy and Childbirth. Oxford University Press, Oxford1989: 913-950
        • Grant A
        • Sleep J
        Relief of perineal pain and discomfort after childbirth.
        in: Chalmers I Enkin M Keirse MJNC Effective Care in Pregnancy and Childbirth. Oxford University Press, Oxford1989: 1347-1358
        • Beecher HK
        The powerful placebo.
        JAMA. 1955; 159: 1602-1606
        • Hathaway D
        Effect of preoperative instruction on postoperative outcomes: A meta-analysis.
        Nurs Res. 1986; 35: 269-275
        • Penzien DB
        • Holroyd KA
        • Hursey KG
        • Holm JE
        • Wittchen H
        The behavioral treatment of migraine headache: A meta-analytic review of the literature.
        in: 1987: 1-67 (Based on a doctoral dissertation by Penzien DB)
        • Bamberg P
        • Caswell CM
        • Lam SK
        • Wong CK
        A meta-analysis comparing the efficacy of omeprazole with H2-receptor antagonists for acute treatment of duodenal ulcer in Asian patients.
        J Gastroenterol Hepatol. 1992; 7: 577-585
        • Capell HA
        • Porter DR
        • Madhok R
        • Hunter JA
        Second line (disease modifying) treatment in rheumatoid arthritis: Which drug for which patient?.
        Ann Rheum Dis. 1993; 52: 423-428
        • Ballantyne JC
        • Carr DB
        • Chalmers TC
        • Dear DBG
        • Angelillo IF
        • Mosteller F
        Postoperative patient-controlled analgesia: Meta-analyses of initial randomized control trials.
        J Clin Anesth. 1993; 5: 182-193
        • Radack K
        • Wyderski RJ
        Conservative management of intermittent claudication.
        Ann Intern Med. 1990; 113: 135-146
        • Holroyd KA
        • Penzien DB
        Pharmacological versus non-pharmacological prophylaxis of recurrent migraine headache: A meta-analytic review of clinical trials.
        Pain. 1990; 42: 1-13
        • Malone MD
        • Strube MJ
        Meta-analysis of non-medical treatments for chronic pain.
        Pain. 1988; 34: 231-244
        • Shekelle PG
        • Adams AH
        • Chassin MR
        • Hurwitz EL
        • Brook RH
        Spinal manipulation for low back pain.
        Ann Intern Med. 1992; 117: 590-598
        • Felson DT
        • Anderson JJ
        • Meenan RF
        The comparative efficacy and toxicity of second-line drugs in rheumatoid arthritis: Results of two metaanalyses.
        Arthr Rheum. 1990; 33: 1449-1461
        • Cutler R
        • Fishbain D
        • Rosomoff H
        • Abdel-Moty E
        • Khalil T
        • Steele-Rosomoff R
        Does non-surgical treatment of chronic pain return patients to work? A review and meta-analysis of the literature.
        in: Proc Am Pain Soc Meeting. 1992: 83
        • Chalmers TC
        • Ballantyne JC
        • Carr DB
        • Dear KBG
        • Angelillo IF
        • Mosteller F
        Comparative effects of analgesic therapies upon postoperative pulmonary function: meta-analyses.
        in: Abstracts—7th World Congress on Pain. 1993: 136
        • Gøtzsche P
        Meta-analysis of NSAIDs: Contribution of drugs, doses, trial designs, and meta-analytic techniques.
        Scand J Rheumatol. 1993; 22: 255-260
        • Gøtzsche PC
        Methodology and overt and hidden bias in reports of 196 double-blind trials of nonsteroidal antiinflammatory drugs in rheumatoid arthritis.
        Control Clin Trials. 1989; 10: 31-56
        • Turner JA
        • Ersek M
        • Herron L
        • Deyo R
        Surgery for lumbar spinal stenosis: Attempted meta-analysis of the literature.
        Spine. 1992; 17: 1-8
        • Hoffman RM
        • Kent DL
        • Deyo RA
        Diagnostic accuracy and clinical utility of thermography for lumbar radiculopathy: A meta-analysis.
        Spine. 1991; 16: 623-628
        • Broome ME
        • Lillis PP
        • Smith MC
        Pain interventions with children: A meta-analysis of research.
        Nurs Res. 1989; 38: 154-158
        • Patel M
        • Gutzwiller F
        • Paccaud F
        • Marazzi A
        A meta-analysis of acupuncture for chronic pain.
        Int J Epidemiol. 1989; 18: 900-906
        • Ter Riet G
        • Kleijnen J
        • Knipschild P
        Acupuncture and chronic pain: A criteria-based meta-analysis.
        J Clin Epidemiol. 1990; 43: 1191-1199
        • Chalmers TC
        • Berrier J
        • Hewitt P
        • Berlin J
        • Reitman D
        • Nagalingam R
        • et al.
        Meta-analysis of randomized controlled trials as a method of estimating rare complications of non-steroidal anti-inflammatory drug therapy.
        Aliment Pharmacol Therapeut. 1988; 2: 9-26
        • Gabriel SE
        • Jaakkimainen L
        • Bombardier C
        Risk of serious gastrointestinal complications related to use of nonsteroidal anti-inflammatory drugs.
        Ann Intern Med. 1991; 115: 787-796
        • Bollini P
        • Garcia-Rodriguez LA
        • Perez-Gutthann S
        • Walker AM
        The impact of research quality and study design on epidemiologic estimates of the effect of non-steroidal anti-inflammatory drugs on upper gastrointestinal tract disease.
        Arch Intern Med. 1992; 152: 1289-1295
        • Ottenbacher K
        • Difabio RP
        Efficacy of spinal manipulation/mobilization therapy: A meta-analysis.
        Spine. 1985; 10: 833-837
        • Anderson R
        • Meeker WC
        • Wirick BE
        • Mootz RD
        • Kirk DH
        • Adams A
        A meta-analysis of clinical trials of spinal manipulation.
        J Manipulative Physiol Therapeut. 1992; 15: 181-194
        • Felson DT
        • Anderson JJ
        • Meenan RF
        Use of short-term efficacy/toxicity tradeoffs to select second-line drugs in rheumatoid arthritis—a meta-analysis of published clinical trials.
        Arthr Rheum. 1992; 35: 1117-1125
        • Koes BW
        • Assendelft WJJ
        • Van der Heijden GJM
        • Bouter LM
        • Knipschild PG
        Spinal manipulation and mobilisation for back and neck pain: A blinded review.
        Br Med J. 1991; 303: 1298-1303
        • Gøtzsche PC
        • Pødenphant J
        • Olesen M
        • Halberg P
        Meta-analysis of second-line antirheumatic drugs: Sample bias and uncertain benefit.
        J Clin Epidemiol. 1992; 45: 587-594
        • Beckerman H
        • de-Bie RA
        • Bouter LM
        • De-Cuyper HJ
        • Oostendorp RAB
        The efficacy of laser therapy for musculoskeletal and skin disorders: A criteria-based meta-analysis of randomized clinical trials.
        Phys Ther. 1992; 72: 483-491
        • Gam AN
        • Thorsen H
        • Lønnberg F
        The effect of low-level laser therapy on musculoskeletal pain: A meta-analysis.
        Pain. 1993; 52: 63-66
        • Schmader KE
        • Studenski S
        Are current therapies useful for the prevention of postherpetic neuralgia.
        J Gen Intern Med. 1989; 4: 83-89
        • Lycka BAS
        Postherpetic neuralgia and systemic corticosteroid therapy: Efficacy and safety.
        Int J Dermatol. 1990; 29: 523-527
        • Naldi L
        • Zucchi A
        • Brevi A
        • Cavalieri D'Oro L
        • Cainelli T
        Corticosteroids and post-herpetic neuralgia.
        Lancet. 1990; 336: 947
        • Crooks RJ
        • Jones DA
        • Fiddian P
        Zoster-associated chronic pain: An overview of clinical trials with acyclovir.
        Scand J Infect. 1991; 78 (Suppl.): 62-68
        • Wilkie DJ
        • Savedra MC
        • Holzemer WL
        • Tesler MD
        • Paul SM
        Use of the McGill Pain Questionnaire to measure pain: A meta-analysis.
        Nurs Res. 1990; 39: 36-40
        • Holroyd KA
        • Penzien DB
        • Cordingley GE
        Propranolol in the management of recurrent migraine: A meta-analytic review.
        Headache. 1991; 31: 333-340
        • Holroyd KA
        • Penzien DB
        Client variables and the behavioral treatment of recurrent tension headache: A meta-analytic review.
        J Behav Med. 1986; 9: 515-535
        • Antczak-Bouckoms A
        • Tung F
        • Chalmers TC
        • Bouckoms A
        Quality assessment and meta-analysis of randomized control trials of ibuprofen.
        Pain. 1987; (Suppl.): S51
        • Haase W
        • Fischer M
        Statistische Metaanalyse von multizentrischen klinischen Studien mit Ibuprofen im Himblick auf die Kohortengröβe.
        Z Rheumatol. 1991; 50: 77-83
        • Vaiani G
        • Grossi E
        Meta-analysis of Italian clinical trials of nabumetone.
        Drugs. 1990; : 48-59
        • Schiassi M
        • Bianchini C
        • Restelli A
        • Zoni G
        Tolerability profile of the antiinflammatory compound imidazole salicylate: A meta-analysis of safety data in 1408 patients.
        Int J Tiss React. 1989; 11: 321-326
        • Suls J
        • Fletcher B
        The relative efficacy of avoidant and nonavoidant coping strategies: A meta-analysis.
        Health Psychol. 1985; 4: 249-288
        • Suls J
        • Wan CK
        Effects of sensory and procedural information on coping with stressful medical procedures and pain: A meta-analysis.
        J Consult Clin Psychol. 1989; 57: 372-379
        • Sand T
        Which factors affect reported headache incidences after lumbar myelography? A statistical analysis of publications in the literature.
        Neuroradiology. 1989; 31: 55-59
        • Sacks HS
        • Ancona-Berk VA
        • Berrier J
        • Nagalingam R
        • Chalmers TC
        Dipyridamole in the treatment of angina pectoris: A meta-analysis.
        Clin Pharmacol Therapeut. 1988; 43: 610-615
        • Robinson M
        • Mills RJ
        • Euler AR
        Ranitidine prevents duodenal ulcers associated with non-steroidal anti-inflammatory drug therapy.
        Aliment Pharmacol Therapeut. 1991; 5: 143-150
        • Mullen PD
        • Laville EA
        • Biddle AK
        • Lorig K
        Efficacy of psychoeducational interventions on pain, depression, and disability in people with arthritis: A meta-analysis.
        J Rheumatol. 1987; 14: 33-39
        • Gøtzsche PC
        Patients' preference in indomethacin trials: An overview.
        Lancet. 1989; 1: 88-91
        • Bosi-Ferraz M
        • Tugwell P
        • Goldsmith CH
        • Atra E
        Meta-analysis of sulfasalazine in ankylosing spondylitis.
        J Rheumatol. 1990; 17: 1482-1486
        • Flor H
        • Fydrich T
        • Turk DC
        Efficacy of multidisciplinary pain treatment centers: A meta-analytic review.
        Pain. 1992; 49: 221-230
        • Bosi-Ferraz M
        • Atra E
        Metaanálise: Nova opcao em pesquisa clínica.
        Rev Paul Med. 1989; 107: 5-9
        • Fernandez E
        • Turk DC
        The utility of cognitive coping strategies for altering pain perception: A meta-analysis.
        Pain. 1989; 38: 123-135
        • Devine EC
        • Cook TD
        Clinical and cost-saving effects of psychoeducational interventions with surgical patients: A meta-analysis.
        Res Nurs Health. 1986; 9: 89-105
        • Blanchard EB
        • Andrasik F
        • Ahles TA
        • Teders ST
        • O'Keefe D
        Migraine and tension headache: A meta-analytic review.
        Behav Ther. 1980; 11: 613-631
        • Onghena P
        • Van Houdenhove B
        Antidepressant-induced analgesia in chronic non-malignant pain: A meta-analysis of 39 placebo controlled studies.
        Pain. 1992; 49: 205-219
        • Hyman RB
        • Feldman HR
        • Harris RB
        • Levin RF
        • Malloy GB
        The effects of relaxation training on clinical symptoms: A meta-analysis.
        Nurs Res. 1989; 38: 216-220
        • Penzien DB
        • Rains JC
        • Holroyd KA
        A review of alternative behavioral treatments for headache.
        Mississippi Psychol. 1992; 17: 8-9
        • Holroyd KA
        • Athens OH
        • Penzien DB
        • Jackson MS
        • Rokicki LA
        • Cordingley GE
        Flunarizine vs propranolol: A meta-analysis of clinical trials.
        Headache. 1992; 32: 256
        • Penzien DB
        • Holroyd KA
        The behavioral treatment of migraine headache: A meta-analytic review of the literature.
        Dissertation Abstracts Int. 1987; 47 (Order No. DA8629941)
        • Gøtzsche PC
        Sensitivity of effect variables in rheumatoid arthritis: A meta-analysis of 130 placebo controlled NSAIDs trials.
        J Clin Epidemiol. 1990; 43: 1313-1318
        • Gøtzsche PC
        Meta-analysis of grip strength: Most common, but superfluous variable in comparative NSAID trials.
        Dan Med Bull. 1989; 36: 493-495
        • Pace NL
        Prevention of succinylcholine myalgias: A meta-analysis.
        Anesth Analg. 1990; 70: 477-483
        • Geis GS
        Overall safety of Arthrotec.
        Scand J Rheumatol. 1992; : 33-36
        • Blanchard EB
        • Andrasik F
        Psychological assessment and treatment of headache: Recent developments and emerging issues.
        J Consult Clin Psychol. 1982; 50: 859-879
        • Lataste X
        • Taylor P
        • Notter M
        DHE nasal spray in the acute management of migraine attacks.
        Cephalalgia. 1989; 9: 342-343
        • Cutler RB
        • Fishbain DA
        • Rosomoff HL
        • Abdel-Moty E
        • Khalil TM
        • Rosomoff RS
        Does non-surgical pain center treatment of chronic pain return patients to work?.
        Spine. 1994; 19: 643-652
        • Laska EM
        • Sunshine A
        • Mueller F
        • Elvers WB
        • Siegel C
        • Rubin A
        Caffeine as an analgesic adjuvant.
        Clin Pharmacol Ther. 1984; 251: 1711-1718
        • Amery WK
        Onset of action of various migraine prophylactics.
        Cephalalgia. 1988; 8: 11-13
        • Carr D
        • Eisenberg E
        • Chalmers TC
        Neurolytic celiac plexus block for cancer pain—a meta-analysis.
        in: Abstracts—7th World Congress on Pain. 1993: 338
        • Koes BW
        • Bouter LM
        • Beckerman H
        • Van der Heijden GJM
        • Knipschild PG
        Physiotherapy exercises and back pain: A blinded review.
        Br Med J. 1991; 302: 1572-1576
        • Devine EC
        Effects of psychoeducational care for adult surgical patients: A meta-analysis of 191 studies.
        Patient Educ Counsel. 1992; 19: 129-142
        • Johanson JF
        • Rimm A
        Optimal nonsurgical treatment of hemorrhoids: A comparative analysis of infrared coagulation, rubber band ligation and injection sclerotherapy.
        Am J Gastroenterol. 1992; 87: 1601-1606
        • Pope JE
        • Anderson JJ
        • Felson DT
        A meta-analysis of the effects of nonsteroidal anti-inflammatory drugs on blood pressure.
        Arch Intern Med. 1993; 153: 477-484
        • Capurso L
        • Koch M
        Prevention of NSAID-induced gastric lesions: H2 antagonists or misoprostol? A meta-analysis of controlled clinical trials.
        Clin Terapeut. 1991; 139: 179-189
        • Howell C
        • Chalmers I
        A review of prospectively controlled comparisons of epidural with non-epidural forms of pain relief during labour.
        Int J Obstet Anesth. 1991; 2: 1-17
        • Radack K
        • Deck C
        β-Adrenergic blocker therapy does not worsen intermittent claudication in subjects with peripheral arterial disease: A meta-analysis of randomized controlled trials.
        Arch Intern Med. 1992; 151: 1769-1776
        • Broome ME
        • Lillis PP
        A descriptive analysis of the pediatric pain management research.
        Appl Nurs Res. 1989; 2: 74-81
        • Dickersin K
        • Higgins K
        • Meinert CL
        Identification of meta-analyses: The need for standard terminology.
        Control Clin Trials. 1990; 11: 52-66
        • Felson DT
        • Anderson JJ
        Critique of meta-analysis of second-line anti-rheumatic drugs.
        J Clin Epidemiol. 1993; 46: 315-321
        • Gøtzsche PC
        • Pødenphant J
        • Olensen M
        • Halberg P
        Response to “Critique of meta-analysis of second-line antirheumatic drugs”.
        J Clin Epidemiol. 1993; 46: 319-321
        • Eysenck HJ
        An exercise of meta-silliness.
        Am Psychol. 1978; 33: 517
        • Wachter KW
        Disturbed by meta-analysis?.
        Science. 1988; 241: 1407-1408