Journal of Clinical Epidemiology
Volume 58, Issue 1 , Pages 1-12 , January 2005

A systematic review finds that diagnostic reviews fail to incorporate quality despite available tools

  • Penny Whiting

      Affiliations

    • Centre for Reviews and Dissemination, University of York, United Kingdom
    • MRC Health Services Research Collaboration, University of Bristol, United Kingdom
    • Corresponding Author InformationCorresponding author. Tel.: +44 0117 928 7204; fax: +44 928 7236.
    • ,
  • Anne W.S. Rutjes

      Affiliations

    • Department of Clinical Epidemiology & Biostatistics, Academic Medical Center, University of Amsterdam, The Netherlands
    • ,
  • Jacqueline Dinnes

      Affiliations

    • Wessex Institute for Health Research and Development, University of Southampton, United Kingdom
    • ,
  • Johannes B. Reitsma

      Affiliations

    • Department of Clinical Epidemiology & Biostatistics, Academic Medical Center, University of Amsterdam, The Netherlands
    • ,
  • Patrick M.M. Bossuyt

      Affiliations

    • Department of Clinical Epidemiology & Biostatistics, Academic Medical Center, University of Amsterdam, The Netherlands
    • ,
  • Jos Kleijnen

      Affiliations

    • Centre for Reviews and Dissemination, University of York, United Kingdom

References 

  1. Sackett DL, Haynes RB, Guyatt GH, Tugwell T. The selection of diagnostic tests. In: Clinical epidemiology: a basic science for clinical medicine. 2nd ed.. London: Little, Brown; 1991;p. 51–68
  2. Jaeschke R, Guyatt G, Sackett DL, Evidence-Based Medicine Working Group . Users' guides to the medical literature. III. How to use an article about a diagnostic test. A. Are the results of the study valid?. JAMA. 1994;271:389–391
  3. Jaeschke R, Guyatt G, Sackett DL, Evidence-Based Medicine Working Group . Users' guides to the medical literature. III. How to use an article about a diagnostic test. B. What are the results and will they help me in caring for my patients?. JAMA. 1994;271:703–707
  4. Irwig L, Tosteson ANA, Gatsonis C, Lau J, Colditz G, Chalmers TC, et al. Guidelines for meta-analyses evaluating diagnostic tests. Ann Intern Med. 1994;120:667–676
  5. Lijmer JG, Mol BW, Heisterkamp S, Bonsel GJ, Prins MH, van der Meulen JH, et al. Empirical evidence of design-related bias in studies of diagnostic tests. JAMA. 1999;282:1061–1066[Erratum in: JAMA 2000;283:1963.] JAMA
  6. Whiting P, Rutjes AWS, Reitsma JB, Glas A, Bossuyt PM, Kleijnen J. Sources of variation and bias in studies of diagnostic accuracy: a systematic review. Ann Intern Med. 2004;140:189–202
  7. Whiting P, Rutjes AWS, Dinnes J, Reitsma JB, Bossuyt PMM, Kleijnen J. Development and validation of methods for assessing the quality of diagnostic accuracy studies. Health Technol Assess. 2004;8(25):1–250
  8. Flynn K, Adams E, Anderson D. Positron emission tomography: systematic review. Report No. MTA94-001-02. Boston, MA: Technology Assessment Unit, Management Decision & Research Center, U.S. Department of Veterans Affairs (VATAP); 1996;
  9. Reid MC, Lachs MS, Feinstein AR. Use of methodological standards in diagnostic test research: getting better but still not good. JAMA. 1995;274:645–651
  10. Sheps SB, Schechter MT. The assessment of diagnostic tests: a survey of current medical research. JAMA. 1984;252:2418–2422
  11. Cooper LS, Chalmers TC, McCally M, Berrier J, Sacks HS. The poor quality of early evaluations of magnetic resonance imaging. JAMA. 1988;259:3277–3280
  12. Beam CA, Sostman HD, Zheng JY. Status of clinical MR evaluations 1985–1988: baseline and design for further assessments. Radiology. 1991;180:265–270
  13. Kent DL, Larson EB. Disease, level of impact, and quality of research methods: three dimensions of clinical efficacy assessment applied to magnetic resonance imaging. Invest Radiol. 1992;27:245–254
  14. Chien PFW, Khan KS, Ogston S, Owen P. The diagnostic accuracy of cervico-vaginal fetal fibronectin in predicting preterm delivery: an overview. Br J Obstet Gynaecol. 1997;104:436–444
  15. de Vries SO, Hunink MG, Polak JF. Summary receiver operating characteristic curves as a technique for meta-analysis of the diagnostic performance of duplex ultrasonography in peripheral arterial disease. Acad Radiol. 1996;3:361–369
  16. Fahey MT, Irwig L, Macaskill P. Meta-analysis of Pap test accuracy. Am J Epidemiol. 1995;141:680–689
  17. Heffner JE, Brown LK, Barbieri C, Deleo JM. Pleural fluid chemical analysis in parapneumonic effusions: a meta-analysis. Am J Respir Crit Care Med. 1995;151:1700–1708[Erratum in: Am J Respir Crit Care Med 1995;152:823.]
  18. Heffner JE, Feinstein D, Barbieri C. Methodologic standards for diagnostic test research in pulmonary medicine. Chest. 1998;114:877–885
  19. Hobbs FDR, Delaney BC, Fitzmaurice DA, Wilson S, Hyde CJ, Thorpe GH, et al. A review of near patient testing in primary care. Health Technol Assess. 1997;1(5):1–231
  20. Lensing AW, Hirsh J. 125I-fibrinogen leg scanning: reassessment of its role for the diagnosis of venous thrombosis in post-operative patients. Thromb Haemost. 1993;69:2–7
  21. Radack K, Park S. Is there a valid association between skin tags and colonic polyps: insights from a quantitative and methodologic analysis of the literature. J Gen Intern Med. 1993;8:413–421
  22. Devous MD, Thisted RA, Morgan GF, Leroy RF, Rowe CC. SPECT brain imaging in epilepsy: a meta-analysis. J Nucl Med. 1998;39:285–293
  23. Mol BW, Dijkman B, Wertheim P, Lijmer J, van der Veen F, Bossuyt PM. The accuracy of serum chlamydial antibodies in the diagnosis of tubal pathology: a meta-analysis. Fertil Steril. 1997;67:1031–1037
  24. Owens DK, Holodniy M, Garber AM, Scott J, Sonnad S, Moses L, et al. Polymerase chain reaction for the diagnosis of HIV infection in adults: a meta-analysis with recommendations for clinical practice and study design. Ann Intern Med. 1996;124:803–815
  25. Rao JK, Weinberger M, Oddone EZ, Allen NB, Landsman P, Feussner JR. The role of antineutrophil cytoplasmic antibody (c-ANCA) testing in the diagnosis of Wegener granulomatosis: a literature review and meta-analysis. Ann Intern Med. 1995;123:925–932
  26. Reed WW, Byrd GS, Gates RH, Howard RS, Weaver MJ. Sputum Gram's stain in community-acquired pneumococcal pneumonia: a meta-analysis. West J Med. 1996;165:197–204
  27. Swart P, Mol BW, van der Veen F, van Beurden M, Redekop WK, Bossuyt PM. The accuracy of hysterosalpingography in the diagnosis of tubal pathology: a meta-analysis. Fertil Steril. 1995;64:486–491
  28. Wells PS, Lensing AW, Davidson BL, Prins MH, Hirsh J. Accuracy of ultrasound for the diagnosis of deep venous thrombosis in asymptomatic patients after orthopedic surgery: a meta-analysis. Ann Intern Med. 1995;122:47–53
  29. Attia J, Margetts P, Guyatt G. Diagnosis of thyroid disease in hospitalized patients: a systematic review. Arch Intern Med. 1999;159:658–665
  30. Badgett RG, Mulrow CD, Otto PM, Ramirez G. How well can the chest radiograph diagnose left ventricular dysfunction?. J Gen Intern Med. 1996;11:625–634
  31. Becker D, Philbrick J, Bachhuber T, Humphries J. D-dimer testing and acute venous thromboembolism. Arch Intern Med. 1996;156:939–946
  32. Bradley KA, Boyd-Wickizer J, Powell SH, Burman ML. Alcohol screening questionnaires in women: a critical review. JAMA. 1998;280:166–171
  33. Buntinx F, Wauters H. The diagnostic value of macroscopic haematuria in diagnosing urological cancers: a meta-analysis. Fam Pract. 1997;14:63–68
  34. Conde-Agudelo A, Kafury-Goeta AC. Triple-marker test as screening for Down syndrome: a meta-analysis. Obstet Gynecol Surv. 1998;53:369–376
  35. Da Silva O, Ohlsson A, Kenyon C. Accuracy of leukocyte indices and C-reactive protein for diagnosis of neonatal sepsis: a critical review. Pediatr Infect Dis J. 1995;14:362–366
  36. De Bernardinis M, Violi V, Roncoroni L, Boselli AS, Giunta A, Peracchia A. Discriminant power and information content of Ranson's prognostic signs in acute pancreatitis: a meta-analytic study. Crit Care Med. 1999;27:2272–2283
  37. Hallan S, Asberg A. The accuracy of C-reactive protein in diagnosing acute appendicitis. Scand J Clin Lab Invest. 1997;57:373–380
  38. Heffner JE, Brown LK, Barbieri CA. Diagnostic value of tests that discriminate between exudative and transudative pleural effusions. Chest. 1997;111:970–980
  39. Kearon C, Julian JA, Newman TE, Ginsberg JS. Noninvasive diagnosis of deep venous thrombosis. Ann Intern Med. 1998;128:663–677
  40. Koelemay MJ, Denhartog D, Prins MH, Kromhout JG, Legemate DA, Jacobs MJ. Diagnosis of arterial disease of the lower extremities with duplex ultrasonography. Br J Surg. 1996;83:404–409
  41. Koumans EH, Johnson RE, Knapp JS, St Louis ME. Laboratory testing for Neisseria gonorrhoeae by recently introduced nonculture tests: a performance review with clinical and public health considerations. Clin Infect Dis. 1998;27:1171–1180
  42. Lacasse Y, Wong E, Guyatt GH, Cook DJ. Transthoracic needle aspiration biopsy for the diagnosis of localised pulmonary lesions: a meta-analysis. Thorax. 1999;54:884–893
  43. Littenberg B, Siegel A, Tosteson ANA, Mead T. Clinical efficacy of SPECT bone imaging for low back pain. J Nucl Med. 1995;36:1707–1713
  44. Metlay JP, Kapoor WN, Fine MJ. Does this patient have community-acquired pneumonia? Diagnosing pneumonia by history and physical examination. JAMA. 1997;278:1440–1445
  45. Mol BW, Bayram N, Lijmer JG, Wiegerinck MA, Bongers MY, van der Veen F, et al. The performance of CA-125 measurement in the detection of endometriosis: a meta-analysis. Fertil Steril. 1998;70:1101–1108
  46. Mol BW, Lijmer JG, Ankum WM, van der Veen F, Bossuyt PM. The accuracy of single serum progesterone measurement in the diagnosis of ectopic pregnancy: a meta-analysis. Hum Reprod. 1998;13:3220–3227
  47. Nuovo J, Melnikow J, Hutchison B, Paliescheskey M. Is cervicography a useful diagnostic test? a systematic overview of the literature. J Am Board Fam Pract. 1997;10:390–397
  48. Pollitt RJ, Green A, McCabe CJ, Booth A, Cooper NJ, Leonard JV, et al. Neonatal screening for inborn errors of metabolism: cost, yield and outcome. Health Technol Assess. 1997;1(7):1–203
  49. Rappeport ED, Mehta S, Wieslander SB, Schwarz Lausten G, Thomsen HS. MR imaging before arthroscopy in knee joint disorders?. Acta Radiol. 1996;37:602–609
  50. van den Hoogen HM, Koes BW, van Eijk JT, Bouter LM. On the accuracy of history, physical examination, and erythrocyte sedimentation rate in diagnosing low back pain in general practice. Spine. 1995;20:318–327
  51. van Tulder MW, Assendelft WJ, Koes BW, Bouter LM. Spinal radiographic findings and nonspecific low back pain: a systematic review of observational studies. Spine. 1997;22:427–434
  52. Panzer RJ, Kido DK, Hindmarsh T. A methodologic assessment of studies comparing magnetic resonance imaging and computed tomography of the brain. Acta Radiol Suppl. 1986;369:269–274
  53. Hrung J, Sonad S, Schwartz J, Langlotz C. Accuracy of MR imagining in the work-up of suspicious breast lesions: a diagnostic meta-analysis. Acad Radiol. 1999;6:387–397
  54. Rothwell PM, Pendlebury ST, Wardlaw J, Warlow CP. Critical appraisal of the design and reporting of studies of imaging and measurement of carotid stenosis. Stroke. 2000;31:1444–1450
  55. van der Wurff P, Hagmeijer RHM, Meyne W. Clinical tests of the sacroiliac joint: a systematic methodological review. Part 1: Reliability. Man Ther. 2000;5:30–36
  56. Windeler J, Richter K, Kobberling J. Description and evaluation of diagnostic-tests in German-language medical journals. Schweiz Med Wochenschr. 1988;118:1437–1441
  57. Mullins M, Becker D, Hagspiel K, Philbrick J. The role of spiral volumetric computed tomography in the diagnosis of pulmonary embolism. Arch Intern Med. 2000;160:293–298
  58. Fiellin D, Reid M, O'Connor P. Screening for alcohol problems in primary care. Arch Intern Med. 2000;160:1977–1989
  59. Berry E, Kelly S, Westwood ME, Davies LM, Gough MJ, Bamford JM, et al. The cost-effectiveness of magnetic resonance angiography for carotid artery stenosis and peripheral vascular disease: a systematic review. Health Technol Assess. 2002;6(7):1–165
  60. Becker D, Philbrick J, Abbitt P. Real-time ultrasonography for the diagnosis of lower extremity deep venous thrombosis: the wave of the future?. Arch Intern Med. 1989;149:1731–1734
  61. Guyatt G, Oxman A, Ali M, Willan A, McIlroy W, Patterson C. Laboratory diagnosis of iron-deficiency anemia: an overview. J Gen Intern Med. 1992;7:145–153[Erratum in: J Gen Intern Med 1992;7:423.]
  62. Hoffman RM, Kent DL, Deyo RA. Diagnostic accuracy and clinical utility of thermography for lumbar radiculopathy: a meta-analysis. Spine. 1991;16:623–628
  63. Holleman D, Simel D. Does the clinical examination predict airflow limitations?. JAMA. 1995;273:313–319
  64. Kent DL, Haynor DR, Larson EB, Deyo RA. Diagnosis of lumbar spinal stenosis in adults: a metaanalysis of the accuracy of CT, MR, and myelography. AJR Am J Roentgenol. 1992;158:1135–1144
  65. Philbrick J, Horowitz R, Feinstein A. Methodologic problems of exercise testing for coronary artery disease: groups, analysis and bias. Am J Cardiol. 1980;46:807–812
  66. Cochrane Methods Working Group on Systematic Review of Screening and Diagnostic Tests: recommended methods [Internet]. Updated June 6, 1996. Available at: http://www.cochrane.org/docs/sadtdoc1. htm.
  67. Liddle J, Williamson M, Irwig L. Method for evaluating research and guideline evidence. Report No. 0943126444. Sydney, Australia: NSW Health Department; 1996;
  68. Arrive L, Renard R, Carrat F, Belkacem A, Dahan H, Le Hir P, et al. A scale of methodological quality for clinical studies of radiologic examinations. Radiology. 2000;217:69–74
  69. Khan KS, Dinnes J, Kleijnen J. Systematic reviews to evaluate diagnostic tests. Eur J Obstet Gynecol Reprod Biol. 2001;95:6–11
  70. Mulrow CD, Linn WD, Gaul MK, Pugh JA. Assessing quality of a diagnostic test evaluation. J Gen Intern Med. 1989;4:288–295
  71. Black WC. How to evaluate the radiology literature. AJR Am J Roentgenol. 1990;154:17–22
  72. Jensen K, Abel U. Methodik diagnostischer Validierungsstudien: Fehler in der Studienplanung und Auswertung. [Methodology of diagnostic validation studies: errors in planning and analysis. In German.] Med Klin. 1999;94:522–529
  73. Greiner M, Gardner IA. Epidemiologic issues in the validation of veterinary diagnostic tests. Prev Vet Med. 2000;45:3–22
  74. Kobberling J, Trampisch HJ, Windeler J. Memorandum for the evaluation of diagnostic measures. J Clin Chem Clin Biochem. 1990;28:873–879
  75. Greenhalgh T. How to read a paper: papers that report diagnostic or screening tests. BMJ. 1997;315:540–543[Erratum in: BMJ 1997;315:942; BMJ1998;316:225.]
  76. Heffner JE. Evaluating diagnostic tests in the pleural space: differentiating transudates from exudates as a model. Clin Chest Med. 1998;19:277–293
  77. How to read clinical journals: II. To learn about a diagnostic test. Can Med Assoc J. 1981;124:703–710
  78. McGee S, Abernethy WB, Simel DL. The rational clinical examination. Is this patient hypovolemic?. JAMA. 1999;281:1022–1029
  79. Riegelman RK, Hirsch RP. Studying a study and testing a test: how to read the health science literature. 3rd ed.. Boston: Little Brown; 1996;
  80. Sackett DL. Evidence-based medicine: how to practice and teach EBM. 2nd ed.. Edinburgh; New York: Churchill Livingstone; 2000;
  81. Deeks JJ. Using evaluations of diagnostic tests: understanding their limitations and making the most of available evidence. Ann Oncol. 1999;10:761–768
  82. Dunn G, Everitt B. Clinical biostatistics: an introduction to evidence-based medicine. London: Edward Arnold; 1995;
  83. Mant D. Testing a test: three critical steps. In:  Jones R,  Kinmouth A editor. Critical reading for primary care. Oxford: Oxford University Press; 1995;p. 183–202
  84. Kraemer HC. Evaluating medical tests: objective and quantitative guidelines. Newbury Park: Sage Publications; 1992;
  85. Freedman LS. Evaluating and comparing imaging techniques: a review and classification of study designs. Br J Radiol. 1987;60:1071–1081
  86. Gifford DR, Cummings JL. Evaluating dementia screening tests: methodologic standards to rate their performance. Neurology. 1999;52:224–227
  87. Thornbury JR, Kido DK, Mushlin AI, Phelps CE, Mooney C, Fryback DG. Increasing the scientific quality of clinical efficacy studies of magnetic resonance imaging. Invest Radiol. 1991;26:829–835
  88. Deyo RA, Haselkorn J, Hoffman RM, Kent D. Designing studies of diagnostic tests for low back pain or radiculopathy. Spine. 1994;19(18 Suppl):S2057–S2065
  89. Sox H, Stern S, Owens D, Abrahms H. Assessment of Diagnostic technology in health care: rationale, methods, problems, and directions. Monograph of the Council On Health Care Technology, Institute of Medicine. Washington, DC: National Academy Press; 1989;
  90. Bruns DE, Huth EJ, Magid E, Young DS. Toward a checklist for reporting of studies of diagnostic accuracy of medical tests. Clin Chem. 2000;46:893–895
  91. Lang TA, Secic M. How to report statistics in medicine: annotated guidelines for authors, editors, and reviewers. Philadelphia: American College of Physicians; 1997;
  92. Haynes R, Sackett DL. Purpose and procedure (abbreviated). Evid Based Med. 1995;1:2
  93. Mower WR. Evaluating bias and variability in diagnostic test reports. Ann Emerg Med. 1999;33:85–91
  94. Deeks J. Systematic reviews of evaluations of diagnostic and screening tests. In:  Egger M,  Davey Smith G,  Altman D editor. Systematic reviews in health care: meta-analysis in context. 2nd ed.. London: BMJ Publishing Group; 2001;p. 248–282
  95. Deeks JJ. Systematic reviews in health care: Systematic reviews of evaluations of diagnostic and screening tests. BMJ. 2001;323:157–162
  96. Badgett RG, Lucey CR, Mulrow CD. Can the clinical examination diagnose left-sided heart failure in adults?. JAMA. 1997;277:1712–1719
  97. Bastian LA, Piscitelli JT. Is this patient pregnant? Can you reliably rule in or rule out early pregnancy by clinical examination?. JAMA. 1997;278:586–591
  98. Selker HP, Zalenski RJ, Antman EM, Aufderheide TP, Bernard SA, Bonow RO, et al. An evaluation of technologies for identifying acute cardiac ischemia in the emergency department: a report from a National Heart Attack Alert Program working group. Emerg Med. 1997;29:13–87[Erratum in: Ann Emerg Med 1997;29:310.]
  99. Anand SS, Wells PS, Hunt D, Brill-Edwards P, Cook D, Ginsberg JS. Does this patient have deep vein thrombosis?. JAMA. 1998;279:1094–1099
  100. Tugwell P, Dennis DT, Weinstein A, Wells G, Shea B, Nichol G, et al. Laboratory evaluation in the diagnosis of Lyme disease: clinical guideline, part 2. Ann Intern Med. 1997;127:1109–1123
  101. Bachmann MO, Nelson SJ. Impact of diabetic retinopathy screening on a British district population: case detection and blindness prevention in an evidence-based model. J Epidemiol Community Health. 1998;52:45–52
  102. Spencer-Green G, Alter D, Welch HG. Test performance in systemic sclerosis: anti-centromere and Anti-Scl-70 antibodies. Am J Med. 1997;103:242–248
  103. Whited JD, Grichnik JM. Does this patient have a mole or a melanoma?. JAMA. 1998;279:696–701
  104. Bell R, Petticrew M, Luengo S, Sheldon TA. Screening for ovarian cancer: a systematic review. Health Technol Assess. 1998;2:1–84
  105. Bonis PA, Ioannidis JP, Cappelleri JC, Kaplan MM, Lau J. Correlation of biochemical response to interferon alfa with histological improvement in hepatitis C: a meta-analysis of diagnostic test characteristics. Hepatology. 1997;26:1035–1044
  106. Mayer J. Systematic review of the diagnostic accuracy of dermatoscopy in detecting malignant melanoma. Med J Aust. 1997;167:206–210
  107. Huicho L, Campos M, Rivera J, Guerrant RL. Fecal screening tests in the approach to acute infectious diarrhea: a scientific overview. Pediatr Infect Dis J. 1996;15:486–494
  108. Pearl WS, Todd KH. Ultrasonography for the initial evaluation of blunt abdominal trauma: a review of prospective trials. Ann Emerg Med. 1996;27:353–361
  109. Bastian LA, Nanda K, Hasselblad V, Simel DL. Diagnostic efficiency of home pregnancy test kits: a meta-analysis. Arch Fam Med. 1998;7:465–469
  110. Owens DK, Holodniy M, McDonald TW, Scott J, Sonnad S. A meta-analytic evaluation of the polymerase chain reaction for the diagnosis of HIV infection in infants. JAMA. 1996;275:1342–1348
  111. Liedberg J, Panmekiate S, Petersson A, Rohlin M. Evidence-based evaluation of three imaging methods for the temporomandibular disc. Dentomaxillofac Radiol. 1996;25:234–241
  112. Loy CT, Irwig LM, Katelaris PH, Talley NJ. Do commercial serological kits for Helicobacter pylori infection differ in accuracy? A meta-analysis. Am J Gastroenterol. 1996;91:1138–1144
  113. Whiting P, Rutjes A, Reitsma J, Bossuyt P, Kleijnen J. The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med Res Methodol. 2003;3:25;Available at: http://www.biomedcentral.com/1471-2288/3/25
  114. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, et al. Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD Initiative. Ann Intern Med. 2003;138:40–44
  115. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, et al. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Clin Chem. 2003;49:7–18
  116. de Vet HC, van der Weijden T, Muris J, Heyrman J, Buntinx F, Knottnerus JA. Systematic reviews of diagnostic research: considerations about assessment and incorporation of methodological quality. Eur J Epidemiol. 2001;17:301–306

PII: S0895-4356(04)00165-9

doi: 10.1016/j.jclinepi.2004.04.008

Journal of Clinical Epidemiology
Volume 58, Issue 1 , Pages 1-12 , January 2005

Article Tools

* Image Usage & Resolution