Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews

      Abstract

      Background and Objectives

      Studies of diagnostic accuracy most often report pairs of sensitivity and specificity. We demonstrate the advantage of using bivariate meta-regression models to analyze such data.

      Methods

      We discuss the methodology of both the summary Receiver Operating Characteristic (sROC) and the bivariate approach by reanalyzing the data of a published meta-analysis.

      Results

      The sROC approach is the standard method for meta-analyzing diagnostic studies reporting pairs of sensitivity and specificity. This method uses the diagnostic odds ratio as the main outcome measure, which removes the effect of a possible threshold but at the same time loses relevant clinical information about test performance. The bivariate approach preserves the two-dimensional nature of the original data. Pairs of sensitivity and specificity are jointly analyzed, incorporating any correlation that might exist between these two measures using a random effects approach. Explanatory variables can be added to the bivariate model and lead to separate effects on sensitivity and specificity, rather than a net effect on the odds ratio scale as in the sROC approach. The statistical properties of the bivariate model are sound and flexible.

      Conclusion

      The bivariate model can be seen as an improvement and extension of the traditional sROC approach.

      Keywords

      To read this article in full you will need to make a payment

      References

        • Sackett D.L.
        • Haynes R.B.
        The architecture of diagnostic research.
        in: Knottnerus J.A. The evidence base of clinical diagnosis. BMJ Publishing Group, London2002: 19-38
        • Guyatt G.H.
        • Tugwell P.X.
        • Feeny D.H.
        • Haynes R.B.
        • Drummond M.
        A framework for clinical evaluation of diagnostic technologies.
        CMAJ. 1986; 134: 587-594
        • Knottnerus J.A.
        • Muris J.W.
        Assessment of the accuracy of diagnostic tests: the cross-sectional study.
        J Clin Epidemiol. 2003; 56: 1118-1128
        • Griner P.F.
        • Mayewski R.J.
        • Mushlin A.I.
        • Greenland P.
        Selection and interpretation of diagnostic tests and procedures. Principles and applications.
        Ann Intern Med. 1981; 94: 557-592
        • Habbema J.D.F.
        • Eijkemans R.
        • Krijnen P.
        • Knottnerus J.A.
        Analysis of data on the accuracy of diagnostic tests.
        in: Knottnerus J.A. The evidence base of clinical diagnosis. BMJ Publishing Group, London2002: 117-144
        • Honest H.
        • Khan K.S.
        Reporting of measures of accuracy in systematic reviews of diagnostic literature.
        BMC Health Serv Res. 2002; 2: 4
        • Walter S.D.
        • Jadad A.R.
        Meta-analysis of screening data: a survey of the literature.
        Stat Med. 1999; 18: 3409-3424
        • Lijmer J.G.
        • Mol B.W.
        • Heisterkamp S.
        • Bonsel G.J.
        • Prins M.H.
        • van der Meulen J.H.
        • et al.
        Empirical evidence of design-related bias in studies of diagnostic tests.
        JAMA. 1999; 282: 1061-1066
        • Irwig L.
        • Macaskill P.
        • Glasziou P.
        • Fahey M.
        Meta-analytic methods for diagnostic test accuracy.
        J Clin Epidemiol. 1995; 48 ([discussion 131–2]): 119-130
        • Kardaun J.W.
        • Kardaun O.J.
        Comparative diagnostic performance of three radiological procedures for the detection of lumbar disk herniation.
        Methods Inf Med. 1990; 29: 12-22
        • Rutter C.M.
        • Gatsonis C.A.
        A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations.
        Stat Med. 2001; 20: 2865-2884
        • Moses L.E.
        • Shapiro D.
        • Littenberg B.
        Combining independent studies of a diagnostic test into a summary ROC curve: data-analytic approaches and some additional considerations.
        Stat Med. 1993; 12: 1293-1316
        • Deeks J.J.
        Systematic reviews in health care: systematic reviews of evaluations of diagnostic and screening tests.
        BMJ. 2001; 323: 157-162
        • Hasselblad V.
        • Hedges L.V.
        Meta-analysis of screening and diagnostic tests.
        Psychol Bull. 1995; 117: 167-178
        • Irwig L.
        • Tosteson A.N.
        • Gatsonis C.
        • Lau J.
        • Colditz G.
        • Chalmers T.C.
        • et al.
        Guidelines for meta-analyses evaluating diagnostic tests.
        Ann Intern Med. 1994; 120: 667-676
        • Littenberg B.
        • Moses L.E.
        Estimating diagnostic accuracy from multiple conflicting reports: a new meta-analytic method.
        Med Decis Making. 1993; 13: 313-321
        • Midgette A.S.
        • Stukel T.A.
        • Littenberg B.
        A meta-analytic method for summarizing diagnostic test performances: receiver-operating- characteristic-summary point estimates.
        Med Decis Making. 1993; 13: 253-257
        • Walter S.D.
        Properties of the summary receiver operating characteristic (SROC) curve for diagnostic test data.
        Stat Med. 2002; 21: 1237-1256
        • Glas A.S.
        • Lijmer J.G.
        • Prins M.H.
        • Bonsel G.J.
        • Bossuyt P.M.
        The diagnostic odds ratio: a single indicator of test performance.
        J Clin Epidemiol. 2003; 56: 1129-1135
        • Scheidler J.
        • Hricak H.
        • Yu K.K.
        • Subak L.
        • Segal M.R.
        Radiological evaluation of lymph node metastases in patients with cervical cancer. A meta-analysis.
        JAMA. 1997; 278: 1096-1101
        • Hilden J.
        The area under the ROC curve and its competitors.
        Med Decis Making. 1991; 11: 95-101
        • Begg C.B.
        • McNeil B.J.
        Assessment of radiologic tests: control of bias and other design considerations.
        Radiology. 1988; 167: 565-569
        • Sorribas A.
        • March J.
        • Trujillano J.
        A new parametric method based on S-distributions for computing receiver operating characteristic curves for continuous diagnostic tests.
        Stat Med. 2002; 21: 1213-1235
        • van Houwelingen H.C.
        • Arends L.R.
        • Stijnen T.
        Advanced methods in meta-analysis: multivariate approach and meta-regression.
        Stat Med. 2002; 21: 589-624
        • van Houwelingen H.C.
        • Zwinderman K.H.
        • Stijnen T.
        A bivariate approach to meta-analysis.
        Stat Med. 1993; 12: 2273-2284
        • Kotz S.
        • Balakrishnan N.
        • Johnson N.L.
        Bivariate and trivariate normal distributions.
        Continuous multivariate distributions. Wiley, New York2000 (251–348)
        • Bossuyt P.M.
        • Reitsma J.B.
        • Bruns D.E.
        • Gatsonis C.A.
        • Glasziou P.P.
        • Irwig L.M.
        • et al.
        The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration.
        Ann Intern Med. 2003; 138: W1-W12
        • Whiting P.
        • Rutjes A.W.
        • Reitsma J.B.
        • Glas A.S.
        • Bossuyt P.M.
        • Kleijnen J.
        Sources of variation and bias in studies of diagnostic accuracy: a systematic review.
        Ann Intern Med. 2004; 140: 189-202
        • Lijmer J.G.
        • Bossuyt P.M.
        • Heisterkamp S.H.
        Exploring sources of heterogeneity in systematic reviews of diagnostic tests.
        Stat Med. 2002; 21: 1525-1537
        • Normand S.L.
        Meta-analysis: formulating, evaluating, combining, and reporting.
        Stat Med. 1999; 18: 321-359
        • Song F.
        Exploring heterogeneity in meta-analysis: is the L'Abbe plot useful?.
        J Clin Epidemiol. 1999; 52: 725-730
        • Thompson S.G.
        • Sharp S.J.
        Explaining heterogeneity in meta-analysis: a comparison of methods.
        Stat Med. 1999; 18: 2693-2708
        • van Houwelingen H.
        • Senn S.
        Investigating underlying risk as a source of heterogeneity in meta-analysis.
        Stat Med. 1999; 18: 110-115
        • Bailey K.R.
        Inter-study differences: how should they influence the interpretation and analysis of results?.
        Stat Med. 1987; 6: 351-360
        • Deville W.L.
        • Buntinx F.
        • Bouter L.M.
        • Montori V.M.
        • De Vet H.C.
        • Van Der Windt D.A.
        • et al.
        Conducting systematic reviews of diagnostic studies: didactic guidelines.
        BMC Med Res Methodol. 2002; 2: 9
        • Shapiro D.E.
        Issues in combining independent estimates of the sensitivity and specificity of a diagnostic test.
        Acad Radiol. 1995; 2 ([discussion S65–9, S83]): S37-S47
        • Rutter C.M.
        • Gatsonis C.A.
        Regression methods for meta-analysis of diagnostic test data.
        Acad Radiol. 1995; 2 ([discussion S65–7, S70–1]): S48-S56
        • Macaskill P.
        Empirical Bayes estimates generated in a hierarchical summary ROC analysis agreed closely with those of a full Bayesian analysis.
        J Clin Epidemiol. 2004; 57: 925-932
        • Glas A.S.
        • Roos D.
        • Deutekom M.
        • Zwinderman A.H.
        • Bossuyt P.M.
        • Kurth K.H.
        Tumor markers in the diagnosis of primary bladder cancer. A systematic review.
        J Urol. 2003; 169: 1975-1982
        • Bipat S.
        • Glas A.S.
        • van der Velden J.
        • Zwinderman A.H.
        • Bossuyt P.M.
        • Stoker J.
        Computed tomography and magnetic resonance imaging in staging of uterine cervical carcinoma: a systematic review.
        Gynecol Oncol. 2003; 91: 59-66
        • Scholten R.J.
        • Opstelten W.
        • van der Plas C.G.
        • Bijl D.
        • Deville W.L.
        • Bouter L.M.
        Accuracy of physical diagnostic tests for assessing ruptures of the anterior cruciate ligament: a meta-analysis.
        J Fam Pract. 2003; 52: 689-694
        • Koelemay M.J.
        • Nederkoorn P.J.
        • Reitsma J.B.
        • Majoie C.B.
        Systematic review of computed tomographic angiography for assessment of carotid artery disease.
        Stroke. 2004; 35: 2306-2312
        • Song F.
        • Khan K.S.
        • Dinnes J.
        • Sutton A.J.
        Asymmetric funnel plots and publication bias in meta-analyses of diagnostic accuracy.
        Int J Epidemiol. 2002; 31: 88-95
        • Reid M.C.
        • Lachs M.S.
        • Feinstein A.R.
        Use of methodological standards in diagnostic test research. Getting better but still not good.
        JAMA. 1995; 274: 645-651
        • Bossuyt P.M.
        • Reitsma J.B.
        • Bruns D.E.
        • Gatsonis C.A.
        • Glasziou P.P.
        • Irwig L.M.
        • et al.
        Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative.
        Ann Intern Med. 2003; 138: 40-44
        • Sheps S.B.
        • Schechter M.T.
        The assessment of diagnostic tests. A survey of current medical research.
        JAMA. 1984; 252: 2418-2422