Abstract
A diagnosis in practice is a sequential process starting with a patient with a particular
set of signs and symptoms. To serve practice, diagnostic research should aim to quantify
the added value of a test to clinical information that is commonly available before
the test will be applied. Routine care databases commonly include all documented patient
information, and therefore seem to be suitable to quantify a tests' added value to
prior information. It is well known, however, that retrospective use of routine care
data in diagnostic research may cause various methodologic problems. But, given the
increased attention of electronic patient records including data from routine patient
care, we believe it is time to reconsider these problems. We discuss four problems
related to routine care databases. First, most databases do not label patients by
their symptoms or signs but by their final diagnosis. Second, in routine care the
diagnostic workup of a patient is by definition determined by previous diagnostic
(test) results. Therefore, routinely documented data are subject to so-called workup
bias. Third, in practice, the reference test is always interpreted with knowledge
of the preceding test information, such that in scientific studies using routine data
the diagnostic value of a test under evaluation is commonly overestimated. Fourth,
routinely documented databases are likely to suffer from missing data. Per problem
we discuss methods that are presently available and may (partly) overcome each problem.
All this could contribute to more frequent and appropriate use of routine care data
in diagnostic research. The discussed methods to overcome the above problems may well
be similarly useful to prospective diagnostic studies.
Keywords
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Journal of Clinical EpidemiologyAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- Problems of spectrum and bias in evaluating the efficacy of diagnostic tests.N Engl J Med. 1978; 299: 926-930
- Clinical epidemiology: the architecture of clinical research.WB Saunders Company, Philadelphia1985
- Clinical prediction rules. Applications and methodological standards.N Engl J Med. 1985; 313: 793-799
- Selection bias and the evaluation of diagnostic tests: a metadissent.J Chronic Dis. 1986; 39: 359-360
- Biases in the assessment of diagnostic tests.Stat Med. 1987; 6: 411-423
- The efficacy of diagnostic imaging.Med Decis Making. 1991; 11: 88-94
- The influence of referral patterns on the characteristics of diagnostic tests.J Clin Epidemiol. 1992; 45: 1143-1154
- Application of logistic regression to the analysis of diagnostic data: exact modeling of a probability tree of multiple binary variables.Med Decis Making. 1992; 12: 93-108
- A primer on the precision and accuracy of the clinical examination.JAMA. 1992; 267: 2638-2644
- Users' guides to the medical literature.JAMA. 1994; 271: 389-391
- Clinical epidemiology introduction to the discipline.Neth J Med. 1995; 47: 2-5
- Measuring the effects of imaging: an evaluative framework.Clin Radiol. 1995; 50: 513-518
- Problems in selecting the adequate patient population from existing data files for assessment studies of new diagnostic tests.J Clin Epidemiol. 1995; 48: 417-422
- An overview of cost-effective radiology.Eur Radiol. 1997; 7: 147-150
- Limitations of sensitivity, specificity, likelihood ratio, and Bayes' theorem in assessing diagnostic probabilities: a clinical example.Epidemiology. 1997; 8: 12-17
- Reduncancy of single diagnostic test evaluation.Epidemiology. 1999; 10: 276-281
- Use of methodological standards in diagnostic test research. Getting better but still not good.JAMA. 1995; 274: 645-651
- Empirical evidence of design-related bias in studies of diagnostic tests.JAMA. 1999; 282: 1061-1066
- Assessment of diagnostic tests when disease verification is subject to selection bias.Biometrics. 1983; 39: 206-215
- Patient-oriented performance measures of diagnostic tests.Med Decis Making. 1984; 4: 7-15
- Patient-oriented performance measures of diagnostic tests.Med Decis Making. 1984; 4: 17-31
- Factors affecting sensitivity and specificity of exercise electrocardiography. Multivariable analysis.Am J Med. 1984; 77: 64-71
- Clinical epidemiology; a basic science for clinical medicine.Little, Brown & Co, Boston1985
- Statistical methods in medical diagnosis.Crit Rev Med Inform. 1986; 1 ([review]): 1-22
- A model for assessing the sensitivity and specificity of tests subject to selection bias. Application to exercise radionuclide ventriculography for diagnosis of coronary artery disease.J Chronic Dis. 1986; 39: 343-355
- Carcinoembryonic antigen.Ann Intern Med. 1986; 104 ([review]): 66-73
- The effect of disease verification and referral on the relationship between symptoms and diseases.Med Decis Making. 1987; 7: 139-148
- Workup bias in prediction research.Med Decis Making. 1987; 7: 115-119
- Factors affecting sensitivity and specificity of a diagnostic test: the exercise thallium scintigram.Am J Med. 1988; 84: 699-710
- The logistic modelling of sensitivity, specificity and predictive value of a diagnostic test.J Clin Epidemiol. 1992; 45: 9-13
- What is the effect of sampling error on ROC analysis in the face of verification bias?.Med Decis Making. 1992; 12 ([letter; comment]): 155-156
- Computerized health information in The Netherlands: a registration network of family practices.Br J Gen Pract. 1992; 42: 102-106
- Transporting databanks of medical information from one location to another.Eff Health Care. 1983; 1: 155-163
- Prediction of bacterial meningitis in children with meningeal signs: reduction of lumbar punctures.Acta Paediatr. 2001; 90: 611-617
- Signs of meningeal irritation at the emergency department; how often bacterial meningitis?.Pediatr Emerg Care. 2001; 17: 161-164
- Serious bacterial infections in febrile infants and children selected for lumbar puncture.Pediatr Infect Dis J. 1994; 13: 950-953
- Diseases that mimic meningitis. Analysis of 650 lumbar punctures.Clin Pediatr. 1990; 29 (258–261): 254-255
- The threshold approach to clinical decision making.N Engl J Med. 1980; 302: 1109-1117
- Probability theory in the use of diagnostic tests. An introduction to critical study of the literature.Ann Intern Med. 1986; 104: 60-66
- Evaluation of diagnostic imaging tests: diagnostic probability estimation.J Clin Epidemiol. 1998; 51: 1293-1298
- Assessment of radiologic tests: control of bias and other design considerations.Radiology. 1988; 167: 565-569
- Bacterial meningitis in the first three months of life.Postgrad Med J. 1995; 71: 36-38
- Childhood bacterial meningitis: initial symptoms and signs related to age, and reasons for consulting a physician.Eur J Pediatr. 1987; 146: 515-518
- A problem oriented patient classification system for general pediatrics I.Tijdschr Kindergeneesk. 1996; 64 ([in Dutch, English summary]): 93-98
- A problem oriented patient classification system for general pediatrics II.Tijdschr Kindergeneesk. 1996; 64 ([in Dutch, English summary]): 99-104
- Lamberts H Wood M Hofmans-Okkes I The international Classification for Primary Care in the European Community; with a multip-language layer. Oxford University Press, Oxford1993
- Verification bias in pediatric studies evaluating diagnostic tests.J Pediatr. 1993; 122: 585-590
- Diagnosis and management of meningitis.Pediatr Infect Dis J. 1992; 11: 785-814
- Value of the ventilation/perfusion scan in acute pulmonary embolism. Results of the prospective investigation of pulmonary embolism diagnosis (PIOPED).JAMA. 1990; 263: 2753-2759
- Variability in radiologists' interpretations of mammograms.N Engl J Med. 1994; 331: 1493-1499
- Assessing rater performance without a “gold standard” using consensus theory.Med Decis Making. 1997; 17: 71-79
- An epidemic of heart failure. Recent evidence from Europe.Eur Heart J. 1998; 19: L2-L9
- Clinical diagnosis and the order of information.Med Decis Making. 1998; 18: 412-417
- Measuring the accuracy of diagnostic systems.Science. 1988; 240: 1285-1293
- Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors.Stat Med. 1996; 15: 361-387
- Regression with missing X's: a review.J Am Stat Assoc. 1992; 87: 1227-1237
- A critical look at methods for handling missing covariates in epidemiologic regression analyses.Am J Epidemiol. 1995; 142: 1255-1264
- Multiple imputation: a primer.Stat Methods Med Res. 1999; 8: 3-15
- Clinical prediction rules. A review and suggested modifications of methodological standards.JAMA. 1997; 277: 488-494
- The meaning and use of the area under a receiver operating characteristic (ROC) curve.Radiology. 1982; 143: 29-36
- A method of comparing the areas under receiver operating characteristic curves derived from the same cases.Radiology. 1983; 148: 839-843
- Clinical biostatistics.Clin Pharmacol Ther. 1977; 22: 485-498
- An evaluation of methods for estimating the area under the receiver operating characteristic (ROC) curve.Med Decis Making. 1985; 5: 149-156
- Hosmer D.W Lemeshow S Applied logistic regression. John Wiley & Sons, Inc, New York1989
- A simulation study of the number of events per variable in logistic regression analysis.J Clin Epidemiol. 1996; 49: 1373-1379
- Treatment thresholds in diagnostic test evaluation: an alternative approach to the comparison of areas under the receiver operating characteristic curve.Med Decis Making. 1997; 17: 447-454
Article info
Publication history
Accepted:
June 12,
2002
Received in revised form:
February 15,
2002
Received:
February 16,
2001
Identification
Copyright
© 2003 Elsevier Inc. Published by Elsevier Inc. All rights reserved.
