Abstract
Objective
If a gold standard is lacking in a diagnostic test accuracy study, expert diagnosis
is frequently used as reference standard. However, interobserver and intraobserver
agreements are imperfect. The aim of this study was to quantify the reproducibility
of a panel diagnosis for pediatric infectious diseases.
Study Design and Setting
Pediatricians from six countries adjudicated a diagnosis (i.e., bacterial infection,
viral infection, or indeterminate) for febrile children. Diagnosis was reached when
the majority of panel members came to the same diagnosis, leaving others inconclusive.
We evaluated intraobserver and intrapanel agreement with 6 weeks and 3 years’ time
intervals. We calculated the proportion of inconclusive diagnosis for a three-, five-,
and seven-expert panel.
Results
For both time intervals (i.e., 6 weeks and 3 years), intrapanel agreement was higher
(kappa 0.88, 95%CI: 0.81-0.94 and 0.80, 95%CI: NA) compared to intraobserver agreement
(kappa 0.77, 95%CI: 0.71-0.83 and 0.65, 95%CI: 0.52-0.78). After expanding the three-expert
panel to five or seven experts, the proportion of inconclusive diagnoses (11%) remained
the same.
Conclusion
A panel consisting of three experts provides more reproducible diagnoses than an individual
expert in children with lower respiratory tract infection or fever without source.
Increasing the size of a panel beyond three experts has no major advantage for diagnosis
reproducibility.
Keywords
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Journal of Clinical EpidemiologyAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- A review of solutions for diagnostic accuracy studies with an imperfect or missing reference standard.J Clin Epidemiol. 2009; 62: 797-806
- A systematic review on the diagnosis of pediatric bacterial pneumonia: when gold is bronze.PLoS One. 2010; 5: e11989
- Evaluation of diagnostic tests when there is no gold standard. A review of methods.Health Technol Assess. 2007; 11 (iii, ix-51)
- Single-center study of interrater agreement in the identification of central line-associated bloodstream infection.Am J Infect Control. 2014; 42: 638-642
- Validation of statewide surveillance system data on central line-associated bloodstream infection in intensive care units in Australia.Infect Control Hosp Epidemiol. 2009; 30: 1045-1049
- Inter-observer agreement in interpreting chest X-rays on children with acute lower respiratory tract infections and concurrent wheezing.Sao Paulo Med J. 2007; 125: 150-154
- Hindsight judgement on ambiguous episodes of suspected infection in critically ill children: poor consensus amongst experts?.Eur J Pediatr. 2003; 162: 840-843
- Is single observer identification of wound infection a reliable endpoint?.J Emerg Med. 2002; 23: 333-335
- Interobserver reliability of radiologists' interpretations of mobile chest radiographs for nursing home-acquired pneumonia.J Am Med Dir Assoc. 2006; 7: 416-419
- Interobserver variability in ventilator-associated pneumonia surveillance.Am J Infect Control. 2010; 38: 237-239
- Clinicians are right not to like Cohen's kappa.BMJ. 2013; 346: f2125
- Ventilator-associated pneumonia in intubated children: comparison of different diagnostic methods.Pediatr Crit Care Med. 2003; 4: 437-443
- Use of expert panels to define the reference standard in diagnostic research: a systematic review of published methods and reporting.PLoS Med. 2013; 10: e1001531
- A host-protein based assay to differentiate between bacterial and viral infections in preschool children (OPPORTUNITY): a double-blind, multicentre, validation study.Lancet Infect Dis. 2017; 17: 431-440
- Interrater agreement of the diagnosis and classification of a first seizure in childhood. The Dutch Study of Epilepsy in Childhood.J Neurol Neurosurg Psychiatry. 2004; 75: 241-245
- Validation of consensus panel diagnosis in dementia.Arch Neurol. 2010; 67: 1506-1512
- Computing inter-rater reliability for observational data: an overview and tutorial.Tutor Quant Methods Psychol. 2012; 8: 23-34
- The measurement of observer agreement for categorical data.Biometrics. 1977; 33: 159-174
- Reducing interrater variability and improving health care: a meta-analytical review.J Eval Clin Pract. 2012; 18: 887-895
- Inter- and intra-observer variability in biopsy of bone and soft tissue sarcomas.Anticancer Res. 2015; 35: 961-966
- Inter- and intra-observer agreement of the AO classification for operatively treated distal radius fractures.Strategies Trauma Limb Reconstr. 2015; 10: 155-159
- Evaluation of inter- and intra-observer reliability of current classification systems for subtrochanteric femoral fractures.Eur J Orthop Surg Traumatol. 2018; 28: 499-502
- Inter- and intra-observer reproducibility of quantitative analysis for FP-CIT SPECT in patients with DLB.Ann Nucl Med. 2017; 31: 758-763
- Reliability and validity of the World Health Organization reading standards for paediatric chest radiographs used in the field in an impact study of Pneumococcal Conjugate Vaccine in Kilifi, Kenya.PLoS One. 2018; 13: e0200715
- A randomized, double-blind, placebo-controlled trial of aripiprazole lauroxil in acute exacerbation of schizophrenia.J Clin Psychiatry. 2015; 76: 1085-1090
- Effect of aripiprazole lauroxil on agitation and hostility in patients with schizophrenia.Int Clin Psychopharmacol. 2016; 31: 69-75
Article info
Publication history
Published online: March 28, 2019
Accepted:
March 18,
2019
Footnotes
Funding: The first author (C.H.) was funded by the TAILORED-Treatment study, which received funding from the EU's Seventh Framework Program FP7 under REA grant agreement No. HEALTH-F3–602860-2013 (TAILORED-Treatment; www.tailored-treatment.eu/).
Conflict of interest statement: The authors have no conflicts of interest relevant to this article to disclose.
Authors' contributions: All authors were responsible for the design of the study. C.H., L.A., S.A., W.A., I.C., T.C., A.G., I.G., A.G., D.G., I.I., A.K., L.L., Y.L., F.L., C.M., S.M., B.N., C.P., A.R., H.R., M.S., R.T., T.T., V.U., W.W., S.W., J.W., K.W., T.W., N.M., L.S., and L.B. performed the experiments. All authors interpreted the data. C.H., C.N., N.M., T.G., A.C., K.O., E.E., P.F., L.S., and L.B. analyzed the data. All authors were responsible for writing and/or critically revisions of the article.
Identification
Copyright
© 2019 Elsevier Inc. All rights reserved.
