Citations alone were enough to predict favorable conclusions in reviews of neuraminidase inhibitors



      To examine the use of supervised machine learning to identify biases in evidence selection and determine if citation information can predict favorable conclusions in reviews about neuraminidase inhibitors.

      Study Design and Setting

      Reviews of neuraminidase inhibitors published during January 2005 to May 2013 were identified by searching PubMed. In a blinded evaluation, the reviews were classified as favorable if investigators agreed that they supported the use of neuraminidase inhibitors for prophylaxis or treatment of influenza. Reference lists were used to identify all unique citations to primary articles. Three classification methods were tested for their ability to predict favorable conclusions using only citation information.


      Citations to 4,574 articles were identified in 152 reviews of neuraminidase inhibitors, and 93 (61%) of these reviews were graded as favorable. Primary articles describing drug resistance were among the citations that were underrepresented in favorable reviews. The most accurate classifier predicted favorable conclusions with 96.2% accuracy, using citations to only 24 of 4,574 articles.


      Favorable conclusions in reviews about neuraminidase inhibitors can be predicted using only information about the articles they cite. The approach highlights how evidence exclusion shapes conclusions in reviews and provides a method to evaluate citation practices in a corpus of reviews.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Greenberg S.A.
        How citation distortions create unfounded authority: analysis of a citation network.
        BMJ. 2009; 339: b2680
        • Gøtzsche P.C.
        Reference bias in reports of drug trials.
        BMJ. 1987; 295: 654-656
        • Moscona A.
        Neuraminidase inhibitors for influenza.
        N Engl J Med. 2005; 353: 1363-1373
        • Smith J.R.
        • Rayner C.R.
        • Donner B.
        • Wollenhaupt M.
        • Klumpp K.
        • Dutkowski R.
        Oseltamivir in seasonal, pandemic, and avian influenza: a comprehensive review of 10-years clinical experience.
        Adv Ther. 2011; 28: 927-959
        • Monto A.S.
        The risk of seasonal and pandemic influenza: prospects for control.
        Clin Infect Dis. 2009; 48: S20-S25
        • Jones M.
        • Del Mar C.
        Safety of neuraminidase inhibitors for influenza.
        Expert Opin Drug Saf. 2006; 5: 603-608
        • Jefferson T.
        • Jones M.
        • Doshi P.
        • Mar C.D.
        Neuraminidase inhibitors for preventing and treating influenza in healthy adults: systematic review and meta-analysis.
        BMJ. 2009; 339: b5106
        • Doshi P.
        Neuraminidase inhibitors—the story behind the Cochrane review.
        BMJ. 2009; 339: b5164
        • Cohen D.
        • Carter P.
        WHO and the pandemic flu “conspiracies”.
        BMJ. 2010; 340: c2912
        • Bax L.
        • Moons K.G.
        Beyond publication bias.
        J Clin Epidemiol. 2011; 64: 459-462
        • Moscati R.
        • Jehle D.
        • Ellis D.
        • Fiorello A.
        • Landi M.
        Positive-outcome bias: comparison of emergency medicine and general medicine literatures.
        Acad Emerg Med. 1994; 1: 267-271
        • Jannot A.-S.
        • Agoritsas T.
        • Gayet-Ageron A.
        • Perneger T.V.
        Citation bias favoring statistically significant studies was present in medical research.
        J Clin Epidemiol. 2013; 66: 296-301
        • Nieminen P.
        • Rucker G.
        • Miettunen J.
        • Carpenter J.
        • Schumacher M.
        Statistically significant papers in psychiatry were cited more often than others.
        J Clin Epidemiol. 2007; 60: 939-946
        • Etter J.-F.
        • Stapleton J.
        Citations to trials of nicotine replacement therapy were biased toward positive results and high-impact-factor journals.
        J Clin Epidemiol. 2009; 62: 831-837
        • Egger M.
        • Smith G.D.
        Meta-analysis bias in location and selection of studies.
        BMJ. 1998; 316: 61-66
        • Okike K.
        • Kocher M.S.
        • Torpey J.L.
        • Nwachukwu B.U.
        • Mehlman C.T.
        • Bhandari M.
        Level of evidence and conflict of interest disclosure associated with higher citation rates in orthopedics.
        J Clin Epidemiol. 2011; 64: 331-338
        • Perneger T.V.
        Citation analysis of identical consensus statements revealed journal-related bias.
        J Clin Epidemiol. 2010; 63: 660-664
        • Kulkarni A.V.
        • Aziz B.
        • Shams I.
        • Busse J.W.
        Author self-citation in the general medicine literature.
        PLoS One. 2011; 6: e20885
        • Falagas M.E.
        • Kavvadia P.
        “Eigenlob”: self-citation in biomedical journals.
        FASEB J. 2006; 20: 1039-1042
        • Greenberg S.A.
        Understanding belief using citation networks.
        J Eval Clin Pract. 2011; 17: 389-393
        • Fiorentino F.
        • Vasilakis C.
        • Treasure T.
        Clinical reports of pulmonary metastasectomy for colorectal cancer: a citation network analysis.
        Br J Cancer. 2011; 104: 1085-1097
        • Kho M.E.
        • Brouwers M.C.
        The systematic review and bibliometric network analysis (SeBriNA) is a new method to contextualize evidence. Part 1: description.
        J Clin Epidemiol. 2012; 65: 1010-1015
        • Kho M.E.
        • Brouwers M.C.
        Application of the systematic review and bibliometric network analysis (SeBriNA) methodology contextualizes evidence: part 2: rituximab for non-Hodgkin's lymphoma.
        J Clin Epidemiol. 2012; 65: 996-1009
        • Trikalinos T.A.
        • Moorthy D.
        • Chung M.
        • Yu W.W.
        • Lee J.
        • Lichtenstein A.H.
        • et al.
        Concordance of randomized and nonrandomized studies was unrelated to translational patterns of two nutrient-disease associations.
        J Clin Epidemiol. 2012; 65: 16-29
        • Lau A.Y.S.
        • Coiera E.W.
        A Bayesian model that predicts the impact of Web searching on decision making.
        J Am Soc Inf Sci Tech. 2006; 57: 873-880
        • Bekhuis T.
        • Demner-Fushman D.
        Screening nonrandomized studies for medical systematic reviews: a comparative study of classifiers.
        Artif Intell Med. 2012; 55: 197-207
        • Cortes C.
        • Vapnik V.
        Support-vector networks.
        Machine Learn. 1995; 20: 273-297
        • Scholkopf B.
        • Smola A.
        Learning with kernels: support vector machines, regularization, optimization, and beyond.
        MIT Press, Cambridge, MA, USA2001
        • Han J.
        • Kamber M.
        Data mining: concepts and techniques.
        2nd ed. Morgan Kaufmann, San Fransisco, CA, USA2006
        • Hand D.J.
        • Yu K.M.
        Idiot's Bayes—not so stupid after all?.
        Int Stat Rev. 2001; 69: 385-398
        • Domingos P.
        • Pazzani M.
        On the optimality of the simple Bayesian classifier under zero-one loss.
        Machine Learn. 1997; 29: 103-130
        • Cover T.
        • Hart P.
        Nearest neighbor pattern classification.
        IEEE Trans Inf Theor. 1967; 13: 21-27
        • Kohavi R.
        • John G.H.
        Wrappers for feature subset selection.
        Artif Intell. 1997; 97: 273-324
        • Guyon I.
        • Elisseeff A.
        An introduction to variable and feature selection.
        J Machine Learn Res. 2003; 3: 1157-1182
      1. Mierswa I, Wurst M, Klinkenberg R, Scholz M and Euler T. YALE: rapid prototyping for complex data mining tasks. In: Proceedings of the 12th ACM SIGKDD international conference on knowledge discovery and data mining, Philadelphia, PA, USA: ACM; 2006:935–40.

        • Latourrette M.
        Toward an explanatory similarity measure for nearest-neighbor classification.
        Machine learning ECML 2000, Springer Berlin Heidelberg2000: 238-245
        • Nguyen H.T.
        • Fry A.M.
        • Loveless P.A.
        • Klimov A.I.
        • Gubareva L.V.
        Recovery of a multidrug-resistant strain of pandemic influenza A 2009 (H1N1) virus carrying a dual H275Y/I223R mutation from a child after prolonged treatment with oseltamivir.
        Clin Infect Dis. 2010; 51: 983-984
        • Burch J.
        • Paulden M.
        • Conti S.
        • Stock C.
        • Corbett M.
        • Welton N.
        • et al.
        Antiviral drugs for the treatment of influenza: a systematic review and economic evaluation.
        Health Technol Assess. 2009; 13: 1-265
        • Adisasmito W.
        • Chan P.K.
        • Lee N.
        • Oner A.F.
        • Gasimov V.
        • Aghayev F.
        • et al.
        Effectiveness of antiviral treatment in human influenza A (H5N1) infections: analysis of a Global Patient Registry.
        J Infect Dis. 2010; 202: 1154-1160
        • Kiso M.
        • Kubo S.
        • Ozawa M.
        • Le Q.M.
        • Nidom C.A.
        • Yamashita M.
        • et al.
        Efficacy of the new neuraminidase inhibitor CS-8958 against H5N1 influenza viruses.
        PLoS Pathog. 2010; 6: e1000786
        • Tumpey T.M.
        • García-Sastre A.
        • Mikulasova A.
        • Taubenberger J.K.
        • Swayne D.E.
        • Palese P.
        • et al.
        Existing antivirals are effective against influenza viruses with genes from the 1918 pandemic virus.
        Proc Natl Acad Sci U S A. 2002; 99: 13849-13854
        • Krumholz H.M.
        • Jackevicius C.A.
        • Ross J.S.
        Tamiflu: 14 flu seasons and still questions.
        BMJ. 2013; 346
        • Jefferson T.
        • Doshi P.
        Multisystem failure: the story of anti-influenza drugs.
        BMJ. 2014; 348: g2263
        • Jefferson T.
        • Jones M.
        • Doshi P.
        • Spencer E.A.
        • Onakpoya I.
        • Heneghan C.J.
        Oseltamivir for influenza in adults and children: systematic review of clinical study reports and summary of regulatory comments.
        BMJ. 2014; 348: g2545
        • Dunn A.G.
        • Arachi D.
        • Hudgins J.
        • Tsafnat G.
        • Coiera E.
        • Bourgeois F.T.
        Financial conflicts of interest and conclusions about neuraminidase inhibitors for influenza: an analysis of systematic reviews.
        Ann Intern Med. 2014; 161: 513-518
        • Shotton D.
        Publishing: open citations.
        Nature. 2013; 502: 295-297