Making protocols available with the article improved evaluation of selective outcome reporting

  • Louis Calméjane
    Affiliations
    Faculté de Médecine, Université Paris Descartes, Sorbonne Paris Cité, Paris, France

    Centre d’Épidémiologie Clinique, Hôpital Hôtel Dieu, Assistance Publique–Hôpitaux de Paris (AP-HP), Paris, France

    INSERM, U1153, Paris, France
    Search for articles by this author
  • Agnès Dechartres
    Correspondence
    Corresponding author. Département Biostatistique, Santé Publique, Information Médicale – Hôpital Pitié Salpêtrière, 47/83 Boulevard de l’Hôpital, 75013 Paris, France. Tel.: +33 (0)1 42 16 05 99; fax: +33 (0)1 42 16 03 32.
    Affiliations
    Faculté de Médecine, Université Paris Descartes, Sorbonne Paris Cité, Paris, France

    Centre d’Épidémiologie Clinique, Hôpital Hôtel Dieu, Assistance Publique–Hôpitaux de Paris (AP-HP), Paris, France

    INSERM, U1153, Paris, France

    Cochrane France, Paris, France

    Sorbonne Université, INSERM U1136, Institut Pierre Louis d’Epidémiologie et de Santé Publique, Département de Biostatistique, Santé Publique, Information Médicale, Hôpital Pitié Salpêtrière, AP-HP, Paris, France
    Search for articles by this author
  • Viet Thi Tran
    Affiliations
    Centre d’Épidémiologie Clinique, Hôpital Hôtel Dieu, Assistance Publique–Hôpitaux de Paris (AP-HP), Paris, France

    INSERM, U1153, Paris, France
    Search for articles by this author
  • Philippe Ravaud
    Affiliations
    Faculté de Médecine, Université Paris Descartes, Sorbonne Paris Cité, Paris, France

    Centre d’Épidémiologie Clinique, Hôpital Hôtel Dieu, Assistance Publique–Hôpitaux de Paris (AP-HP), Paris, France

    INSERM, U1153, Paris, France

    Cochrane France, Paris, France

    Department of Epidemiology, Mailman School of Public Health, Columbia University, New York, USA
    Search for articles by this author
Published:September 06, 2018DOI:https://doi.org/10.1016/j.jclinepi.2018.08.020

      Highlights

      • In this methodological review based on 274 trials published in high impact factor journals with a protocol available, we found that selective outcome reporting (SOR) could not be evaluated for 30 RCTs (10.9%) because of insufficient description of the primary outcomes in trial registries.
      • With protocols available, there were only three RCTs (1.1%) for which the risk of SOR could not be assessed, and we suspected additional cases of SOR that were not identified with trial registries alone.
      • Making protocols available along with the article may improve evaluation of SOR.

      Abstract

      Objective

      To compare primary outcomes reported in publications, protocols and registries and to evaluate the contribution of available protocols to assess selective outcome reporting (SOR) as compared with registration alone.

      Study Design and Setting

      We included all randomized controlled trials (RCTs) published in 2015 and 2016 in the five leading general medical journals. For each RCT, we evaluated whether the protocol was available and searched for registration. We extracted all primary outcomes reported in publications, registries, and protocols. We evaluated whether SOR was suspected (i.e., at least one discrepancy in primary outcomes), unclear, or not suspected based on comparisons of publications and (1) trial registration alone or (2) protocols in addition to registration.

      Results

      Selective outcome reporting was suspected for 77/274 (28.1%), unclear for 30 (10.9%), and not suspected for 167 (60.9%) when comparing publications and trial registration alone. With protocols available, the classification changed for 38 RCTs (13.9%): 11 not suspected of SOR based on registration became suspected of SOR with protocols available, and 27 with unclear assessment based on registration became suspected of SOR (n = 7) and not suspected of SOR (n = 20) with protocols available.

      Conclusions

      Compared to registration alone, making protocols available allows for a more precise evaluation of SOR.

      Keywords

      To read this article in full you will need to make a payment

      References

      1. The Oxford 2011 levels of evidence.
        (Available at:) (Accessed September 24, 2018)
        • Jones C.W.
        • Keil L.G.
        • Holland W.C.
        • Caughey M.C.
        • Platts-Mills T.F.
        Comparison of registered and published outcomes in randomized controlled trials: a systematic review.
        BMC Med. 2015; 13: 282
        • Kirkham J.J.
        • Dwan K.M.
        • Altman D.G.
        • Gamble C.
        • Dodd S.
        • Smyth R.
        • et al.
        The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews.
        BMJ. 2010; 340: c365
        • Raghav K.P.
        • Mahajan S.
        • Yao J.C.
        • Hobbs B.P.
        • Berry D.A.
        • Pentz R.D.
        • et al.
        From protocols to publications: a study in selective reporting of outcomes in randomized trials in oncology.
        J Clin Oncol. 2015; 33: 3583-3590
        • Zarin D.A.
        • Tse T.
        • Williams R.J.
        • Califf R.M.
        • Ide N.C.
        The ClinicalTrials.gov results database--update and key issues.
        N Engl J Med. 2011; 364: 852-860
        • Dwan K.
        • Gamble C.
        • Williamson P.R.
        • Kirkham J.J.
        Systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review.
        PLoS One. 2013; 8: e66844
        • De Angelis C.
        • Drazen J.M.
        • Frizelle F.A.
        • Haug C.
        • Hoey J.
        • Horton R.
        • et al.
        Clinical trial registration: a statement from the International Committee of Medical Journal Editors.
        Lancet. 2004; 364: 911-912
        • Zarin D.A.
        • Tse T.
        • Ide N.C.
        Trial registration at ClinicalTrials.gov between May and October 2005.
        N Engl J Med. 2005; 353: 2779-2787
        • Dechartres A.
        • Ravaud P.
        • Atal I.
        • Riveros C.
        • Boutron I.
        Association between trial registration and treatment effect estimates: a meta-epidemiological study.
        BMC Med. 2016; 14: 100
        • Farquhar C.M.
        • Showell M.G.
        • Showell E.A.E.
        • Beetham P.
        • Baak N.
        • Mourad S.
        • et al.
        Clinical trial registration was not an indicator for low risk of bias.
        J Clin Epidemiol. 2017; 84: 47-53
        • Mathieu S.
        • Boutron I.
        • Moher D.
        • Altman D.G.
        • Ravaud P.
        Comparison of registered and published primary outcomes in randomized controlled trials.
        JAMA. 2009; 302: 977-984
        • Harriman S.L.
        • Patel J.
        When are clinical trials registered? An analysis of prospective versus retrospective registration.
        Trials. 2016; 17: 187
        • Chan A.W.
        Bias, spin, and misreporting: time for full access to trial protocols and results.
        PLoS Med. 2008; 5: e230
        • Glasziou P.
        • Altman D.G.
        • Bossuyt P.
        • Boutron I.
        • Clarke M.
        • Julious S.
        • et al.
        Reducing waste from incomplete or unusable reports of biomedical research.
        Lancet. 2014; 383: 267-276
      2. Medicine Io: sharing clinical trial data: maximizing benefits, minimizing risk. The National Academies Press, Washington, DC2015
      3. Higgins J.P.T. Green S. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011 (Available at:)
        • Zarin D.A.
        • Tse T.
        • Williams R.J.
        • Rajakannan T.
        Update on trial registration 11 years after the ICMJE policy was established.
        N Engl J Med. 2017; 376: 383-391
        • Dwan K.
        • Altman D.G.
        • Cresswell L.
        • Blundell M.
        • Gamble C.L.
        • Williamson P.R.
        Comparison of protocols and registry entries to published reports for randomised controlled trials.
        Cochrane Database Syst Rev. 2011; : MR000031
        • Litonjua A.A.
        • Carey V.J.
        • Laranjo N.
        • Harshfield B.J.
        • McElrath T.F.
        • O'Connor G.T.
        • et al.
        Effect of prenatal supplementation with vitamin d on asthma or recurrent wheezing in offspring by age 3 years: the VDAART randomized clinical trial.
        JAMA. 2016; 315: 362-370
        • Chow C.K.
        • Redfern J.
        • Hillis G.S.
        • Thakkar J.
        • Santo K.
        • Hackett M.L.
        • et al.
        Effect of lifestyle-focused text messaging on risk factor modification in patients with coronary heart disease: a randomized clinical trial.
        JAMA. 2015; 314: 1255-1263
        • Schulz K.F.
        • Altman D.G.
        • Moher D.
        CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials.
        BMJ. 2010; 340: c332
        • Al-Marzouki S.
        • Roberts I.
        • Evans S.
        • Marshall T.
        Selective reporting in clinical trials: analysis of trial protocols accepted by the Lancet.
        Lancet. 2008; 372: 201
        • Chan A.W.
        • Hrobjartsson A.
        • Haahr M.T.
        • Gotzsche P.C.
        • Altman D.G.
        Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles.
        JAMA. 2004; 291: 2457-2465
        • Chan A.W.
        • Krleza-Jeric K.
        • Schmid I.
        • Altman D.G.
        Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research.
        CMAJ. 2004; 171: 735-740
      4. The COMPare trials project.
        (Available at:)
        • Chan A.
        • Tetzlaff J.M.
        • Altman D.G.
        • Laupacis A.
        • Gøtzsche P.C.
        • Krleža-Jerić K.
        • et al.
        Spirit 2013 statement: defining standard protocol items for clinical trials.
        Ann Intern Med. 2013; 158: 200-207
        • Ioannidis J.P.A.
        • Greenland S.
        • Hlatky M.A.
        • Khoury M.J.
        • Macleod M.R.
        • Moher D.
        • et al.
        Increasing value and reducing waste in research design, conduct, and analysis.
        Lancet. 2014; 383: 166-175
        • Weber W.E.
        • Merino J.G.
        • Loder E.
        Trial registration 10 years on.
        BMJ. 2015; 351: h3572
        • Moher D.
        Reporting research results: a moral obligation for all researchers.
        Can J Anesth. 2007; 54: 331
        • Bala M.M.
        • Akl E.A.
        • Sun X.
        • Bassler D.
        • Mertz D.
        • Mejza F.
        • et al.
        Randomized trials published in higher vs. lower impact journals differ in design, conduct, and analysis.
        J Clin Epidemiol. 2013; 66: 286-295
        • Dechartres A.
        • Trinquart L.
        • Atal I.
        • Moher D.
        • Dickersin K.
        • Boutron I.
        • et al.
        Evolution of poor reporting and inadequate methods over time in 20 920 randomised controlled trials included in Cochrane reviews: research on research study.
        BMJ. 2017; 357: j2490