Advertisement

Letter to the Editor – Not even the top general medical journals are free of spin: A wake-up call based on an overview of reviews

      In order to make treatment decisions, clinicians often use systematic reviews of randomised controlled trials [
      • Bastian H
      • Glasziou P
      • Chalmers I.
      Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?.
      ]. Unfortunately, many reviews present spin (distorted interpretation of results [
      • Chiu K
      • Grundy Q
      • Bero L.
      Spin' in published biomedical literature: A methodological systematic review.
      ]), which could lead consumers of research to use unhelpful or even harmful interventions [
      • Boutron I
      • Dutton S
      • Ravaud P
      • et al.
      Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes.
      ].
      We conducted an overview to investigate the presence of spin in systematic reviews of interventions from the top 5 general medical journals, as well as to investigate characteristics that were associated with spin in the abstract. The top 5 journals based on the 2016 list of Impact Factors [
      JCR
      Journal Citation Reports (Journal Impact Factor 2017).
      ] were: 1) The Lancet; 2) Journal of American Medical Association (JAMA); 3) British Medical Journal (BMJ); 4) Annals of Internal Medicine (AIM); and 5) JAMA Internal Medicine. We included systematic reviews of clinical trials from these journals published between January 2011 and November 2017, evaluating the effect of interventions in comparison with inert interventions, placebo or no treatment.
      Two independent authors screened articles and extracted our data. We found 295 potentially eligible articles, from which we randomly selected 196: 32 from JAMA Internal Medicine; 39 from BMJ; 40 from Lancet; 42 from JAMA; and 43 from AIM (APPENDIX 1).
      We used the seven-item SPIN checklist [
      • Yavchitz A
      • Ravaud P
      • Altman DG
      • et al.
      A new classification of spin in systematic reviews and meta-analyses was developed and ranked according to the severity.
      ] to assess spin in both abstract and full text. The overall classification indicated that 94% of abstracts and 67% of full texts presented at least one item of the SPIN checklist. The most prevalent item in the abstracts was ‘beneficial effect despite high risk of bias’ (78% ‘yes’ and ‘not reported’). Regarding full texts, the most prevalent spin item was ‘selective reporting of harm outcomes’ (37% ‘yes’ and ‘not reported’) (TABLE 1). All journals presented at least one item of spin in the abstract, with only two journals showing better overall classification for all reviews: abstracts in the BMJ with 85% and AIM with 93%; and full texts in the BMJ with 51% and AIM with 61% (TABLE 2). The mean SPIN-score for all abstracts and full texts was low (APPENDIX 2). Abstracts and full texts were not considered consistent in terms of spin (APPENDIX 3).
      TABLE 1Number and percentages (%) of reviews achieving each item of spin checklist for the abstract and full text for the total sample (n=196).
      ITEMDESCRIPTIONABSTRACTFULL TEXT
      YESNOT REPORTEDNOYESNOT REPORTEDNO
      1. RecommendationRecommendation for clinical practice not supported by the findings12 (6.1)1 (0.5)183 (93.4)16 (8.2)0 (0.0)180 (91.8)
      2. Title claims a beneficial effect not supported by the findingsThe title claims a beneficial effect of the experimental treatment not supported by the findings1 (0.5)0 (0.0)195 (99.5)1 (0.5)0 (0.0)195 (99.5)
      3. Selective reporting of outcomesSelective reporting of outcomes favouring the beneficial effect of the experimental intervention26 (13.3)0 (0.0)170 (86.7)16 (8.2)0 (0.0)180 (91.8)
      4. Safety based on non-statistically significant resultsThe conclusion claims safety based on non-statistically significant results3 (1.5)3 (1.5)190 (97.0)5 (2.5)6 (3.1)185 (94.4)
      5. Beneficial effect despite high risk of biasThe conclusion claims the beneficial effect of the experimental treatment despite high risk of bias in primary studies8 (4.1)145 (74.0)43 (21.9)39 (19.9)7 (3.6)150 (76.5)
      6. Selective reporting of harm outcomesSelective reporting of harm outcomes of the experimental intervention1 (0.5)103 (52.6)92 (46.9)4 (2.0)69 (35.2)123 (62.8)
      7. Conclusion extrapolates findings to a different interventionThe conclusion extrapolates the review's findings to a different intervention0 (0.0)0 (0.0)196 (100.0)0 (0.0)0 (0.0)196 (100.0)
      TABLE 2Abstracts and full texts mean SPIN-score (SD) and number (%) of reviews with spin (levels of SPIN-score; range 0 to 7; i. e. no presence of spin to high level of spin) presented separately for each selected journal (n=196).
      JournalsTotal of reviewsAbstracts mean SPIN-score (SD)Full texts mean SPIN-score (SD)
      None of the abstracts and full texts presented SPIN score higher than 4.
      Abstracts with spin
      None of the abstracts and full texts presented SPIN score higher than 4.
      Full texts with spin
      At least one item SPIN-score 1 to 4SPIN-score (1 or 2)SPIN-score (3 or 4)At least one item SPIN-score 1 to 4SPIN-score (1 or 2)SPIN-score (3 or 4)
      The Lancet401.6 (0.7)0.9 (0.7)39 (
      The remaining percentage did not present spin (score=0).
      97.5)
      36 (90.0)3 (7.5)29 (72.5)29 (72.5)0 (0.0)
      JAMA421.6 (0.7)0.8 (0.7)42 (100.0)40 (95.2)2 (4.8)30 (71.4)29 (69.0)1 (2.4)
      BMJ391.3 (0.7)0.6 (0.6)33 (
      The remaining percentage did not present spin (score=0).
      84.6)
      33 (84.6)0 (0.0)20 (51.3)20 (51.3)0 (0.0)
      AIM431.5 (0.9)0.8 (0.8)40 (
      The remaining percentage did not present spin (score=0).
      93.0)
      35 (81.4)5 (11.6)26 (60.5)24 (55.8)2 (4.7)
      JAMA Internal Medicine321.7 (0.7)1.2 (0.8)31 (
      The remaining percentage did not present spin (score=0).
      96.9)
      27 (84.4)4 (12.5)26 (81.2)25 (78.1)1 (3.1)
      Note: SD - Standard Deviation.
      low asterisk The remaining percentage did not present spin (score=0).
      # None of the abstracts and full texts presented SPIN score higher than 4.
      The reviews’ methodological quality was assessed by the AMSTAR-2 checklist [
      • Shea BJ
      • Reeves BC
      • Wells G
      • et al.
      AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both.
      ]. We observed critically low overall confidence in 77% reviews, 19% with low, 3% with moderate and only 1% with high overall confidence (APPENDIX 4).
      We also found an association between abstracts with higher levels of spin and reviews with critically low methodological quality (ß 0.22; 95%CI: 0.14, 0.63), when compared to reviews of high methodological quality (ß -0.12; 95%CI: -2.69, 0.18) (APPENDIX 2).
      A recent systematic review has proven the presence of spin in different healthcare fields [
      • Nascimento DP
      • Ostelo R
      • van Tulder MW
      • et al.
      Do not make clinical decisions based on abstracts of healthcare research: A systematic review.
      ]. While some journals in rehabilitation health have already started to modify their policies [

      Heinemann AW, Chan L, Hoenig HM. Archives Seeks to Improve Abstract Reporting Quality. Archives of physical medicine and rehabilitation2019;100(10):1990-91. doi: 10.1016/j.apmr.2019.05.027

      ,
      • Nascimento DP
      • Costa LOP.
      Spin of results in scientific articles might kill you.
      ], awareness of spin is essentially needed among all journal editorial boards. Health professionals should ultimately know how to interpret the study results by themselves. Hopefully, this letter will help authors, journal editors and reviewers to improve the integrity and transparency of health research journals. Therefore, we strongly recommend training in adequate reporting and interpretation for all research peers.

      Declaration of Competing Interest

      The authors declare no competing interests.

      Source of Funding

      All authors declare no support or no financial relationships from any organization for the submitted work with any organisations.

      Prior presentation(s)

      None to declare.

      Appendix. Supplementary materials

      References

        • Bastian H
        • Glasziou P
        • Chalmers I.
        Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?.
        PLoS Med. 2010; 7e1000326https://doi.org/10.1371/journal.pmed.1000326
        • Chiu K
        • Grundy Q
        • Bero L.
        Spin' in published biomedical literature: A methodological systematic review.
        PLoS Biol. 2017; 15e2002173https://doi.org/10.1371/journal.pbio.2002173
        • Boutron I
        • Dutton S
        • Ravaud P
        • et al.
        Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes.
        Jama. 2010; 303: 2058-2064https://doi.org/10.1001/jama.2010.651
        • JCR
        Journal Citation Reports (Journal Impact Factor 2017).
        Clarivate Analytics. 2021; (Available from:)
        • Yavchitz A
        • Ravaud P
        • Altman DG
        • et al.
        A new classification of spin in systematic reviews and meta-analyses was developed and ranked according to the severity.
        J Clin Epidemiol. 2016; 75 ([published Online First: 2016/02/05]): 56-65https://doi.org/10.1016/j.jclinepi.2016.01.020
        • Shea BJ
        • Reeves BC
        • Wells G
        • et al.
        AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both.
        BMJ. 2017; 358: j4008https://doi.org/10.1136/bmj.j4008
        • Nascimento DP
        • Ostelo R
        • van Tulder MW
        • et al.
        Do not make clinical decisions based on abstracts of healthcare research: A systematic review.
        J Clin Epidemiol. 2021; : 136-157https://doi.org/10.1016/j.jclinepi.2021.03.030
      1. Heinemann AW, Chan L, Hoenig HM. Archives Seeks to Improve Abstract Reporting Quality. Archives of physical medicine and rehabilitation2019;100(10):1990-91. doi: 10.1016/j.apmr.2019.05.027

        • Nascimento DP
        • Costa LOP.
        Spin of results in scientific articles might kill you.
        Braz J Phys Ther. 2019; 23: 365-366https://doi.org/10.1016/j.bjpt.2019.07.007