Advertisement

Retracted systematic reviews continued to be frequently cited: a citation analysis

  • Zijun Wang
    Affiliations
    Evidence-based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou, China
    Search for articles by this author
  • Qianling Shi
    Affiliations
    The First School of Clinical Medicine, Lanzhou University, Lanzhou, China
    Search for articles by this author
  • Qi Zhou
    Affiliations
    Evidence-based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou, China
    Search for articles by this author
  • Siya Zhao
    Affiliations
    School of Public Health, Lanzhou University, Lanzhou, China
    Search for articles by this author
  • Ruizhen Hou
    Affiliations
    The First School of Clinical Medicine, Lanzhou University, Lanzhou, China
    Search for articles by this author
  • Shuya Lu
    Affiliations
    Department of Pediatric, Sichuan Provincial People's Hospital, University of Electronic Science and Technology of China, Chengdu, China

    Chinese Academy of Sciences Sichuan Translational Medicine Research Hospital, Chengdu, China
    Search for articles by this author
  • Xia Gao
    Affiliations
    The Second School of Clinical Medicine, Lanzhou University, Lanzhou, China
    Search for articles by this author
  • Yaolong Chen
    Correspondence
    Corresponding author. Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou 730000, China. Tel.: 13893104140.
    Affiliations
    Evidence-based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou, China

    Research Unit of Evidence-Based Evaluation and Guidelines, Chinese Academy of Medical Sciences (2021RU017), School of Basic Medical Sciences, Lanzhou University Lanzhou, Lanzhou, China

    Lanzhou University Institute of Health Data Science, Lanzhou, China

    WHO Collaborating Center for Guideline Implementation and Knowledge Translation, Lanzhou, China

    Guideline International Network Asia, Lanzhou, China

    Chinese GRADE Center, Lanzhou, China
    Search for articles by this author
  • on behalf ofPRESS working group
Open AccessPublished:May 27, 2022DOI:https://doi.org/10.1016/j.jclinepi.2022.05.013

      Highlights

      • Systematic reviews continue to be cited after being retracted, even after the retraction notice has been published.
      • Although the citations became less frequent over time since retraction, retracted studies were still being cited even years after the retraction.
      • Only few citations declared the retraction of the cited SR in the reference section.

      Abstract

      Background and Objectives

      To survey the citations of retracted non-Cochrane systematic reviews (SRs) in scientific literature.

      Methods

      We searched the Web of Science and Google Scholar from their inception to 30 April 2020 to find the citations of 153 previously identified retracted non-Cochrane SRs. We calculated the numbers of citations before and after retraction separately. We also described how the citation addressed the retraction and how it was used in the article.

      Results

      A We identified 954 citations of 128 retracted SRs. The number of retracted SRs and citations reached the peak in 2014 and 2016, respectively, and the majority of the citations (n = 580, 60.8%) were in articles published after the SR was retracted. The mean number of citation per retracted SRs was 7.5. 2.6 before and 4.5 after the publication of the retraction notice. Twenty-nine (5.0%) citations indicated the retraction of the SRs in the reference section. Nine of these citations supported the retracted SR's results, and 15 disagreed with them.

      Conclusion

      Retracted SRs continue to be cited after the publication of the retraction notice. Standardized methods are needed to guide the management of retractions and avoid inappropriate citations of retracted articles.

      Keywords

      What is new?

        Key findings

      • Retracted systematic reviews continue to be cited after the publication of the retraction notice.
      • The failure of the journal to mark the retraction on time and the author's intentional or unintentional citation of retracted SRs without mentioning the retraction status may lead to the retracted SRs’ being cited even more after the retraction than before.

        What this adds to what was known?

      • A retracted article can still, despite the false results, influence future research in a positive way if correctly cited. As for the retraction, what should be done is to stop the spread of wrong information and cite them standardly.

        What is the implication and what should change now?

      • Authors should maintain academic rigor when citing SRs, and confirm the validity of all references before publication.
      • Periodical journals should establish an appropriate identification and management mechanism for retractions.

      1. Introduction

      Scientific rigor is the cornerstone of medical research and the main guarantor of scientific progress and effective dissemination of research achievements [
      • Bolboacă S.D.
      • Buhai D.V.
      • Aluaș M.
      • Bulboacă A.E.
      Post retraction citations among manuscripts reporting a radiology-imaging diagnostic method.
      ,
      • Cosentino A.M.
      • Veríssimo D.
      Ending the citation of retracted papers.
      ]. In recent years, the academic and professional pressure to publish has increased, which has seriously damaged the scientific integrity and rigor of research [
      • Nath S.B.
      • Marcus S.C.
      • Druss B.G.
      Retractions in the research literature: misconduct or mistakes?.
      ]. The annual number of articles withdrawn has increased 19-fold between 2001 and 2010, or 11-fold when adjusted for the total amount of published literature and repeat offenders [
      • Grieneisen M.L.
      • Zhang M.
      A comprehensive survey of retracted articles from the scholarly literature.
      ]. Retraction, usually disclosed in the form of a retraction statement issued by the journal, is the ultimate means to correct serious defects or errors in the literature, and alerts the reader of the unreliability of the article's content [
      • Wager E.
      • Barbour V.
      • Yentis S.
      • Kleinert S.
      Retractions: guidance from the committee on publication Ethics (COPE).
      ]. However, it is not uncommon that retracted literature continues to be cited [
      • Bolboacă S.D.
      • Buhai D.V.
      • Aluaș M.
      • Bulboacă A.E.
      Post retraction citations among manuscripts reporting a radiology-imaging diagnostic method.
      ,
      • Nair S.
      • Yean C.
      • Yoo J.
      • Leff J.
      • Delphin E.
      • Adams D.C.
      Reasons for article retraction in anesthesiology: a comprehensive analysis.
      ,
      • Hamilton D.G.
      Continued citation of retracted radiation oncology literature-do we have a problem?.
      ], including both citations that occur before the publication of the retraction notice, and erroneous citations that are made after the retraction. The former is a serious but inevitable issue that can affect the reliability of future research [
      • Hamilton D.G.
      Continued citation of retracted radiation oncology literature-do we have a problem?.
      ], but reasons and consequences of citations after the publication of the retraction notice are worth a detailed investigation.
      It is generally recognized that retraction should be the end of the life of a paper, but in practice this is not the case [
      • Cosentino A.M.
      • Veríssimo D.
      Ending the citation of retracted papers.
      ]. An article published by Wakefield et al. in 1998 was formally retracted by The Lancet in 2010 because of false ethical claims. However, the paper continues to be cited after the retraction [
      • Suelzer E.M.
      • Deal J.
      • Hanus K.L.
      • Ruggeri B.
      • Sieracki R.
      • Witkowski E.
      Assessment of Citations of the Retracted Article by Wakefield et al. With Fraudulent Claims of an Association Between Vaccination and Autism.
      ,
      • Chen C.M.
      • Hu Z.G.
      • Milbank J.
      • Schultz T.
      A visual analytic study of retracted articles in scientific literature.
      ,
      • Chen C.M.
      • Leydesdorff L.
      Patterns of connections and movements in dual-map overlays: a new method of publication portfolio analysis.
      ]. As of December, 2021 [], this paper has been cited more than 1,400 times according to the Web of Science, and nearly 50% of the citations occurred after the publication of the retraction notice. An investigation of post-retraction citations in the field of diagnostic radiography showed that 96% of the studies citing retracted articles used the citation to support their own findings, and less than 2% indicated in the text body or references that they cited retracted articles [
      • Bolboacă S.D.
      • Buhai D.V.
      • Aluaș M.
      • Bulboacă A.E.
      Post retraction citations among manuscripts reporting a radiology-imaging diagnostic method.
      ]. The Committee on Publication Ethics (COPE) states that the retraction of an article should be clearly identifiable on all electronic sources, including the journal website and any bibliographic databases [
      • Wager E.
      • Barbour V.
      • Yentis S.
      • Kleinert S.
      Retractions: guidance from the committee on publication Ethics (COPE).
      ]. Citing studies with an unclear retraction status has the potential to mislead clinical practice and future research, cause cascading damage to the quality of research, and undermine the scientific integrity of research.
      Systematic reviews (SRs) are considered as the backbone of evidence-based decision-making, and provide the best research evidence for clinical practice [
      • Sutton A.
      • Clowes M.
      • Preston L.
      • Booth A.
      Meeting the review family: exploring review types and associated information retrieval requirements.
      ,
      • Liberati A.
      • Altman D.G.
      • Tetzlaff J.
      • Mulrow C.
      • Gøtzsche P.C.
      • Ioannidis J.P.A.
      • et al.
      The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration.
      ,
      • Siddaway A.P.
      • Wood A.M.
      • Hedges L.V.
      How to do a systematic review: a best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses.
      ]. Meta-analyses and systematic reviews were the second most common type of study among articles retracted between 1971 and 2020 [
      • Gaudino M.
      • Robinson N.B.
      • Audisio K.
      • Rahouma M.
      • Benedetto U.
      • Kurlansky P.
      • et al.
      Trends and characteristics of retracted articles in the biomedical literature, 1971 to 2020.
      ]. Citations of retracted SRs may thus influence the reliability of clinical practice guidelines and affect policy making. On behalf of the Publication Science of Retracted Studies (PRESS) working group, we have previously analyzed the retraction status and reasons of non-Cochrane SRs in the field of medicine [
      • Shi Q.
      • Wang Z.
      • Zhou Q.
      • Hou R.
      • Gao X.
      • He S.
      • et al.
      More consideration is needed for retracted non-cochrane systematic reviews in medicine: a systematic review.
      ]. In the present study, we explored further the citations of retracted SRs and the factors affecting the number of their citations.

      2. Methods

      2.1 Literature search

      We included the 159 non-Cochrane SRs in medicine retracted between 2004 and 2020, which were identified in our previous study [
      • Shi Q.
      • Wang Z.
      • Zhou Q.
      • Hou R.
      • Gao X.
      • He S.
      • et al.
      More consideration is needed for retracted non-cochrane systematic reviews in medicine: a systematic review.
      ], and excluded the articles that were republished or not cited by any studies. To collect the citations of retracted SRs, two researchers (Zijun Wang and Qianling Shi) searched the “Cited Reference Search” of Web of Science as well as Google Scholar using the titles of the retrieved retracted SRs with no language restriction from their inception to 30 April 2020. We included citations of retracted SRs published in medical journals. We excluded citations that either (Ⅰ) did not actually cite the retracted SR; (Ⅱ) for which the full text could not be downloaded; or (Ⅲ) were duplicates.
      Two researchers (Zijun Wang and Qianling Shi) independently downloaded and screened the full-texts of the articles citing retracted SRs.

      2.2 Date extraction

      Basic information of the retracted SRs (first author, the World Health Organization [WHO] region of the first author, publication date, retraction date, main reason of retraction, number of citations per retracted SR, and whether the SR was cited by the retraction notice) were retrieved from our previous publication. Six researchers grouped in three pairs (Group 1: Zijun Wang, Qianling Shi; Group 2: Siya Zhao, Shuya Lu; Group 3: Qi Zhou, Ruizhen Hou) independently extracted the following data for each include retracted SR: (I) basic information of the citations: dates of the citations, number of citations in articles that were published before and after the retraction, respectively, the number of citations that occurred within one month before or after the SR was retracted, first author, the WHO region and affiliation of the first author, journal impact factor (the Journal Citation Reports 2018) and rank (determined by comparing the impact factor of a journal within the Journal Citation Reports category [
      Web of science. Journal citation Reports: learn the basics.
      ]), and the field (medical versus non-medical) and study type of each citing article; and (II) format of citations: whether the retracted SR together with the retraction notice, whether the SR was labeled as retracted in the citation, whether the reason of retraction was reported, section of the article where the SR was cited and interpretation of the retraction of the SR. We categorized the citations into positive, negative, and neutral according to the definition by Barilan and Halevi [
      • BarIlan J.
      • Halevi G.
      Post retraction citations in context: a case study.
      ]. Positive citation means the findings of the cited articles were used to corroborate the citing article; negative means the authors mentioned the retracted article but disagreed with its results; and neutral means the cited article was mentioned only as a publication that appears in the literature without any judgment on its validity. Any discrepancies were settled by discussion.

      2.3 Data analysis

      Two researchers (Zijun Wang and Siya Zhao) collated and analyzed the data using Microsoft Excel 2019. For continuous variables, we abstracted the mean, standard deviation (SD), median and range (minimum to maximum) or interquartile range (IQR); for dichotomous variables, we abstracted frequency and percentage. Scatter plot was used to visualize the time between retraction and citation. We also described and how the citation was used to support the citing article, and how the retraction status was addressed.

      3. Results

      3.1 Included studies and basic information

      Six of the 159 retracted SRs identified in our previous study were subsequently republished and thus excluded from this analysis. Our initial search revealed 1,181 citations of the remaining 153 SRs. After application of the exclusion criteria, 1,066 citations (112 retraction notices and 954 articles) were included (Fig. 1). Twenty-five of the 153 retracted SRs identified in our previous study were not cited and thus excluded from this analysis.
      Figure thumbnail gr1
      Fig. 1Flow diagram of the literature search. ∗SRs: Systematic reviews.
      The number of citations of retracted SRs increased over time following the trend of the SR retractions with a gap of 2-3 years, reaching the peak in 2016 (n = 189, 19.8%) (Fig. 2). The median time between the original publication of the retracted SR and its subsequent retraction is 15 months. Most of the retracted SRs (n = 107, 83.6%) and almost half of the citations (n = 399, 41.8%) were published in the Western Pacific region (WPR). All included SR except one were cited less than 30 times. The impact factors of the journals where the retracted SRs were cited ranged from 0.090 to 70.67. Almost all (n = 942, 98.7%) citations were in articles related to medicine. A total of 206 citations (21.6%) were in narrative reviews, and 205 (21.5%) in SRs (Fig. 3). Most of the citations (n = 580, 60.8%) were published after the retraction notice. Twenty-nine (5.0%) citations marked the retraction of the SR in the reference list, although the remaining articles (n = 551, 95.0%) did not indicate in any way that the cited SR was retracted (Supplement 1).
      Figure thumbnail gr2
      Fig. 2Publication years of retracted systematic reviews with their citations.
      Figure thumbnail gr3
      Fig. 3Study types of articles citing retracted systematic reviews.

      3.2 Frequencies of citations of retracted SRs

      The median number of citations per SR was 5.5 (IQR 2-9), with the highest number of citations of a single SR being 69 [
      • Cremonini F.
      • Di Caro S.
      • Delgado-Aros S.
      • Sepulveda A.
      • Gasbarrini G.
      • Gasbarrini A.
      • et al.
      Meta-analysis: the relationship between Helicobacter pylori infection and gastro-oesophageal reflux disease [retraction of: Cremonini F, Di Caro S, Delgado-Aros S, Sepulveda A, Gasbarrini G, Gasbarrini A, Camilleri M. Aliment Pharmacol Ther. 2003 Aug 1;18(3):279-89].
      ]. Seventy-eight (60.9%) and 109 (85.2%) SRs were cited before and after the retraction notice was published, respectively. The mean number of citations per SR in articles published after retraction was higher than before retraction (4.5 ± 7.3 vs. 2.6 ± 4.0). SRs led by authors from universities were cited more frequently (9.4 ± 8.8) than SRs led by authors with other types of affiliations. The mean number of citations per SR was higher for SRs led by authors from the WHO Region of the Americas (20.9 ± 21.4) than from other regions. Although the clear majority of the retracted SRs were published in the WPR, these SRs were not as frequently cited (6.2 ± 5.1) as those from most other regions. The number of citations per SR was also higher after than before retraction among SRs led by authors from WPR. The number of citations per SR was highest for SRs published in the highest-ranking quartile of journals (12.6 ± 15.2), lowest in the lowest-ranking quartile (5.1 ± 3.1) (Table 1).
      Table 1Number of citations of retracted SRs stratified by selected variables
      CategoryTotalCitations before the retractionCitations after the retraction
      Mean (SD)/NumberMedian (range)Mean (SD)/NumberMedian (range)Mean (SD)/NumberMedian (range)
      Publication year
       2003∼200535.5 (47.4)35.5 (2-69)0NA (NA)35.5 (47.4)35.5 (2-69)
       2006∼201011.8 (9.4)10 (2-27)3.4 (4.2)2 (0-10)8.4 (11.7)2 (0-27)
       2011∼20157.5 (6.5)6 (1-29)3.0 (4.5)1 (0-23)4.1 (4.2)3 (0-22)
       2016∼20194.9 (4.7)4 (1-21)1.2 (1.9)0 (0-7)3.1 (3.2)2 (0-12)
      Affiliation of the first author
       Hospital7.6 (8.5)6 (1-69)2.1 (3.2)1 (0-20)4.2 (7.5)2 (0-69)
       University9.4 (8.8)6.5 (1-29)4.0 (5.7)2 (0-23)5.0 (6.2)2.5 (0-27)
       Unsure2626 (NA)44 (NA)2222 (NA)
      WHO region of the first author's main affiliation (sort by WHO region)
       WPR6.2 (5.1)5 (1-29)2.3 (3.2)1 (0-20)3.6 (3.5)2 (0-20)
       EUR12.0 (11.6)9.5 (1-29)4.3 (6.7)2 (0-16)7.2 (8.7)4 (0-27)
       AMR20.9 (21.4)17 (2-69)5.0 (8.2)1.5 (0-23)15.4 (23.0)4 (2-69)
       EMR4.3 (4.9)2 (1-10)0.3 (0.6)0 (0-1)3.7 (4.7)2 (0-9)
       AFR11 (NA)0NA (NA)0NA (NA)
      Journal rank
       Q112.6 (15.2)6 (1-69)2.2 (2.9)1 (0-11)9.7 (15.1)4 (0-69)
       Q25.4 (4.5)4 (1-23)1.8 (3.3)1 (0-16)3.3 (2.9)2 (0-10)
       Q36.4 (6.5)4 (1-29)2.4 (4.7)1 (0-23)3.7 (3.3)2 (0-12)
       Q45.1 (3.1)5 (1-10)2.1 (2.7)0 (0-8)2.8 (1.5)2 (1-5)
       Not cited by SCI7.8 (5.7)7 (1-25)4.1 (4.8)3 (0-20)3.4 (3.5)2 (0-14)
      Total7.5 (8.4)5.5 (1-69)2.6 (4.0)1 (0-23)4.5 (7.3)2 (0-69)
      Abbreviations: SD, standard deviation; WPR, Western pacific region; EUR, European region; AMR, region of the Americas; EMR, Eastern Mediterranean region; AFR, African region; NA, not applicable; Q, quartile within the journal category; SCI, science citation index.
      Unsure: The author's institution could not be identified.

      3.3 The retraction reasons of SRs and the number of citations

      The mean number of citations of the 57 SRs retracted due to fraudulent peer review was 3.1 ± 2.5 after retraction, and 2.9 ± 3.8 before retraction (Table 2).
      Table 2Number of citations to retracted SRs by different retraction reasons
      Retraction reasonsCitation before retractionCitation after retraction
      Mean (SD)Median (range)Mean (SD)Median (range)
      Fraudulent peer-review2.9 (3.8)2 (0-20)3.1 (2.5)2 (0-10)
      Unreliable data2.3 (4.3)1 (0-23)4.7 (5.8)2 (0-22)
      Plagiarism3.1 (4.0)1.5 (0-16)5.2 (6.0)4 (0-27)
      Duplicate publication2 (3.4)0 (0-8)3.67 (3.9)2 (0-11)
      Authorship issues4.3 (6.3)1 (0-16)3.5 (2.5)3.5 (0-6)
      Abbreviations: SD, standard deviation.

      3.4 Time between retraction and citation

      One hundred and nine SRs were cited 580 times after the retraction notice was published. The median time from retraction to citation among these citations was 16 (IQR 6.8-33) months, and the frequency of citations decreased over time since retraction (Fig. 4). Two of the retracted SRs were still cited ten or more years after retraction. One of these two SRs was retracted because of duplicate publication, whereas the reason for the retraction of the other one was not mentioned in the retraction notice. Both SRs were cited positively.
      Figure thumbnail gr4
      Fig. 4Time between retraction and citation among citations published after the retraction of the cited systematic review.

      3.5 Characteristics of citations that declared the retraction status

      Twenty-nine citations of a total of 10 retracted SRs were labeled with “Retracted” or “Retraction” in the reference list. Seven articles cited both the retracted article and the corresponding retraction notice. Eighteen citations mentioned the retraction in the text, 15 also mentioned the reasons for retraction. The SR of DiNicolantonio et al. [
      • DiNicolantonio J.J.
      • Di Pasquale P.
      • Taylor R.S.
      • Hackam D.G.
      Low sodium versus normal sodium diets in systolic heart failure: systematic review and meta-analysis [retracted in: Heart. 2013 Jun;99(11):820].
      ] was the most frequently cited with 12 citations. One citation also explained the retraction reason and the general situation of the research, in addition to the “retraction” label. In some citations, a dot was added after the reference number in the original SR to distinguish it, that is, [
      Web of science. Journal citation Reports: learn the basics.
      ,
      • Neal B.
      • Land M.A.
      • Woodward M.
      An update on the salt wars-genuine controversy, poor science, or vested interest?.
      ].
      Among the 29 citations, nine cited the retracted SRs positively, five neutrally and 15 negatively. The most common types of studies among the citing articles were narrative reviews (n = 11) and SRs (n = 7). SRs cited the retracted articles both positively (n = 3) and negatively (n = 4), seven narrative reviews cited the SRs negatively. Eighteen citations, among which eight were negative, cited the SRs in the discussion (Table 3).
      Table 3The reason of retracting the cited systematic review, study type, place of citation, and reason of citation among citations labeled as retracted
      Due to the small sample size, we didn't use percentages to avoid misunderstanding.
      CategoryPositive (n = 9)Neutral (n = 5)Negative (n = 15)Total
      Retraction reason
       Plagiarism4105
       Unreliable data141217
       Fraudulent peer-review3025
       Authorship issues1012
      Study type
       Cross-sectional study3014
       Randomized controlled trial1113
       Systematic review3047
       Narrative review22711
       Non-randomized study of intervention(s)0101
       Editorial material0112
       Cohort study0011
      Section of the citation
       Background4048
       Results0033
       Discussion55818
      a Due to the small sample size, we didn't use percentages to avoid misunderstanding.
      All nine positive citations were used to support or corroborate the citing article's findings (Table 4). Most neutral citations used the retracted SRs to present the current research status without a judgment of the validity of the reference; one study addressed the rationality of the reasons for retraction. Negative citations denied the result of the retracted SRs, the reasons for the citation being either the unreliable result, the lack of high-quality evidence in these research area, or fraudulent peer-review. It is worth noting that one study claimed that a retracted SR was republished, so they included this study and explained the results cautiously [
      • Grover S.
      • Sahoo S.
      • Rabha A.
      • Koirala R.
      ECT in schizophrenia: a review of the evidence.
      ], but in reality, the “republished SR” was published earlier than the original SR and the included studies were different. The reason for the citation may have thus been based on a misunderstanding.
      Table 4The classification of the citation reasons
      Category of citationsCitation reasonNumberExample
      PositiveSupport or corroborate the citation's findings9Han et al. 2017 [
      • Han B.
      • Li H.
      • Ma Q.
      Clinical therapeutic strategy of recombinant human brain natriuretic peptide and dopamine in cardiorenal syndrome type 4 patients combined with hypotension.
      ]
      NeutralPresent the current research status4Colin et al. 2015 [
      • Colin-Ramirez E.
      • McAlister F.A.
      • Zheng Y.
      • Sharma S.
      • Armstrong P.W.
      • Ezekowitz J.A.
      The long-term effects of dietary sodium restriction on clinical outcomes in patients with heart failure. The SODIUM-HF (Study of Dietary Intervention under 100 mmol in Heart Failure): a pilot study.
      ]
      Analyze the rationality of the retraction reasons1Satel et al. 2019 [
      • Satel S.
      How much do we need to know about supervised consumption sites? It depends who's asking.
      ]
      NegativeUnreliable result9Zhou et al. 2014 [
      • Zhou D.
      • Yu H.
      • He F.
      • Reilly K.H.
      • Zhang J.
      • Li S.
      • et al.
      Nut consumption in relation to cardiovascular disease risk and type 2 diabetes: a systematic review and meta-analysis of prospective studies.
      ]
      Lack of high-quality evidence4He et al. 2014 [
      • He F.J.
      • MacGregor G.A.
      Salt and sugar: their effects on blood pressure.
      ]
      Fraudulent peer-review2Woodcock et al. 2017 [
      • Woodcock B.G.
      Peer review in clinical pharmacology using the 8-D assessment.
      ]

      4. Discussion

      Our study analyzed 954 citations of 128 retracted SRs. The numbers of retracted SR and their citations reached their peaks in 2014 and 2016, respectively, and the majority of the citations were published after retraction. In particular SRs retracted because of unreliable data were cited more frequently after than before retraction. Although the citations became less frequent over time since retraction, retracted studies were still being cited even years after the retraction. Only few citations declared the retraction of the cited SR in the reference section. Most citations that mentioned the retraction status addressed the problems of the evidence, but some of the citations were used to re-confirm the results of the retracted SRs.
      Citations before the time of retraction are likely to refer to the original content of the SRs. This cannot be avoided. However, our study shows that most citations appeared after the retraction notice was published. These makes us suspect whether the retraction of the SR was known to the citing article's authors. There are two possible reasons for this. First, the retraction status of the SRs may not have been shown on the website the authors used to retrieve the article. COPE states that retractions should be clearly identifiable in all electronic versions of the manuscript, for example by tagging tag such as “retraction” or “retracted” appearing on the page, with a link to the retraction notice [
      • Wager E.
      • Barbour V.
      • Yentis S.
      • Kleinert S.
      Retractions: guidance from the committee on publication Ethics (COPE).
      ]. But journals and databases do not always update the status of articles on time. Only about 60% of the 129 retracted SRs were correctly labeled by PubMed [
      • Shi Q.
      • Wang Z.
      • Zhou Q.
      • Hou R.
      • Gao X.
      • He S.
      • et al.
      More consideration is needed for retracted non-cochrane systematic reviews in medicine: a systematic review.
      ]. According to Reuben et al. [
      • Reuben S.S.
      • Buvenandran A.
      • Katz B.
      • Kroin J.S.
      A prospective randomized trial on the role of perioperative celecoxib administration for total knee arthroplasty: improving clinical outcomes.
      ], most retracted articles were labeled in EMBASE as retracted 3 to 6 months after the retraction. Wright et al. [
      • Wright K.
      • McDaid C.
      Reporting of article retractions in bibliographic databases and online journals.
      ] analyzed nine journals with retracted articles, and found out that only three journals labeled the retraction on the PDF versions. It is also possible that the cited study was retracted between the submission and acceptance of the citing article. Although it would be good practice to confirm the validity of references at the page proofing stage, authors may be reluctant to do so in order not to cause delays for the publication of their own article.
      Another possible explanation is “secondhand” citations of SRs that were cited elsewhere before the retraction. “Secondhand” citations mean citing an article that has previously been cited in another paper, without having accessed the original [
      • Candal-Pedreira C.
      • Ruano-Ravina A.
      • Fernández E.
      • Ramos J.
      • Campos-Varela I.
      • Pérez-Ríos M.
      Does retraction after misconduct have an impact on citations? A pre-post study.
      ]. As about 70-90% of citations in scientific articles are copied from the lists of references used in other papers [
      • Simkin M.V.
      • Roychowdhury V.P.
      Stochastic modeling of citation slips.
      ], many authors may not be aware that they are citing retracted SRs, and potentially false information will continue to be spread. This reason may also lead a citation of a retracted article long after its retraction. We found retracted SRs that were cited more than 10 years after the retraction, without mentioning the retraction status in the text.
      Many other factors also affect the number of citations of SRs and MAs. Retracted SRs published in journals with higher impact factors were cited more often, which has been observed also for other types of studies [
      • Bolboacă S.D.
      • Buhai D.V.
      • Aluaș M.
      • Bulboacă A.E.
      Post retraction citations among manuscripts reporting a radiology-imaging diagnostic method.
      ,
      • Nath S.B.
      • Marcus S.C.
      • Druss B.G.
      Retractions in the research literature: misconduct or mistakes?.
      ,
      • Candal-Pedreira C.
      • Ruano-Ravina A.
      • Fernández E.
      • Ramos J.
      • Campos-Varela I.
      • Pérez-Ríos M.
      Does retraction after misconduct have an impact on citations? A pre-post study.
      ]. A possible reason for our results could also be self-citation by authors and journals. To self-promote and increase the impact factor, some journal may have a high self-citation rate [
      • Sanfilippo F.
      • Tigano S.
      • Morgana A.
      • Murabito P.
      • Astuto M.
      Self-citation policies and journal self-citation rate among Critical Care Medicine journals.
      ], especially among journals with low impact factors [
      Journal self-citation in the journal citation Reports – science edition.
      ]. Some authors may take the initiative to cite studies published in the same journal to improve the chance of being accepted, and retracted articles may be cited by mistake. Some authors whose article has been retracted may themselves continue to cite it without mentioning the retraction status [
      • Candal-Pedreira C.
      • Ruano-Ravina A.
      • Fernández E.
      • Ramos J.
      • Campos-Varela I.
      • Pérez-Ríos M.
      Does retraction after misconduct have an impact on citations? A pre-post study.
      ,
      • Madlock-Brown C.R.
      • Eichmann D.
      The (lack of) impact of retraction on citation networks.
      ].
      Retraction does not mean an end to citing an article. Although nine citations confirmed the results of the retracted SRs in our study, the risks related to the conclusion depend on the reasons of retraction [
      • Shi Q.
      • Wang Z.
      • Zhou Q.
      • Hou R.
      • Gao X.
      • He S.
      • et al.
      More consideration is needed for retracted non-cochrane systematic reviews in medicine: a systematic review.
      ,
      • Faggion Jr., C.M.
      More detailed guidance on the inclusion/exclusion of retracted articles in systematic reviews is needed.
      ]. For example, if an article is retracted due to authorship issues, it will most likely not influence the result or conclusion and affect the quality of the citation [
      • Huh S.
      • Kim S.Y.
      • Cho H.M.
      Characteristics of retractions from Korean medical journals in the KoreaMed database: a bibliometric analysis.
      ]. One citation we identified [
      • Kumar K.
      • Kumar A.
      • Keegan R.M.
      • Deshmukh R.
      Recent advances in the neurobiology and neuropharmacology of Alzheimer's disease.
      ] confirmed the results of a retracted SR, however, the citation referred to a secondary finding of the SR on the types of adverse events of antipsychotic drugs, and did not cite the results of the meta-analysis. In this case, the result of the SR which was retracted due to unreliable data does not affect the authenticity of this citation. Therefore, a retracted article can still, despite the false results, influence future research in a positive way if correctly cited. As for the retraction, what should be done is to stop the spread of wrong information and cite them standardly [
      • Candal-Pedreira C.
      • Ruano-Ravina A.
      • Fernández E.
      • Ramos J.
      • Campos-Varela I.
      • Pérez-Ríos M.
      Does retraction after misconduct have an impact on citations? A pre-post study.
      ].
      To identify and correctly cite retracted SRs, we suggest the following actions to be taken by different stakeholders:
      • 1)
        Authors should maintain academic rigor when citing SRs, and confirm the validity of all references before publication [
        • Cosentino A.M.
        • Veríssimo D.
        Ending the citation of retracted papers.
        ,
        International Committee of Medical Journal Editors
        Manuscript preparation: preparing for submission.
        ]. If retracted SRs must be cited, the retraction status should be marked and the reason of citation should be explained clearly.
      • 2)
        Periodical journals should establish an appropriate identification and management mechanism for retractions. This should including updating the publication status timely, adding the tag “Retraction” or “Retracted” in the title and on the PDF of original articles [
        • Wager E.
        • Barbour V.
        • Yentis S.
        • Kleinert S.
        Retractions: guidance from the committee on publication Ethics (COPE).
        ], and add the link of retraction notice onto the online version.
      • 3)
        Retraction index databases and bibliographic index database should be improved. In view of the problems of incomplete retraction information and delayed update in the Retraction Watch Database (RWdb) [
        • Brainard J.
        Rethinking retractions.
        ] (the world's largest paper retraction database at present), RWdb should speed up its updating, and preferably use real-time monitoring of the retraction of articles (of course, it cannot be denied that RWdb provides a valuable service with limited resources). The same applies to databases that are frequently used by researchers to search literature and may also have delays in correctly labeling the retracted articles [
        • Wright K.
        • McDaid C.
        Reporting of article retractions in bibliographic databases and online journals.
        ].
      • 4)
        Guidelines for retraction should be developed. There are only few handbooks that guide how to cite retracted articles. The International Committee of Medical Journal Editors (ICMJE) [
        International Committee of Medical Journal Editors
        Manuscript preparation: preparing for submission.
        ] has issued instructions using examples on how to cite retracted papers, but their use is not widely enforced by journals. A clear and unified guideline could help authors to standardize and accurately cite retracted articles, which is convenient for editors to review and avoid misunderstanding by readers.

      4.1 Strengths and limitations

      Our study has several advantages. First, the topic of this study is novel, and to our knowledge, it is the first paper to analyze the citation of retracted SRs. Second, our study shows that the citation of retracted SRs is not necessarily wrong. We also give several suggestions on how to cite retractions which will provide guidance for authors to cite the retracted articles. Our study also has some limitations. We only searched Google Scholar and Web of Science to identify the citations limited by the integrity of the database, and some citations may thus have been missed.

      5. Conclusions

      Systematic reviews continue to be cited after being retracted, even after the retraction notice has been published. The failure of the journal to mark the retraction on time and the author's intentional or unintentional citation of retracted SRs without mentioning the retraction status may lead to the retracted SRs' being cited even more after the retraction than before. An inappropriate citation of retracted SRs can thus enhance the spread of misinformation. We suggest to develop guidance and form a standardized approach to the management of retractions to avoid misquotation and, improve the scientific research rigor of citing existing literature.

      CRediT authorship contribution statement

      Zijun Wang: Methodology, Validation, Formal analysis, Investigation, Data curation, Visualization, Writing – original draft. Qianling Shi: Investigation, Data curation, Writing – review & editing. Qi Zhou: Methodology, Investigation, Writing – review & editing. Siya Zhao: Formal analysis, Visualization. Ruizhen Hou: Investigation. Shuya Lu: Investigation. Xia Gao: Investigation. Yaolong Chen: Conceptualization, Supervision.

      Acknowledgments

      We thank Janne Estill, Institute of Global Health of University of Geneva, and Meng Lv, Department of Nephrology, Children's Hospital of Chongqing Medical University, for providing guidance and comments for our manuscript when revising. We gratefully acknowledge the assistance of Janne Estill for English language editing.

      References

        • Bolboacă S.D.
        • Buhai D.V.
        • Aluaș M.
        • Bulboacă A.E.
        Post retraction citations among manuscripts reporting a radiology-imaging diagnostic method.
        PLoS One. 2019; 14: e0217918
        • Cosentino A.M.
        • Veríssimo D.
        Ending the citation of retracted papers.
        Conserv Biol. 2016; 30: 676-678
        • Nath S.B.
        • Marcus S.C.
        • Druss B.G.
        Retractions in the research literature: misconduct or mistakes?.
        Med J Aust. 2006; 185: 152-154
        • Grieneisen M.L.
        • Zhang M.
        A comprehensive survey of retracted articles from the scholarly literature.
        PLoS One. 2012; 7: e44118
        • Wager E.
        • Barbour V.
        • Yentis S.
        • Kleinert S.
        Retractions: guidance from the committee on publication Ethics (COPE).
        Croat Med J. 2009; 50: 532-535
        • Nair S.
        • Yean C.
        • Yoo J.
        • Leff J.
        • Delphin E.
        • Adams D.C.
        Reasons for article retraction in anesthesiology: a comprehensive analysis.
        Can J Anaesth. 2020; 67: 57-63
        • Hamilton D.G.
        Continued citation of retracted radiation oncology literature-do we have a problem?.
        Int J Radiat Oncol Biol Phys. 2019; 103: 1036-1042
        • Suelzer E.M.
        • Deal J.
        • Hanus K.L.
        • Ruggeri B.
        • Sieracki R.
        • Witkowski E.
        Assessment of Citations of the Retracted Article by Wakefield et al. With Fraudulent Claims of an Association Between Vaccination and Autism.
        JAMA Netw Open. 2019; 2: e1915552
        • Chen C.M.
        • Hu Z.G.
        • Milbank J.
        • Schultz T.
        A visual analytic study of retracted articles in scientific literature.
        J Am Soc Inf Sci Technol. 2013; 64: 234-253
        • Chen C.M.
        • Leydesdorff L.
        Patterns of connections and movements in dual-map overlays: a new method of publication portfolio analysis.
        J Am Soc Inf Sci Technol. 2014; 65: 334-351
      1. Web of science. Search date December 22, 2021.
        (Available at)
        • Sutton A.
        • Clowes M.
        • Preston L.
        • Booth A.
        Meeting the review family: exploring review types and associated information retrieval requirements.
        Health Info Libr J. 2019; 36: 202-222
        • Liberati A.
        • Altman D.G.
        • Tetzlaff J.
        • Mulrow C.
        • Gøtzsche P.C.
        • Ioannidis J.P.A.
        • et al.
        The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration.
        BMJ. 2009; 339: b2700
        • Siddaway A.P.
        • Wood A.M.
        • Hedges L.V.
        How to do a systematic review: a best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses.
        Annu Rev Psychol. 2019; 70: 747-770
        • Gaudino M.
        • Robinson N.B.
        • Audisio K.
        • Rahouma M.
        • Benedetto U.
        • Kurlansky P.
        • et al.
        Trends and characteristics of retracted articles in the biomedical literature, 1971 to 2020.
        JAMA Intern Med. 2021; 181: 1118-1121
        • Shi Q.
        • Wang Z.
        • Zhou Q.
        • Hou R.
        • Gao X.
        • He S.
        • et al.
        More consideration is needed for retracted non-cochrane systematic reviews in medicine: a systematic review.
        J Clin Epidemiol. 2021; 139: 57-67
      2. Web of science. Journal citation Reports: learn the basics.
        (Available at)
        https://clarivate.libguides.com/jcr
        Date accessed: December 22, 2021
        • BarIlan J.
        • Halevi G.
        Post retraction citations in context: a case study.
        Scientometrics. 2017; 113: 547-565
        • Cremonini F.
        • Di Caro S.
        • Delgado-Aros S.
        • Sepulveda A.
        • Gasbarrini G.
        • Gasbarrini A.
        • et al.
        Meta-analysis: the relationship between Helicobacter pylori infection and gastro-oesophageal reflux disease [retraction of: Cremonini F, Di Caro S, Delgado-Aros S, Sepulveda A, Gasbarrini G, Gasbarrini A, Camilleri M. Aliment Pharmacol Ther. 2003 Aug 1;18(3):279-89].
        Aliment Pharmacol Ther. 2004; 19 (145): 145
        • DiNicolantonio J.J.
        • Di Pasquale P.
        • Taylor R.S.
        • Hackam D.G.
        Low sodium versus normal sodium diets in systolic heart failure: systematic review and meta-analysis [retracted in: Heart. 2013 Jun;99(11):820].
        Heart. 2013; 10
        • Neal B.
        • Land M.A.
        • Woodward M.
        An update on the salt wars-genuine controversy, poor science, or vested interest?.
        Curr Hypertens Rep. 2013; 15: 687-693
        • Grover S.
        • Sahoo S.
        • Rabha A.
        • Koirala R.
        ECT in schizophrenia: a review of the evidence.
        Acta Neuropsychiatr. 2019; 31: 115-127
        • Han B.
        • Li H.
        • Ma Q.
        Clinical therapeutic strategy of recombinant human brain natriuretic peptide and dopamine in cardiorenal syndrome type 4 patients combined with hypotension.
        Pak J Pharm Sci. 2017; 30: 1449-1453
        • Colin-Ramirez E.
        • McAlister F.A.
        • Zheng Y.
        • Sharma S.
        • Armstrong P.W.
        • Ezekowitz J.A.
        The long-term effects of dietary sodium restriction on clinical outcomes in patients with heart failure. The SODIUM-HF (Study of Dietary Intervention under 100 mmol in Heart Failure): a pilot study.
        Am Heart J. 2015; 169: 274-281
        • Satel S.
        How much do we need to know about supervised consumption sites? It depends who's asking.
        Addiction. 2019; 114: 2116-2117
        • Zhou D.
        • Yu H.
        • He F.
        • Reilly K.H.
        • Zhang J.
        • Li S.
        • et al.
        Nut consumption in relation to cardiovascular disease risk and type 2 diabetes: a systematic review and meta-analysis of prospective studies.
        Am J Clin Nutr. 2014; 100: 270-277
        • He F.J.
        • MacGregor G.A.
        Salt and sugar: their effects on blood pressure.
        Pflugers Arch. 2015; 467: 577-586
        • Woodcock B.G.
        Peer review in clinical pharmacology using the 8-D assessment.
        Int J Clin Pharmacol Ther. 2017; 55: 200-201
        • Reuben S.S.
        • Buvenandran A.
        • Katz B.
        • Kroin J.S.
        A prospective randomized trial on the role of perioperative celecoxib administration for total knee arthroplasty: improving clinical outcomes.
        Anesth Analg. 2008; 106: 1258-1264
        • Wright K.
        • McDaid C.
        Reporting of article retractions in bibliographic databases and online journals.
        J Med Libr Assoc. 2011; 99: 164-167
        • Candal-Pedreira C.
        • Ruano-Ravina A.
        • Fernández E.
        • Ramos J.
        • Campos-Varela I.
        • Pérez-Ríos M.
        Does retraction after misconduct have an impact on citations? A pre-post study.
        BMJ Glob Health. 2020; 5: e003719
        • Simkin M.V.
        • Roychowdhury V.P.
        Stochastic modeling of citation slips.
        Scientometrics. 2005; 62: 367-384
        • Sanfilippo F.
        • Tigano S.
        • Morgana A.
        • Murabito P.
        • Astuto M.
        Self-citation policies and journal self-citation rate among Critical Care Medicine journals.
        J Intensive Care. 2021; 9: 15
      3. Journal self-citation in the journal citation Reports – science edition.
        (Available at)
        • Madlock-Brown C.R.
        • Eichmann D.
        The (lack of) impact of retraction on citation networks.
        Sci Eng Ethics. 2015; 21: 127-137
        • Faggion Jr., C.M.
        More detailed guidance on the inclusion/exclusion of retracted articles in systematic reviews is needed.
        J Clin Epidemiol. 2019; 116: 133-134
        • Huh S.
        • Kim S.Y.
        • Cho H.M.
        Characteristics of retractions from Korean medical journals in the KoreaMed database: a bibliometric analysis.
        PLoS One. 2016; 11: e0163588
        • Kumar K.
        • Kumar A.
        • Keegan R.M.
        • Deshmukh R.
        Recent advances in the neurobiology and neuropharmacology of Alzheimer's disease.
        Biomed Pharmacother. 2018; 98: 297-307
        • International Committee of Medical Journal Editors
        Manuscript preparation: preparing for submission.
        (Available at)
        • Brainard J.
        Rethinking retractions.
        Science. 2018; 362: 390-393