Advertisement

Restricting evidence syntheses of interventions to English-language publications is a viable methodological shortcut for most medical topics: a systematic review

Open AccessPublished:April 29, 2021DOI:https://doi.org/10.1016/j.jclinepi.2021.04.012

      Abstract

      Objectives

      To assess the impact of restricting systematic reviews of conventional or alternative medical treatments or diagnostic tests to English-language publications.

      Study design and setting

      We systematically searched MEDLINE (Ovid), the Science Citation Index Expanded (Web of Science), and Current Contents Connect (Web of Science) up to April 24, 2020. Eligible methods studies assessed the impact of restricting systematic reviews to English-language publications on effect estimates and conclusions. Two reviewers independently screened the literature; one investigator performed the data extraction, a second investigator checked for completeness and accuracy. We synthesized the findings narratively.

      Results

      Eight methods studies (10 publications) met the inclusion criteria; none addressed language restrictions in diagnostic test accuracy reviews. The included studies analyzed nine to 147 meta-analyses and/or systematic reviews. The proportions of non-English-language publications ranged from 2% to 100%. Based on five methods studies, restricting literature searches or inclusion criteria to English-language publications led to a change in statistical significance in 23/259 meta-analyses (9%). Most commonly, the statistical significance was lost, but had no impact on the conclusions of systematic reviews.

      Conclusion

      Restricting systematic reviews to English-language publications appears to have little impact on the effect estimates and conclusions of systematic reviews.

      Keywords

      What is new?
      What does this add to what is already known?
      • While previous systematic reviews summarized the evidence from methods studies that assessed the impact of language restrictions in systematic reviews of conventional medicine, our systematic review evaluated a diverse range of medical topics, including conventional as well as complementary and alternative medicine.
      • We identified four new methods studies that were not part of previous reviews.
      • Our findings confirm the results of previous systematic reviews but extend the knowledge to complementary and alternative medicine.
      Key findings
      • Restricting the search strategies and/or inclusion criteria of systematic reviews to English-language publications had little impact on the effect estimates and conclusions for most medical topics.
      • Complementary and alternative medicine reviews focusing solely on English-language studies could lead to different results and conclusions than language-inclusive reviews in this topic area.
      • We could not identify any study assessing the impact of English-language restrictions on diagnostic test accuracy reviews.
      What is the implication; what should change now?
      • Restricting search strategies and/or inclusion criteria to English-language publications seems to be a viable approach for rapid reviews and systematic reviews with limited resources. Evidence syntheses addressing complementary and alternative medicine topics should also include publications in languages other than English.

      1. Introduction

      Systematic reviews aim to identify all available (published and unpublished) studies that address a specific research question. Due to their use of systematic and transparent methods to reduce bias, systematic reviews are considered the most reliable form of evidence syntheses. [
      • Moher D
      • Tetzlaff J
      • Tricco AC
      • Sampson M
      • Altman DG.
      Epidemiology and reporting characteristics of systematic reviews.
      ] Because the majority of peer-reviewed scientific manuscripts are published in English, investigators conducting systematic reviews often impose language restrictions on literature searches or study inclusion criteria. [
      • Rosselli D.
      The language of biomedical sciences.
      ] Typically, authors limit included studies to those published in English and in languages spoken by the author teams, most commonly French, German, Italian, and Spanish. [
      • Moher D
      • Fortin P
      • Jadad AR
      • Juni P
      • Klassen T
      • Le Lorier J
      • et al.
      Completeness of reporting of trials published in languages other than English: implications for conduct and reporting of systematic reviews.
      ,
      • Moher D
      • Pham B
      • Lawson ML
      • Klassen TP.
      The inclusion of reports of randomised trials published in languages other than English in systematic reviews.
      Such language restrictions provide an option to save time as well as human and financial resources. Furthermore, methods research indicates that randomized controlled trials published in languages other than English often have methodological shortcomings and receive little weight in systematic reviews. [
      • Moher D
      • Fortin P
      • Jadad AR
      • Juni P
      • Klassen T
      • Le Lorier J
      • et al.
      Completeness of reporting of trials published in languages other than English: implications for conduct and reporting of systematic reviews.
      ,
      • Juni P
      • Holenstein F
      • Sterne J
      • Bartlett C
      • Egger M.
      Direction and impact of language bias in meta-analyses of controlled trials: empirical study.
      Nevertheless, the reliance of systematic reviews on data published in English only may introduce language bias. Cochrane defines language bias as a situation where research findings are published in a particular language, depending on the nature and direction of the results. [

      Higgins JPT, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (eds), Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019) 2019.

      ] For example, statistically significant results are more frequently published in English-language journals than results that do not achieve statistical significance. [
      • Egger M
      • Zellweger-Zahner T
      • Schneider M
      • Junker C
      • Lengeler C
      • Antes G.
      Language bias in randomised controlled trials published in English and German.
      ] Conversely, treatment effects reported in non-English publications are often larger than those reported in English publications. [
      • Juni P
      • Holenstein F
      • Sterne J
      • Bartlett C
      • Egger M.
      Direction and impact of language bias in meta-analyses of controlled trials: empirical study.
      ,
      • Egger M
      • Juni P
      • Bartlett C
      • Holenstein F
      • Sterne J.
      How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study.
      • Pham B
      • Klassen TP
      • Lawson ML
      • Moher D.
      Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary.
      Methods research assessing the impact of language restrictions on systematic review findings and conclusions renders contradicting results. In 2012, a systematic review assessed the impact of restricting evidence syntheses to English-language publications. [
      • Morrison A
      • Polisena J
      • Husereau D
      • Moulton K
      • Clark M
      • Fiander M
      • et al.
      The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies.
      ] It concluded that language restrictions did not introduce bias in meta-analyses of conventional medicine interventions. [
      • Morrison A
      • Polisena J
      • Husereau D
      • Moulton K
      • Clark M
      • Fiander M
      • et al.
      The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies.
      ] By contrast, excluding non-English publications from systematic reviews on complementary and alternative medicine topics decreased the magnitude of the intervention effect. [
      • Pham B
      • Klassen TP
      • Lawson ML
      • Moher D.
      Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary.
      ]
      Methods guidance from organizations that conduct systematic reviews, such as Cochrane [

      Higgins JPT, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (eds), Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019) 2019.

      ], the Agency for Healthcare Research and Quality, [

      Quality AfHRa. Methods guide for effectiveness and comparative effectiveness reviews. 2014.

      ] or the Campbell Collaboration, [

      3. CSRCPaGSN. Methodological expectations of Campbell Collaboration intervention reviews: Conduct standards 2019. Available from: https://onlinelibrarywileycom/page/journal/18911803/homepage/author-guidelines. 2019.

      ] do not recommend limiting systematic reviews to English-language publications. Nevertheless, 78% of published non-Cochrane systematic reviews apply language restrictions to their searches and/or study inclusion criteria. [
      • Jackson JL
      • Kuriyama A.
      How often do systematic reviews exclude articles not published in English?.
      ]
      The objective of our review was to systematically assess the impact of language restrictions on systematic review results and conclusions. Specifically, we strove to answer the following research question: Compared with no language restrictions, what is the impact of restricting literature searches and/or study inclusion criteria to English-language publications on the effect estimates, certainty of evidence ratings, and conclusions of systematic reviews?

      2. Materials and methods

      This systematic review was conducted in accordance with Cochrane methodology. [

      Higgins JPT, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (eds), Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019) 2019.

      ] Throughout the manuscript, we followed the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols statement. [
      • Moher D
      • Liberati A
      • Tetzlaff J
      • Altman DG
      • Group P.
      Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
      ] We registered the systematic review protocol in the Open Science Framework (https://osf.io/ksfc4/).

      2.1 Search strategy and criteria

      An experienced information specialist searched the following databases:
      • Ovid MEDLINE(R) ALL 1946 to April 24, 2020,
      • Science Citation Index Expanded (Web of Science), from inception to April 28, 2020,
      • Current Contents Connect (Web of Science), from inception to April 28, 2020,
      • Scientific Resource Center Methods Library up to 2017 (last update).
      The search strategy included controlled vocabulary and free text, first developed for MEDLINE (Ovid) and then adapted for all other electronic databases. The MEDLINE search strategy was peer-reviewed by another information specialist following the Peer Review of Electronic Search Strategies (PRESS) statement. [
      • McGowan J
      • Sampson M
      • Salzwedel DM
      • Cogo E
      • Foerster V
      • Lefebvre C.
      PRESS peer review of electronic search strategies: 2015 guideline statement.
      ] See Appendix 1 for the database search strategies.
      We conducted supplementary searches of relevant conference proceedings (e.g., Cochrane Colloquia, Health Technology Assessment International) and focused web searches (e.g., Google and Google Scholar) to identify additional grey literature.
      Further, we searched the bibliographies and reference lists of selected papers for relevant citations that were missed by our database searches. We performed forward citation tracking (via Web of Science or Scopus) and similar article searches (via PubMed or Scopus) based on key publications up to July 14, 2020. See Appendix 2 for the documentation of the reference-based database searches.

      2.2 Eligibility criteria

      We were interested in any comparative study design on any type of conventional and alternative medical treatment as well as in diagnostic tests in all medical fields assessing the effect of restricting literature searches and/or inclusion criteria to English-language publications. Table 1 presents the a priori–defined eligibility criteria.
      Table 1Eligibility criteria
      InclusionExclusion
      Topics of interest:Methods studies on any type of medical treatment (CM, CAM) or diagnostic tests in any medical fieldOther areas (e.g., education, environmental science), animals, or laboratory studies
      Intervention:Restriction of literature searches and/or restriction of inclusion criteria to English-language publicationsAny other language restrictions
      Comparator:No language restrictionsAny language restrictions
      Outcomes:Primary outcomes:
      • Differences in treatment effect estimates or diagnostic interventions
      • Differences in certainty of evidence ratings
      • Differences in the conclusions of review authors
      Secondary outcomes:
      • Differences in statistical heterogeneity
      • Differences in the numbers of included study participants
      • Differences in the included studies’ risk of bias ratings
      • Number of missed studies
      • Languages of missed studies
      • Origin countries of missed studies
      Studies that did not report any of the listed outcomes
      Publication languages:Any languageNo restrictions
      Study design:Any comparative study design (empirical and simulation studies)Other types of studies
      Abbreviations: CAM, complementary and alternative medicine; CM, conventional medicine.

      2.3 Study selection and data extraction

      After pilot-testing the study selection at the title/abstract level, two investigators independently reviewed each title and abstract in Covidence. [

      Covidence systematic review software VHI, Melbourne, Australia. Available at: www.covidence.org.

      ] Included abstracts were retrieved as full-text publications and reviewed independently by two investigators. Discrepancies were resolved by discussion or by involving a third reviewer. After pilot-testing the data extraction forms, one investigator extracted data from the included studies; a second investigator checked for completeness and accuracy. We extracted relevant information related to study design, methodology, number and characteristics of non-English studies, impact of language restrictions on treatment estimates, certainty of evidence ratings, authors’ conclusions, and statistical heterogeneity. The majority of the studies applied no restriction during the search or screening process and assessed the impact of trials published in languages other than English by retrospectively removing them from meta-analyses. We identified and extracted the number of non-English publications as the number of potentially missed studies if the language restrictions were applied.

      2.4 Data synthesis

      We summarized the results narratively and grouped them by outcomes of interest. We did not identify enough studies with a similar design to be able to conduct meta-analyses.

      2.5 Risk of bias

      Because no validated risk of bias tool is available to critically assess methods studies, we did not rate the studies’ risk of bias. However, we describe the characteristics of each study in detail.

      3. Results

      Literature searches detected 4106 unique citations after deduplication, of which we retrieved 68 as full-text publications. Eight methods studies corresponding to 10 publications met our eligibility criteria. [
      • Moher D
      • Pham B
      • Lawson ML
      • Klassen TP.
      The inclusion of reports of randomised trials published in languages other than English in systematic reviews.
      ,
      • Juni P
      • Holenstein F
      • Sterne J
      • Bartlett C
      • Egger M.
      Direction and impact of language bias in meta-analyses of controlled trials: empirical study.
      ,
      • Egger M
      • Juni P
      • Bartlett C
      • Holenstein F
      • Sterne J.
      How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study.
      • Pham B
      • Klassen TP
      • Lawson ML
      • Moher D.
      Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary.
      ,
      • Nussbaumer-Streit B
      • Klerings I
      • Dobrescu AI
      • Persad E
      • Stevens A
      • Garritty C
      • et al.
      Excluding non-English publications from evidence-syntheses did not change conclusions: a meta-epidemiological study.
      ,
      • Moher D
      • Pham B
      • Klassen TP
      • Schulz KF
      • Berlin JA
      • Jadad AR
      • et al.
      What contributions do languages other than English make on the results of meta-analyses?.
      ,
      • Gregoire G
      • Derderian F
      • Le Lorier J
      Selecting the language of the publications included in a meta-analysis: is there a tower of babel bias?.
      ,
      • Hugues A
      • Di Marco J
      • Bonan I
      • Rode G
      • Cucherat M
      • Gueyffier F.
      Publication language and the estimate of treatment effects of physical therapy on balance and postural control after stroke in meta-analyses of randomised controlled trials.
      ,
      • Hartling L
      • Featherstone R
      • Nuspl M
      • Shave K
      • Dryden DM
      • Vandermeer B.
      Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews.
      All assessed the impact of language restrictions in systematic reviews of medical interventions, none in diagnostic test accuracy reviews. Fig. 1 depicts the literature review flow. Appendix 3 lists the studies excluded at the full-text level and the reasons for exclusion.
      In the following sections, we first summarize the characteristics of the included methods studies. We then present the impact of excluding non-English publications on the treatment effect estimates, certainty of evidence ratings, and conclusions. Finally, we summarize the characteristics of the non-English publications and assess the differences between language-inclusive and language-restrictive reviews in terms of heterogeneity and statistical measures of publication bias.

      3.1 Characteristics of the included studies

      Eight methods studies corresponding to 10 publications determined the impact of language restrictions on the effect estimates of medical treatments. [
      • Moher D
      • Pham B
      • Lawson ML
      • Klassen TP.
      The inclusion of reports of randomised trials published in languages other than English in systematic reviews.
      ,
      • Juni P
      • Holenstein F
      • Sterne J
      • Bartlett C
      • Egger M.
      Direction and impact of language bias in meta-analyses of controlled trials: empirical study.
      ,
      • Egger M
      • Juni P
      • Bartlett C
      • Holenstein F
      • Sterne J.
      How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study.
      • Pham B
      • Klassen TP
      • Lawson ML
      • Moher D.
      Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary.
      ,
      • Nussbaumer-Streit B
      • Klerings I
      • Dobrescu AI
      • Persad E
      • Stevens A
      • Garritty C
      • et al.
      Excluding non-English publications from evidence-syntheses did not change conclusions: a meta-epidemiological study.
      ,
      • Moher D
      • Pham B
      • Klassen TP
      • Schulz KF
      • Berlin JA
      • Jadad AR
      • et al.
      What contributions do languages other than English make on the results of meta-analyses?.
      ,
      • Gregoire G
      • Derderian F
      • Le Lorier J
      Selecting the language of the publications included in a meta-analysis: is there a tower of babel bias?.
      ,
      • Hugues A
      • Di Marco J
      • Bonan I
      • Rode G
      • Cucherat M
      • Gueyffier F.
      Publication language and the estimate of treatment effects of physical therapy on balance and postural control after stroke in meta-analyses of randomised controlled trials.
      ,
      • Hartling L
      • Featherstone R
      • Nuspl M
      • Shave K
      • Dryden DM
      • Vandermeer B.
      Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews.
      Two of these also assessed the effects of language restrictions on systematic review conclusions. [
      • Nussbaumer-Streit B
      • Klerings I
      • Dobrescu AI
      • Persad E
      • Stevens A
      • Garritty C
      • et al.
      Excluding non-English publications from evidence-syntheses did not change conclusions: a meta-epidemiological study.
      ,
      • Hartling L
      • Featherstone R
      • Nuspl M
      • Shave K
      • Dryden DM
      • Vandermeer B.
      Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews.
      The number of included meta-analysis and/or systematic reviews in the methods studies ranged from nine [
      • Hugues A
      • Di Marco J
      • Bonan I
      • Rode G
      • Cucherat M
      • Gueyffier F.
      Publication language and the estimate of treatment effects of physical therapy on balance and postural control after stroke in meta-analyses of randomised controlled trials.
      ] to 147. [
      • Dechartres A
      • Atal I
      • Riveros C
      • Meerpohl J
      • Ravaud P.
      Association between publication characteristics and treatment effect estimates: a meta-epidemiologic study.
      ] Table 2 presents the characteristics of the included studies and the main results. In the case of companion articles, we presented them once with multiple references. [
      • Moher D
      • Pham B
      • Lawson ML
      • Klassen TP.
      The inclusion of reports of randomised trials published in languages other than English in systematic reviews.
      ,
      • Juni P
      • Holenstein F
      • Sterne J
      • Bartlett C
      • Egger M.
      Direction and impact of language bias in meta-analyses of controlled trials: empirical study.
      ,
      • Egger M
      • Juni P
      • Bartlett C
      • Holenstein F
      • Sterne J.
      How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study.
      ,
      • Pham B
      • Klassen TP
      • Lawson ML
      • Moher D.
      Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary.
      Table 2Summary characteristics and results of the included studies
      Author, y, Medical topicsMethodological approach N of included MAs and/or SRsN of Non-English publications/All N of participants in non-English publications/AllImpact on magnitude of effect estimatesImpact on statistical significance
      Comparing language inclusive with language restrictive (without non-English publications) methods.
      /statistical heterogeneity
      Comparing language inclusive with language restrictive (without non-English publications) methods.
      Impact on overall conclusions
      Comparing language inclusive with language restrictive (without non-English publications) methods.
      Dechartres et al., 2018
      • Dechartres A
      • Atal I
      • Riveros C
      • Meerpohl J
      • Ravaud P.
      Association between publication characteristics and treatment effect estimates: a meta-epidemiologic study.
      Obstetrics, respiratory diseases, surgery, psychiatry, cardiology, infectious diseases, neurology, cancer, anesthesia/critical care
      Retrospectively removed non-English publications from existing systematic reviews 147 MAs393/5303 NRNRChanged in 18/147 (12%); in 17 it became nonsignificant/NRNR
      Grégoire et al., 1995
      • Gregoire G
      • Derderian F
      • Le Lorier J
      Selecting the language of the publications included in a meta-analysis: is there a tower of babel bias?.
      NR
      Retrospectively added non-English publications to existing meta-analyses 28 MAs19/NR NR3/4 MAs changed the result.Changed and became nonsignificant in 1/4/NRNR
      Hartling et al., 2017
      • Hartling L
      • Featherstone R
      • Nuspl M
      • Shave K
      • Dryden DM
      • Vandermeer B.
      Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews.
      Acute respiratory infections, infectious diseases, developmental psychosocial and learning problems
      Retrospectively removed non-English publications from existing systematic reviews 129 MAs34/780 3262/34350In 9/15 MAs, the change in the effect estimate was negligible or small, in two moderates, in two large (more than 20%). In two cases, all included studies were non-English; therefore, no effect estimate was available without the non-English publications.Changed and became nonsignificant in 1/15/NRNo changes
      Hugues et al., 2020
      • Hugues A
      • Di Marco J
      • Bonan I
      • Rode G
      • Cucherat M
      • Gueyffier F.
      Publication language and the estimate of treatment effects of physical therapy on balance and postural control after stroke in meta-analyses of randomised controlled trials.
      Postural therapy on balance and postural control after stroke
      Retrospectively removed non-English publications from existing systematic reviews nine MAs13/145 693/5912NRChanged in 1/9 MAs; none became nonsignificant (11%)/NRNR
      Jüni et al., 2002
      • Juni P
      • Holenstein F
      • Sterne J
      • Bartlett C
      • Egger M.
      Direction and impact of language bias in meta-analyses of controlled trials: empirical study.
      Egger et al., 2003
      • Egger M
      • Juni P
      • Bartlett C
      • Holenstein F
      • Sterne J.
      How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study.
      Tobacco addiction, obstetrics and gynecology, psychiatry, cardiology and angiology, infectious diseases, neurology, rheumatology, and orthopedics
      Retrospectively removed non-English publications from existing systematic reviews 50 MAs115/600 NRIn 29/50 MAs, the changes were less than 5%; 5/50 showed more benefits; 16/50 showed fewer benefits.No changes/ NRNR
      Moher et al., 2000
      • Moher D
      • Pham B
      • Klassen TP
      • Schulz KF
      • Berlin JA
      • Jadad AR
      • et al.
      What contributions do languages other than English make on the results of meta-analyses?.
      Infectious disease, circulatory disease, complications in pregnancy and childbirth
      Retrospectively removed non-English publications from existing systematic reviews 18 MAs33/211 NRNRNo changes/NRNR
      Moher et al., 2003
      • Moher D
      • Pham B
      • Lawson ML
      • Klassen TP.
      The inclusion of reports of randomised trials published in languages other than English in systematic reviews.
      Pham et al., 2005
      • Pham B
      • Klassen TP
      • Lawson ML
      • Moher D.
      Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary.
      CM, CAM
      Retrospectively removed non-English publications from existing systematic reviews 42 MAs133/662 17810/120545In CM, no biased results (ROR = 1.02; 95%CI = 0.83 to 1.26); in CAM, the intervention effect was reduced, with 63% on average (ROR = 1.63; 95%CI: 1.03 to 2.60).NR/In 3/42 MAs (1 CM and 2 CAM topics), the degree of heterogeneity decreased from I2 greater than 0.30 to less or equal to 0.3 (no heterogeneity)NR
      Nussbaumer-Streit et al., 2020
      • Nussbaumer-Streit B
      • Klerings I
      • Dobrescu AI
      • Persad E
      • Stevens A
      • Garritty C
      • et al.
      Excluding non-English publications from evidence-syntheses did not change conclusions: a meta-epidemiological study.
      Cardiovascular disease, cerebrovascular disease, chronic respiratory disease, osteoarthritis, mental health
      Retrospectively removed non-English publications from existing systematic reviews 59 SRs31/1281 NRMinimal impact in 40/87 outcomes; in the other 47, no non-English publications were included in the analysis.Changed in 2/59; one became nonsignificant/NRNo changes
      Abbreviations: CAM, complementary and alternative medicine; CM, conventional medicine; MA, meta-analysis; NR, not reported; SR, systematic review.
      low asterisk Comparing language inclusive with language restrictive (without non-English publications) methods.
      Methodologically, all studies employed meta-epidemiological designs; seven studies retrospectively removed non-English publications from the selected systematic reviews; [
      • Moher D
      • Pham B
      • Lawson ML
      • Klassen TP.
      The inclusion of reports of randomised trials published in languages other than English in systematic reviews.
      ,
      • Juni P
      • Holenstein F
      • Sterne J
      • Bartlett C
      • Egger M.
      Direction and impact of language bias in meta-analyses of controlled trials: empirical study.
      ,
      • Egger M
      • Juni P
      • Bartlett C
      • Holenstein F
      • Sterne J.
      How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study.
      • Pham B
      • Klassen TP
      • Lawson ML
      • Moher D.
      Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary.
      ,
      • Nussbaumer-Streit B
      • Klerings I
      • Dobrescu AI
      • Persad E
      • Stevens A
      • Garritty C
      • et al.
      Excluding non-English publications from evidence-syntheses did not change conclusions: a meta-epidemiological study.
      ,
      • Moher D
      • Pham B
      • Klassen TP
      • Schulz KF
      • Berlin JA
      • Jadad AR
      • et al.
      What contributions do languages other than English make on the results of meta-analyses?.
      ,
      • Hugues A
      • Di Marco J
      • Bonan I
      • Rode G
      • Cucherat M
      • Gueyffier F.
      Publication language and the estimate of treatment effects of physical therapy on balance and postural control after stroke in meta-analyses of randomised controlled trials.
      ,
      • Hartling L
      • Featherstone R
      • Nuspl M
      • Shave K
      • Dryden DM
      • Vandermeer B.
      Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews.
      , one study duplicated the computerized search and retrospectively added non-English publications to the original meta-analyses. [
      • Gregoire G
      • Derderian F
      • Le Lorier J
      Selecting the language of the publications included in a meta-analysis: is there a tower of babel bias?.
      ]

      3.2 Impact of language restrictions on treatment effect estimates

      All eight studies [
      • Moher D
      • Pham B
      • Lawson ML
      • Klassen TP.
      The inclusion of reports of randomised trials published in languages other than English in systematic reviews.
      ,
      • Juni P
      • Holenstein F
      • Sterne J
      • Bartlett C
      • Egger M.
      Direction and impact of language bias in meta-analyses of controlled trials: empirical study.
      ,
      • Egger M
      • Juni P
      • Bartlett C
      • Holenstein F
      • Sterne J.
      How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study.
      • Pham B
      • Klassen TP
      • Lawson ML
      • Moher D.
      Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary.
      ,
      • Nussbaumer-Streit B
      • Klerings I
      • Dobrescu AI
      • Persad E
      • Stevens A
      • Garritty C
      • et al.
      Excluding non-English publications from evidence-syntheses did not change conclusions: a meta-epidemiological study.
      ,
      • Moher D
      • Pham B
      • Klassen TP
      • Schulz KF
      • Berlin JA
      • Jadad AR
      • et al.
      What contributions do languages other than English make on the results of meta-analyses?.
      ,
      • Gregoire G
      • Derderian F
      • Le Lorier J
      Selecting the language of the publications included in a meta-analysis: is there a tower of babel bias?.
      ,
      • Hugues A
      • Di Marco J
      • Bonan I
      • Rode G
      • Cucherat M
      • Gueyffier F.
      Publication language and the estimate of treatment effects of physical therapy on balance and postural control after stroke in meta-analyses of randomised controlled trials.
      ,
      • Hartling L
      • Featherstone R
      • Nuspl M
      • Shave K
      • Dryden DM
      • Vandermeer B.
      Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews.
      quantified changes in the treatment effects and statistical significance when language restrictions were applied to the systematic reviews. Overall, restricting systematic reviews to English-language publications had a small impact on the treatment effect estimates and their statistical significance in reviews of conventional medicine topics, but a larger impact on reviews of complementary and alternative medicine topics. Table 2 summarizes the main of results.
      In two methods studies, the statistical significance did not change in any of the meta-analyses after the non-English publications were removed. [
      • Juni P
      • Holenstein F
      • Sterne J
      • Bartlett C
      • Egger M.
      Direction and impact of language bias in meta-analyses of controlled trials: empirical study.
      ,
      • Egger M
      • Juni P
      • Bartlett C
      • Holenstein F
      • Sterne J.
      How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study.
      ,
      • Moher D
      • Pham B
      • Klassen TP
      • Schulz KF
      • Berlin JA
      • Jadad AR
      • et al.
      What contributions do languages other than English make on the results of meta-analyses?.
      Five publications reported changes in the statistical significance results after the exclusion of non-English-language trials. [
      • Dechartres A
      • Atal I
      • Riveros C
      • Meerpohl J
      • Ravaud P.
      Association between publication characteristics and treatment effect estimates: a meta-epidemiologic study.
      ,
      • Nussbaumer-Streit B
      • Klerings I
      • Dobrescu AI
      • Persad E
      • Stevens A
      • Garritty C
      • et al.
      Excluding non-English publications from evidence-syntheses did not change conclusions: a meta-epidemiological study.
      ,
      • Gregoire G
      • Derderian F
      • Le Lorier J
      Selecting the language of the publications included in a meta-analysis: is there a tower of babel bias?.
      ,
      • Hugues A
      • Di Marco J
      • Bonan I
      • Rode G
      • Cucherat M
      • Gueyffier F.
      Publication language and the estimate of treatment effects of physical therapy on balance and postural control after stroke in meta-analyses of randomised controlled trials.
      ,
      • Hartling L
      • Featherstone R
      • Nuspl M
      • Shave K
      • Dryden DM
      • Vandermeer B.
      Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews.
      Out of 259 meta-analyses from five publications that included non-English-language studies, [
      • Dechartres A
      • Atal I
      • Riveros C
      • Meerpohl J
      • Ravaud P.
      Association between publication characteristics and treatment effect estimates: a meta-epidemiologic study.
      ,
      • Nussbaumer-Streit B
      • Klerings I
      • Dobrescu AI
      • Persad E
      • Stevens A
      • Garritty C
      • et al.
      Excluding non-English publications from evidence-syntheses did not change conclusions: a meta-epidemiological study.
      ,
      • Gregoire G
      • Derderian F
      • Le Lorier J
      Selecting the language of the publications included in a meta-analysis: is there a tower of babel bias?.
      ,
      • Hugues A
      • Di Marco J
      • Bonan I
      • Rode G
      • Cucherat M
      • Gueyffier F.
      Publication language and the estimate of treatment effects of physical therapy on balance and postural control after stroke in meta-analyses of randomised controlled trials.
      ,
      • Hartling L
      • Featherstone R
      • Nuspl M
      • Shave K
      • Dryden DM
      • Vandermeer B.
      Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews.
      restricting the literature searches or inclusion criteria to the English language led to a change in statistical significance in 23 cases (9%), 3 in complementary and alternative medicine and 20 in conventional medicine. Most commonly, the statistical significance was lost by excluding non-English-language publications (20 cases).
      Changes in treatment effects after the exclusion of non-English publications varied across the included studies. Hartling et al. identified 15/129 meta-analyses that included non-English publications. The authors reported small to moderate changes in treatment effect estimates in the majority of meta-analyses (13/15). In two meta-analyses, however, excluding non-English studies led to a large change (more than 20%) in the treatment effect estimates. [
      • Hartling L
      • Featherstone R
      • Nuspl M
      • Shave K
      • Dryden DM
      • Vandermeer B.
      Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews.
      ] In another methods study, the changes in treatment effect estimates were smaller than 5% in 21/50 meta-analyses. [
      • Juni P
      • Holenstein F
      • Sterne J
      • Bartlett C
      • Egger M.
      Direction and impact of language bias in meta-analyses of controlled trials: empirical study.
      ,
      • Egger M
      • Juni P
      • Bartlett C
      • Holenstein F
      • Sterne J.
      How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study.
      For systematic reviews dealing with complementary and alternative medicine topics, however, the exclusion of non-English publications resulted in a 63% smaller protective effect estimate (random effects ROR 1.63; 95% CI 1.03–2.60). [
      • Moher D
      • Pham B
      • Lawson ML
      • Klassen TP.
      The inclusion of reports of randomised trials published in languages other than English in systematic reviews.
      ,
      • Pham B
      • Klassen TP
      • Lawson ML
      • Moher D.
      Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary.

      3.3 Impact of language restrictions on certainty of evidence ratings

      None of the included studies reported on this outcome.

      3.4 Impact of language restrictions on overall conclusions

      Two studies determined changes in the authors’ conclusions as a consequence of excluding non-English publications. [
      • Nussbaumer-Streit B
      • Klerings I
      • Dobrescu AI
      • Persad E
      • Stevens A
      • Garritty C
      • et al.
      Excluding non-English publications from evidence-syntheses did not change conclusions: a meta-epidemiological study.
      ,
      • Hartling L
      • Featherstone R
      • Nuspl M
      • Shave K
      • Dryden DM
      • Vandermeer B.
      Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews.
      Both showed that the exclusion of non-English publications did not change the review authors’ overall conclusions.
      Nussbaumer-Streit et al. identified only two reviews out of 59, both focused on conventional medicine, in which the exclusion of non-English publications changed the statistical significance of one outcome. [
      • Nussbaumer-Streit B
      • Klerings I
      • Dobrescu AI
      • Persad E
      • Stevens A
      • Garritty C
      • et al.
      Excluding non-English publications from evidence-syntheses did not change conclusions: a meta-epidemiological study.
      ] However, the authors’ conclusion remained the same in both reviews, with less certainty, regardless of the change in statistical significance. [
      • Nussbaumer-Streit B
      • Klerings I
      • Dobrescu AI
      • Persad E
      • Stevens A
      • Garritty C
      • et al.
      Excluding non-English publications from evidence-syntheses did not change conclusions: a meta-epidemiological study.
      ]
      Hartling et al. evaluated 15 meta-analyses with and without non-English publications included in the analyses. [
      • Hartling L
      • Featherstone R
      • Nuspl M
      • Shave K
      • Dryden DM
      • Vandermeer B.
      Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews.
      ] No changes in the overall review conclusions were reported due to language restrictions. [
      • Hartling L
      • Featherstone R
      • Nuspl M
      • Shave K
      • Dryden DM
      • Vandermeer B.
      Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews.
      ]

      3.5 Impact of language restrictions on statistical heterogeneity

      Changes in statistical heterogeneity due to language restrictions were assessed in one study, published in two articles. [
      • Moher D
      • Pham B
      • Lawson ML
      • Klassen TP.
      The inclusion of reports of randomised trials published in languages other than English in systematic reviews.
      ,
      • Pham B
      • Klassen TP
      • Lawson ML
      • Moher D.
      Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary.
      The authors noted a reduced statistical heterogeneity, from an I2 greater than 0.30 to less or equal to 0.3 (no heterogeneity) in 3/42 reviews (one related to conventional and two to complementary and alternative topics) after the exclusion of non-English publications.

      3.6 Impact of language restrictions on the number of studies and included participants

      The proportion of primary non-English-language publications ranged from 2% [
      • Nussbaumer-Streit B
      • Klerings I
      • Dobrescu AI
      • Persad E
      • Stevens A
      • Garritty C
      • et al.
      Excluding non-English publications from evidence-syntheses did not change conclusions: a meta-epidemiological study.
      ] to 22% [
      • Moher D
      • Pham B
      • Lawson ML
      • Klassen TP.
      The inclusion of reports of randomised trials published in languages other than English in systematic reviews.
      ,
      • Pham B
      • Klassen TP
      • Lawson ML
      • Moher D.
      Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary.
      , except in two meta-analyses, reported by Hartling et al., which included only non-English publications. These meta-analyses were on conventional as well as on complementary and alternative treatments of acute respiratory infections. [
      • Hartling L
      • Featherstone R
      • Nuspl M
      • Shave K
      • Dryden DM
      • Vandermeer B.
      Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews.
      ] In one methods study, the proportion of primary non-English-language publications could not be calculated because the authors provided only the number of relevant non-English-language publications after the search strategy of the included meta-analyses was duplicated, not the total number of included studies. [
      • Gregoire G
      • Derderian F
      • Le Lorier J
      Selecting the language of the publications included in a meta-analysis: is there a tower of babel bias?.
      ]
      The total number of participants in the non-English-language publications was reported in four studies [
      • Moher D
      • Pham B
      • Lawson ML
      • Klassen TP.
      The inclusion of reports of randomised trials published in languages other than English in systematic reviews.
      ,
      • Pham B
      • Klassen TP
      • Lawson ML
      • Moher D.
      Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary.
      ,
      • Hugues A
      • Di Marco J
      • Bonan I
      • Rode G
      • Cucherat M
      • Gueyffier F.
      Publication language and the estimate of treatment effects of physical therapy on balance and postural control after stroke in meta-analyses of randomised controlled trials.
      ,
      • Hartling L
      • Featherstone R
      • Nuspl M
      • Shave K
      • Dryden DM
      • Vandermeer B.
      Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews.
      and ranged from 693 to 17,810, a smaller number compared to the English-language publications (range from 4910 to 102,735). [
      • Moher D
      • Pham B
      • Lawson ML
      • Klassen TP.
      The inclusion of reports of randomised trials published in languages other than English in systematic reviews.
      ,
      • Pham B
      • Klassen TP
      • Lawson ML
      • Moher D.
      Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary.
      ,
      • Hugues A
      • Di Marco J
      • Bonan I
      • Rode G
      • Cucherat M
      • Gueyffier F.
      Publication language and the estimate of treatment effects of physical therapy on balance and postural control after stroke in meta-analyses of randomised controlled trials.
      ,
      • Hartling L
      • Featherstone R
      • Nuspl M
      • Shave K
      • Dryden DM
      • Vandermeer B.
      Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews.

      3.7 Impact of language restrictions on the risk of bias of the included studies

      Three studies compared the methodological quality of the English and non-English publications. [
      • Juni P
      • Holenstein F
      • Sterne J
      • Bartlett C
      • Egger M.
      Direction and impact of language bias in meta-analyses of controlled trials: empirical study.
      ,
      • Egger M
      • Juni P
      • Bartlett C
      • Holenstein F
      • Sterne J.
      How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study.
      ,
      • Dechartres A
      • Atal I
      • Riveros C
      • Meerpohl J
      • Ravaud P.
      Association between publication characteristics and treatment effect estimates: a meta-epidemiologic study.
      ,
      • Hugues A
      • Di Marco J
      • Bonan I
      • Rode G
      • Cucherat M
      • Gueyffier F.
      Publication language and the estimate of treatment effects of physical therapy on balance and postural control after stroke in meta-analyses of randomised controlled trials.
      The non-English-language publications had a higher risk of bias compared with the English-language publications, especially due to inadequate reporting of the blinding process, incomplete outcome data, and concealment of allocation.
      Two studies assessed the presence of publication bias with and without non-English publications: one used a median Z-score, [
      • Moher D
      • Pham B
      • Lawson ML
      • Klassen TP.
      The inclusion of reports of randomised trials published in languages other than English in systematic reviews.
      ,
      • Pham B
      • Klassen TP
      • Lawson ML
      • Moher D.
      Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary.
      and one used the funnel plot asymmetry coefficient. [
      • Egger M
      • Juni P
      • Bartlett C
      • Holenstein F
      • Sterne J.
      How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study.
      ] The funnel plot asymmetry was reduced after the exclusion of non-English publications in 3/31 meta-analyses, two of them focused on complementary and alternative medicine. [
      • Moher D
      • Pham B
      • Lawson ML
      • Klassen TP.
      The inclusion of reports of randomised trials published in languages other than English in systematic reviews.
      ,
      • Pham B
      • Klassen TP
      • Lawson ML
      • Moher D.
      Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary.
      The combined asymmetry coefficient of 49 meta-analyses changed from -0.49 to 0.07 after the non-English publications were excluded. [
      • Egger M
      • Juni P
      • Bartlett C
      • Holenstein F
      • Sterne J.
      How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study.
      ] This indicates that excluding the non-English publications from the analyses could reduce small study effects.

      3.8 Languages and countries of non-english studies

      The medical topics of the non-English publications were diverse, the majority from conventional medicine (e.g., cardiology, orthopedics, rheumatology, infectious diseases); four methods studies [
      • Moher D
      • Pham B
      • Lawson ML
      • Klassen TP.
      The inclusion of reports of randomised trials published in languages other than English in systematic reviews.
      ,
      • Juni P
      • Holenstein F
      • Sterne J
      • Bartlett C
      • Egger M.
      Direction and impact of language bias in meta-analyses of controlled trials: empirical study.
      ,
      • Egger M
      • Juni P
      • Bartlett C
      • Holenstein F
      • Sterne J.
      How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study.
      ,
      • Dechartres A
      • Atal I
      • Riveros C
      • Meerpohl J
      • Ravaud P.
      Association between publication characteristics and treatment effect estimates: a meta-epidemiologic study.
      ,
      • Pham B
      • Klassen TP
      • Lawson ML
      • Moher D.
      Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary.
      also included publications focusing on complementary and alternative medicine (e.g., acupuncture, St. John’s wort). Across all eight included methods studies, the most frequent languages of the non-English publications were German, Chinese, and French (Fig. 2). The origin country of the non-English publications was not mentioned in any of the methods studies. [
      • Moher D
      • Pham B
      • Lawson ML
      • Klassen TP.
      The inclusion of reports of randomised trials published in languages other than English in systematic reviews.
      ,
      • Juni P
      • Holenstein F
      • Sterne J
      • Bartlett C
      • Egger M.
      Direction and impact of language bias in meta-analyses of controlled trials: empirical study.
      ,
      • Egger M
      • Juni P
      • Bartlett C
      • Holenstein F
      • Sterne J.
      How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study.
      • Pham B
      • Klassen TP
      • Lawson ML
      • Moher D.
      Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary.
      ,
      • Nussbaumer-Streit B
      • Klerings I
      • Dobrescu AI
      • Persad E
      • Stevens A
      • Garritty C
      • et al.
      Excluding non-English publications from evidence-syntheses did not change conclusions: a meta-epidemiological study.
      • Hartling L
      • Featherstone R
      • Nuspl M
      • Shave K
      • Dryden DM
      • Vandermeer B.
      Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews.
      Fig 2
      Fig. 2Languages of non-English publications across all included methods studies

      4. Discussion

      The latest systematic review on the impact of language restrictions in systematic reviews and/or meta-analyses was published in 2012 and focused solely on conventional medicine topics and how language restrictions impact effect estimates. [
      • Morrison A
      • Polisena J
      • Husereau D
      • Moulton K
      • Clark M
      • Fiander M
      • et al.
      The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies.
      ] Our systematic review extends the scope to complementary and alternative medicine and diagnostic test accuracy reviews and the impact of language restrictions on systematic review conclusions. We identified four new methods studies that were not part of the previous review. [
      • Dechartres A
      • Atal I
      • Riveros C
      • Meerpohl J
      • Ravaud P.
      Association between publication characteristics and treatment effect estimates: a meta-epidemiologic study.
      ,
      • Nussbaumer-Streit B
      • Klerings I
      • Dobrescu AI
      • Persad E
      • Stevens A
      • Garritty C
      • et al.
      Excluding non-English publications from evidence-syntheses did not change conclusions: a meta-epidemiological study.
      ,
      • Hugues A
      • Di Marco J
      • Bonan I
      • Rode G
      • Cucherat M
      • Gueyffier F.
      Publication language and the estimate of treatment effects of physical therapy on balance and postural control after stroke in meta-analyses of randomised controlled trials.
      ,
      • Hartling L
      • Featherstone R
      • Nuspl M
      • Shave K
      • Dryden DM
      • Vandermeer B.
      Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews.
      Three focused on conventional medicine topics; one also included complementary and alternative publications. [
      • Dechartres A
      • Atal I
      • Riveros C
      • Meerpohl J
      • Ravaud P.
      Association between publication characteristics and treatment effect estimates: a meta-epidemiologic study.
      ] Overall, the new methods studies included more meta-analyses and/or systematic reviews compared to those included in the previous review (344 vs. 230).
      Overall, we analyzed eight methods studies; the number of included meta-analyses and/or systematic reviews ranged from 9 [
      • Hugues A
      • Di Marco J
      • Bonan I
      • Rode G
      • Cucherat M
      • Gueyffier F.
      Publication language and the estimate of treatment effects of physical therapy on balance and postural control after stroke in meta-analyses of randomised controlled trials.
      ] to 147 [
      • Dechartres A
      • Atal I
      • Riveros C
      • Meerpohl J
      • Ravaud P.
      Association between publication characteristics and treatment effect estimates: a meta-epidemiologic study.
      ]. In the case of companion articles, we presented them once with multiple references. [
      • Moher D
      • Pham B
      • Lawson ML
      • Klassen TP.
      The inclusion of reports of randomised trials published in languages other than English in systematic reviews.
      ,
      • Juni P
      • Holenstein F
      • Sterne J
      • Bartlett C
      • Egger M.
      Direction and impact of language bias in meta-analyses of controlled trials: empirical study.
      ,
      • Egger M
      • Juni P
      • Bartlett C
      • Holenstein F
      • Sterne J.
      How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study.
      ,
      • Pham B
      • Klassen TP
      • Lawson ML
      • Moher D.
      Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary.
      The majority of studies retrospectively removed non-English publications from the selected systematic reviews. [
      • Moher D
      • Pham B
      • Lawson ML
      • Klassen TP.
      The inclusion of reports of randomised trials published in languages other than English in systematic reviews.
      ,
      • Juni P
      • Holenstein F
      • Sterne J
      • Bartlett C
      • Egger M.
      Direction and impact of language bias in meta-analyses of controlled trials: empirical study.
      ,
      • Egger M
      • Juni P
      • Bartlett C
      • Holenstein F
      • Sterne J.
      How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study.
      • Pham B
      • Klassen TP
      • Lawson ML
      • Moher D.
      Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary.
      ,
      • Nussbaumer-Streit B
      • Klerings I
      • Dobrescu AI
      • Persad E
      • Stevens A
      • Garritty C
      • et al.
      Excluding non-English publications from evidence-syntheses did not change conclusions: a meta-epidemiological study.
      ,
      • Moher D
      • Pham B
      • Klassen TP
      • Schulz KF
      • Berlin JA
      • Jadad AR
      • et al.
      What contributions do languages other than English make on the results of meta-analyses?.
      ,
      • Hugues A
      • Di Marco J
      • Bonan I
      • Rode G
      • Cucherat M
      • Gueyffier F.
      Publication language and the estimate of treatment effects of physical therapy on balance and postural control after stroke in meta-analyses of randomised controlled trials.
      ,
      • Hartling L
      • Featherstone R
      • Nuspl M
      • Shave K
      • Dryden DM
      • Vandermeer B.
      Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews.
      This approach simulates language restrictions at the study selection level. Only one study duplicated the computerized search and retrospectively added non-English publications to the original meta-analyses [
      • Gregoire G
      • Derderian F
      • Le Lorier J
      Selecting the language of the publications included in a meta-analysis: is there a tower of babel bias?.
      ], simulating the language restrictions already applied to the search approach.
      The results of our review indicate that the impact of restricting systematic reviews to English-language publications is negligible for most conventional medicine topics. Based on data from 259 meta-analyses, the effect estimates and conclusions rarely changed when non-English-language publications were excluded from the systematic reviews. An exception, however, occurred in complementary and alternative medicine topics. Excluding non-English-language studies from the systematic reviews of complementary and alternative medicine topics often led to reduced treatment effects and changes in the statistical significance. Conceivable explanations could be that studies on those topics are more commonly published in languages other than English, or that non-English complementary and alternative medicine publications are more biased than English-language ones and overestimate treatment effects. Larger treatment effects [
      • Juni P
      • Holenstein F
      • Sterne J
      • Bartlett C
      • Egger M.
      Direction and impact of language bias in meta-analyses of controlled trials: empirical study.
      ,
      • Egger M
      • Juni P
      • Bartlett C
      • Holenstein F
      • Sterne J.
      How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study.
      ,
      • Dechartres A
      • Atal I
      • Riveros C
      • Meerpohl J
      • Ravaud P.
      Association between publication characteristics and treatment effect estimates: a meta-epidemiologic study.
      ,
      • Pham B
      • Klassen TP
      • Lawson ML
      • Moher D.
      Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary.
      and higher risks for bias were observed in non-English publications across all medical topics. [
      • Juni P
      • Holenstein F
      • Sterne J
      • Bartlett C
      • Egger M.
      Direction and impact of language bias in meta-analyses of controlled trials: empirical study.
      ,
      • Egger M
      • Juni P
      • Bartlett C
      • Holenstein F
      • Sterne J.
      How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study.
      ,
      • Dechartres A
      • Atal I
      • Riveros C
      • Meerpohl J
      • Ravaud P.
      Association between publication characteristics and treatment effect estimates: a meta-epidemiologic study.
      ,
      • Hugues A
      • Di Marco J
      • Bonan I
      • Rode G
      • Cucherat M
      • Gueyffier F.
      Publication language and the estimate of treatment effects of physical therapy on balance and postural control after stroke in meta-analyses of randomised controlled trials.
      .
      The main reason authors restrict inclusion criteria to English-only is related to the time-saving factor as well as to human and financial resources. However, none of the studies assessed the resources saved by limiting a systematic review to English-only publications. It is possible that limiting the search strategies to the English language might save more time than limiting the eligibility criteria without limiting the search strategy accordingly. This way, non-English abstracts were removed from the search result, reducing the number of records retrieved and removing the necessity to ascertain the publication language of an eligible study. However, all included methods studies retrospectively removed non-English studies from the assessed systematic review. This approach allows conclusions about the impact of limiting the eligibility criteria—but not the search strategy—to English-only. Grégoire et al. (1995) were the only team that based their analysis on meta-analyses that had excluded non-English studies at the search stage and conducted new searches looking specifically for publications written in the languages that had been excluded. Three-quarters of the meta-analyses results changed due to this approach, but only one showed a change in statistical significance; however, it is not clear how comparable the new search methods were compared to those of the original authors.
      Our study has several limitations. First, we did not assess the methodological quality of the included studies because no validated risk of bias tool exists for methods studies. However, we extracted the methodological approach of each included study and did not identify major flaws that would raise concerns regarding their trustworthiness.
      It is also possible that language bias in systematic reviews is not caused by the stated inclusion criteria or the use of language restrictions in the database searches, but by the choice of the information sources used in the review. For example, international English-language bibliographic databases, such as MEDLINE, Embase, CINAHL, PsycInfo, etc. contain only a fraction of non-English-language scientific literature. If a systematic review limits its information sources to these types of databases, non-English-language studies will be missed, not because of the explicit inclusion criteria or search strategies used, but because they are not indexed in the chosen databases. In turn, methodological studies that evaluate language bias by retrospectively removing non-English studies from existing systematic reviews could be starting from an already biased foundation. Our study did not assess the impact of searching specific databases; the focus was on the language restrictions on the inclusion criteria and/or search strategy.
      Third, we were not able to identify methods studies assessing the impact of language restrictions on diagnostic test accuracy systematic reviews. These reviews often require the screening of a larger volume of abstracts than treatment reviews because methodological search filters are strongly discouraged. Excluding non-English studies could help save even more resources than for treatment reviews. [
      • Arevalo-Rodriguez I
      • Steingart KR
      • Tricco AC
      • Nussbaumer-Streit B
      • Kaunelis D
      • Alonso-Coello P
      • et al.
      Current methods for development of rapid reviews about diagnostic tests: an international survey.
      ] Unfortunately, our results cannot be generalized to those types of reviews. Future research is needed to assess the impact of restricting inclusion criteria to English-only publications on diagnostic test accuracy systematic reviews.
      According to our results, language restrictions have a minimal effect on treatment effect estimates and review conclusions in most medical fields. Nevertheless, one must always consider the appropriateness of language restrictions individually for each topic. The current COVID-19 pandemic, for example, required a language-inclusive approach, especially at the beginning of the pandemic, because many of the early studies were published in Chinese or other languages. [
      • Nussbaumer-Streit B
      • Klerings I
      • Gartlehner G.
      • Reply to letter to the editor
      Lessons from COVID-19 to future evidence synthesis efforts: first living search strategy and out of date scientific publishing and indexing industry.
      ]

      5. Conclusion

      Restricting the inclusion criteria to English-language publications seems to be a viable approach for systematic treatment reviews for most medical topics, with a negligible impact on the effect estimates and conclusions. However, systematic reviewers should always assess the appropriateness of this approach depending on the specific topic of the review. Further research to assess the impact of language restrictions already applied to the search strategy and on diagnostic test accuracy reviews is needed.

      Author contributions

      Dobrescu Andreea-Iulia: Conceptualization, Project administration, Formal analysis, Investigation, Methodology, Writing - original draft, Writing - review and editing. Nussbaumer-Streit Barbara: Investigation, Methodology, Writing - review and editing. Klerings Irma: Methodology (search methods), Data collection, Data curation, Software, Writing - review and editing. Wagner Gernot: Investigation, Writing - review and editing. Persad Emma: Investigation, Writing - review and editing. Sommer Isolde: Investigation, Writing - review and editing. Herkner Harald: Investigation, Writing - review and editing. Gartlehner Gerald: Supervision, Methodology, Writing - review and editing.

      Conflict of interest

      None.

      Acknowledgments

      We would like to thank Tarquin Mittermayr, Information Specialist from the Austrian Institute for Health Technology Assessments, who peer-reviewed the search strategy. We would also like to thank Sandra Hummel for administrative support throughout this project.

      Funding source

      This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

      References

        • Moher D
        • Tetzlaff J
        • Tricco AC
        • Sampson M
        • Altman DG.
        Epidemiology and reporting characteristics of systematic reviews.
        PLoS Med. 2007; 4: e78
        • Rosselli D.
        The language of biomedical sciences.
        Lancet. 2016; 387: 1720-1721
        • Moher D
        • Fortin P
        • Jadad AR
        • Juni P
        • Klassen T
        • Le Lorier J
        • et al.
        Completeness of reporting of trials published in languages other than English: implications for conduct and reporting of systematic reviews.
        Lancet. 1996; 347: 363-366
        • Moher D
        • Pham B
        • Lawson ML
        • Klassen TP.
        The inclusion of reports of randomised trials published in languages other than English in systematic reviews.
        Health Technol Assess. 2003; 7: 1-90
        • Juni P
        • Holenstein F
        • Sterne J
        • Bartlett C
        • Egger M.
        Direction and impact of language bias in meta-analyses of controlled trials: empirical study.
        Int J Epidemiol. 2002; 31: 115-123
      1. Higgins JPT, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (eds), Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019) 2019.

        • Egger M
        • Zellweger-Zahner T
        • Schneider M
        • Junker C
        • Lengeler C
        • Antes G.
        Language bias in randomised controlled trials published in English and German.
        Lancet. 1997; 350: 326-329
        • Egger M
        • Juni P
        • Bartlett C
        • Holenstein F
        • Sterne J.
        How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study.
        Health Technol Assess. 2003; 7: 1-76
        • Dechartres A
        • Atal I
        • Riveros C
        • Meerpohl J
        • Ravaud P.
        Association between publication characteristics and treatment effect estimates: a meta-epidemiologic study.
        Ann Intern Med. 2018; 169: 385-393
        • Pham B
        • Klassen TP
        • Lawson ML
        • Moher D.
        Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary.
        J Clin Epidemiol. 2005; 58: 769-776
        • Morrison A
        • Polisena J
        • Husereau D
        • Moulton K
        • Clark M
        • Fiander M
        • et al.
        The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies.
        Int J Technol Assess Health Care. 2012; 28: 138-144
      2. Quality AfHRa. Methods guide for effectiveness and comparative effectiveness reviews. 2014.

      3. 3. CSRCPaGSN. Methodological expectations of Campbell Collaboration intervention reviews: Conduct standards 2019. Available from: https://onlinelibrarywileycom/page/journal/18911803/homepage/author-guidelines. 2019.

        • Jackson JL
        • Kuriyama A.
        How often do systematic reviews exclude articles not published in English?.
        J Gen Intern Med. 2019; 34: 1388-1389
        • Moher D
        • Liberati A
        • Tetzlaff J
        • Altman DG
        • Group P.
        Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
        BMJ. 2009; 339: b2535
        • McGowan J
        • Sampson M
        • Salzwedel DM
        • Cogo E
        • Foerster V
        • Lefebvre C.
        PRESS peer review of electronic search strategies: 2015 guideline statement.
        J Clin Epidemiol. 2016; 75: 40-46
      4. Covidence systematic review software VHI, Melbourne, Australia. Available at: www.covidence.org.

        • Nussbaumer-Streit B
        • Klerings I
        • Dobrescu AI
        • Persad E
        • Stevens A
        • Garritty C
        • et al.
        Excluding non-English publications from evidence-syntheses did not change conclusions: a meta-epidemiological study.
        J Clin Epidemiol. 2020; 118: 42-54
        • Moher D
        • Pham B
        • Klassen TP
        • Schulz KF
        • Berlin JA
        • Jadad AR
        • et al.
        What contributions do languages other than English make on the results of meta-analyses?.
        J Clin Epidemiol. 2000; 53: 964-972
        • Gregoire G
        • Derderian F
        • Le Lorier J
        Selecting the language of the publications included in a meta-analysis: is there a tower of babel bias?.
        J Clin Epidemiol. 1995; 48: 159-163
        • Hugues A
        • Di Marco J
        • Bonan I
        • Rode G
        • Cucherat M
        • Gueyffier F.
        Publication language and the estimate of treatment effects of physical therapy on balance and postural control after stroke in meta-analyses of randomised controlled trials.
        PLoS One. 2020; 15e0229822
        • Hartling L
        • Featherstone R
        • Nuspl M
        • Shave K
        • Dryden DM
        • Vandermeer B.
        Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews.
        BMC Med Res Methodol. 2017; 17: 64
        • Arevalo-Rodriguez I
        • Steingart KR
        • Tricco AC
        • Nussbaumer-Streit B
        • Kaunelis D
        • Alonso-Coello P
        • et al.
        Current methods for development of rapid reviews about diagnostic tests: an international survey.
        BMC Med Res Methodol. 2020; 20: 115
        • Nussbaumer-Streit B
        • Klerings I
        • Gartlehner G.
        • Reply to letter to the editor
        Lessons from COVID-19 to future evidence synthesis efforts: first living search strategy and out of date scientific publishing and indexing industry.
        J Clin Epidemiol. 2020; 123: 173-174