Advertisement

Methods used to select results to include in meta-analyses of nutrition research: A meta-research study

Open AccessPublished:November 12, 2021DOI:https://doi.org/10.1016/j.jclinepi.2021.11.016

      Abstract

      Objectives

      To investigate how often review authors encounter multiple results from included studies that are eligible for inclusion in a particular meta-analysis, and how often methods to select results are specified.

      Methods

      MEDLINE and Epistemonikos were searched (January 2018–June 2019) to identify systematic reviews with meta-analysis of the association between food/diet and health-related outcomes. A random sample of these reviews was selected, and for the first presented (index) meta-analysis, rules used to select effect estimates to include in this meta-analysis were extracted from the reviews and their protocols. All effect estimates from the primary studies that were eligible for inclusion in the index meta-analyses were extracted (e.g., when a study report presented effect estimates for blood pressure at 3 weeks and 6 weeks, both unadjusted and adjusted for covariates, and all were eligible for inclusion in a meta-analysis of the effect of red meat consumption on blood pressure, we extracted all estimates, and classified the study as having “multiplicity of results”).

      Results

      Forty-two systematic reviews with 325 studies (104 randomized, 221 non-randomized) were included; 14 reviews had a protocol. In 29% of review protocols and 69% of reviews, authors specified at least one decision rule to select effect estimates when multiple were available. In 68% of studies included in the index meta-analyses, there was at least one type of multiplicity of results.

      Conclusions

      Authors of systematic reviews of nutrition studies should anticipate encountering multiplicity of results in the included primary studies. Specification of methods to handle multiplicity when designing reviews is therefore recommended.

      Keywords

      What is new
      • Key Findings
      • Authors of systematic reviews of nutrition research should anticipate encountering multiplicity of results in the included primary studies (i.e., multiple effect estimates being eligible for inclusion in a particular meta-analysis).
      • Decision rules to select results for inclusion in meta-analyses of nutrition research were infrequently pre-specified.
      • What this adds to what was known?
      • Previous studies have found that multiplicity of results of continuous outcomes in studies included in systematic reviews was common, and methods used to select results to include in meta-analyses were infrequently pre-specified in systematic review protocols. However, none of the previous studies examined meta-analyses in nutrition research, inclusion of randomized or non-randomized studies, or where the outcome was non-continuous (e.g., binary, count or time-to-event); circumstances for which different forms of multiplicity might arise. Our study addressed this gap.
      • What is the implication and what should change now?
      • Pre-specification of decision rules to handle multiplicity when designing reviews is recommended. In the systematic review, we recommend reporting any modifications to the specified rules, or any additions that were introduced to cover multiplicity scenarios that had not been anticipated when designing the review.

      1. Background

      The Global Burden of Disease study 2019 reported that diet has a significant impact on health outcomes. Diet quality was found to be the fifth leading risk factor for disability-adjusted life years [
      GBD 2019 Risk Factors Collaborators
      Global burden of 87 risk factors in 204 countries and territories, 1990-2019: a systematic analysis for the global burden of disease study 2019.
      ]. Large and long-term prospective observational studies and short-term clinical trials have found associations between particular dietary factors and non-communicable diseases [
      • Estruch R
      • Ros E
      • Salas-Salvadó J
      • Covas MI
      • Corella D
      • Arós F
      • et al.
      Primary prevention of cardiovascular disease with a mediterranean diet supplemented with extra-virgin olive oil or nuts.
      ,
      • Taylor MK
      • Mahnken JD
      • Sullivan DK.
      NHANES 2011-2014 reveals cognition of US older adults may benefit from better adaptation to the mediterranean diet.
      ,
      • Dong X
      • Li S
      • Chen J
      • Li Y
      • Wu Y
      • Zhang D.
      Association of dietary ω-3 and ω-6 fatty acids intake with cognitive performance in older adults: national health and nutrition examination survey (NHANES) 2011-2014.
      ]. Systematic reviews (SRs) based on such studies are being used to inform recommendations in dietary guidelines [
      • Kelley GA
      • Kelley KS.
      Systematic reviews and meta-analysis in nutrition research.
      ,
      • Blake P
      • Durão S
      • Naude CE
      • Bero L.
      An analysis of methods used to synthesize evidence and grade recommendations in food-based dietary guidelines.
      ,
      • Dai Z
      • Kroeger CM
      • Lawrence M
      • Scrinis G
      • Bero L.
      Comparison of methodological quality between the 2007 and 2019 Canadian dietary guidelines.
      ]. However, flaws in the design, conduct and reporting of SRs may yield misleading results, and in turn, misinform guideline recommendations [
      • Schunemann HJ
      • Wiercioch W
      • Etxeandia I
      • Falavigna M
      • Santesso N
      • Mustafa R
      • et al.
      Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise.
      ].
      One challenge SR authors often face is a multiplicity of results in the primary studies [
      • Page MJ
      • McKenzie JE
      • Chau M
      • Green SE
      • Forbes A
      Methods to select results to include in meta-analyses deserve more consideration in systematic reviews.
      ]. For example, a study report may present multiple effect estimates for the association between red meat consumption and gout, where these estimates may arise from the fitting of multiple statistical models with different outcome definitions of gout (e.g., diagnosed according to different criteria), different exposure levels (e.g., <1, 1-2, 2-3 or >3 servings/week), or where adjustment is made for different sets of confounders. Although inclusion of multiple effect estimates from a particular study in a meta-analysis is possible (using methods that adjust for statistical dependency) [
      • Lopez-Lopez JA
      • Page MJ
      • Lipsey MW
      • Higgins JPT.
      Dealing with effect size multiplicity in systematic reviews and meta-analyses.
      ], more commonly only one of the available effect estimates is selected for inclusion. There are various methods that can be used to select a single effect estimate [
      • Page MJ
      • McKenzie JE
      • Chau M
      • Green SE
      • Forbes A
      Methods to select results to include in meta-analyses deserve more consideration in systematic reviews.
      ,
      • Lopez-Lopez JA
      • Page MJ
      • Lipsey MW
      • Higgins JPT.
      Dealing with effect size multiplicity in systematic reviews and meta-analyses.
      ]. However, when this selection is based on the statistical significance, magnitude or direction of effect, this may introduce bias into the meta-analysis effect estimate [
      • Page MJ
      • Forbes A
      • Chau M
      • Green SE
      • McKenzie JE.
      Investigation of bias in meta-analyses due to selective inclusion of trial effect estimates: empirical study.
      ]. We refer to this selection process as “selective inclusion of results.”
      To help mitigate the potential for selective inclusion of results, it has been recommended that methods for dealing with multiplicity should be pre-specified [
      • Page MJ
      • Forbes A
      • Chau M
      • Green SE
      • McKenzie JE.
      Investigation of bias in meta-analyses due to selective inclusion of trial effect estimates: empirical study.
      ]. This includes pre-specification of “eligibility criteria” for each meta-analysis, indicating which results are eligible for inclusion in the meta-analysis (e.g., intervention groups, measurement instruments, time points), and “decision rules,” which specify the methods that will be used to select a single result from a study when multiple results are eligible for inclusion in the same meta-analysis (see Box 1 for examples of eligibility criteria and decision rules).
      Box 1. Examples of eligibility criteria and decision rules to select results
      Example of eligibility criteria to select results: Systematic reviewers state that only study effect estimates that were adjusted for sex and age would be included in a meta-analysis of the association between fruit consumption and coronary heart disease.
      Example of a decision rule to select results when multiple are available: Systematic reviewers state that if multiple effect estimates quantifying the association between different levels of fruit consumption and coronary heart disease were available, as would arise if intake was categorised into quartiles in a study report, only the contrast between the highest (e.g., quartile 4), and lowest (e.g., quartile 1) intake would be included in the meta-analysis.
      Previous research has examined the extent of multiplicity of results, and the methods used to select results for inclusion in several meta-analyses [
      • Page MJ
      • Forbes A
      • Chau M
      • Green SE
      • McKenzie JE.
      Investigation of bias in meta-analyses due to selective inclusion of trial effect estimates: empirical study.
      ,
      • Tendal B
      • Nuesch E
      • Higgins JP
      • Juni P
      • Gotzsche PC.
      Multiplicity of data in trial reports and the reliability of meta-analyses: empirical study.
      ,
      • Mayo-Wilson E
      • Fusco N
      • Li T
      • Hong H
      • Canner JK
      • Dickersin K
      Multiple outcomes and analyses in clinical trials create challenges for interpretation and research synthesis.
      ]. These studies have focused on a range of conditions and examined multiplicity in meta-analyses of randomized trials with continuous outcomes. All studies found that multiplicity of results was common, and Page et al. [
      • Page MJ
      • McKenzie JE
      • Chau M
      • Green SE
      • Forbes A
      Methods to select results to include in meta-analyses deserve more consideration in systematic reviews.
      ] and Tendal et al. [
      • Tendal B
      • Nuesch E
      • Higgins JP
      • Juni P
      • Gotzsche PC.
      Multiplicity of data in trial reports and the reliability of meta-analyses: empirical study.
      ] found that specification of methods to select results for inclusion were rarely reported. However, none of the studies have examined meta-analyses including randomized or non-randomized studies, or non-continuous outcomes (e.g., binary, count or time-to-event). Understanding whether the extent of multiplicity or the methods used to select results for inclusion vary by these factors is important for developing tailored guidance.
      We aimed to address the identified gaps by investigating the i) extent of multiplicity of results in study reports of nutrition research, and ii) the methods specified in systematic reviews to select results for inclusion in meta-analyses of all outcome types, including randomized or non-randomized study designs. We focus our investigation on reviews of nutrition because of their critical role in informing public health policy and because there has been no previous investigation of multiplicity in this area. Findings from this investigation may indicate the need for development of specific guidance for reviews of nutrition that address multiplicity issues unique to this field.

      2. Methods

      The study protocol has been published [
      • Page MJ
      • Bero L
      • Kroeger CM
      • Dai Z
      • McDonald S
      • Forbes A
      • et al.
      Investigation of risk of bias due to unreported and selectively included results in meta-analyses of nutrition research: the ROBUST study protocol.
      ]. Here, we provide an overview of the methods, with modifications to the protocol reported in Supplementary Table 1. Our manuscript describes one component of the ROBUST study [
      • Page MJ
      • Bero L
      • Kroeger CM
      • Dai Z
      • McDonald S
      • Forbes A
      • et al.
      Investigation of risk of bias due to unreported and selectively included results in meta-analyses of nutrition research: the ROBUST study protocol.
      ]. The ROBUST study aims to explore the extent of multiplicity in study reports, bias due to selective inclusion of results, and bias due to missing results, in systematic reviews of food/diet-outcome relationships. The results of the other components of the ROBUST study will be described in subsequent manuscripts.

      2.1 Eligibility criteria, search, and selection of SRs

      SRs that satisfied the definition of an SR, as outlined in the 2019 edition of the Cochrane Handbook for Systematic Reviews of Interventions [
      • Higgins JP
      • Thomas J
      • Chandler J
      • Cumpston M
      • Li T
      • Page MJ
      • et al.
      Cochrane handbook for systematic reviews of interventions.
      ], and that had explicitly stated methods of study identification (e.g., a search strategy) and of study selection (e.g., eligibility criteria and selection process), and included a meta-analysis of study results, were eligible for inclusion in this study. We included such SRs with meta-analysis that:
      • included studies that enrolled, regardless of their age and background, (a) people who were generally healthy, (b) a mixture of generally healthy people and people with diet-related risk factors (e.g., overweight, high blood pressure) or a particular health condition (e.g., type II diabetes or cardiovascular disease), or (c) people with non-specified health status;
      • included randomized trials or non-randomized studies that assessed the effects of at least one type of food (e.g., eggs, fish) or at least one dietary pattern (e.g., Mediterranean diet) on any continuous (e.g., systolic blood pressure) or non-continuous (e.g., gout incidence) health-related outcome;
      • were published between January 1, 2018 and June 30, 2019 (i.e., within 18 months before the drafting of our study protocol);
      • were written in English;
      • provided citations for all included studies in the SR, and;
      • presented the summary statistics or effect estimate and its precision (e.g., standard error or 95% confidence interval) for each included study, and the meta-analytic summary effect estimate and its precision in the text or forest plot, for at least one meta-analysis of a continuous or non-continuous outcome.
      We excluded
      • SRs that did not include any meta-analysis of a non-continuous or continuous outcome;
      • meta-analyses or pooled analyses of studies conducted outside the context of a SR;
      • SRs that only focused on nutrient-specific associations with outcomes (e.g., examining the effects of single nutrients such as folic acid, salt), as the focus of this study is to assess evidence on the effects of consuming whole foods or dietary patterns on health outcomes, in line with the food-based rather than a nutrient-based approach adopted by the Food and Agriculture Organisation of the United Nations and many other countries developing dietary guidelines;
      • SRs that included studies enrolling only participants with a health condition, or who were obese, or who were frail or elderly people at risk of malnutrition, and;
      • SRs that were co-authored by any of our research team members.
      We searched for eligible SRs indexed in the PubMed and Epistemonikos [
      • Rada G
      • Perez D
      • Araya-Quintanilla F
      • Avila C
      • Bravo-Soto G
      • Bravo-Jeria R
      • et al.
      Epistemonikos: a comprehensive database of systematic reviews for health decision-making.
      ] databases from January 1, 2018 to June 30, 2019 (search strategies reported in supplement 2). The search results were exported into Microsoft Excel, all duplicate records were removed, and the remaining records were randomly sorted. In the piloting phase, four investigators (MJP, CMK, ZD, and SM) independently assessed 50 abstracts against the inclusion criteria (rating each as “Eligible,” “Ineligible”, or “Unsure”), discussed any discrepancies, and made any necessary changes to the screening form. Following piloting, two investigators (MJP and one of CMK, ZD or SM) independently screened titles and abstracts of 450 records. Two investigators (MJP and one of CMK, ZD or SM) then independently assessed the full text of records that were rated as “Eligible” or “Unsure” against the eligibility criteria. This screening process was repeated (in batches of 500 records) until we reached the target sample of 50 SRs, including 25 meta-analyses of continuous, and 25 meta-analyses of non-continuous outcomes. If the total number of eligible SRs exceeded this target at the end of a batch, we planned to randomly sample 25 SRs of each type. Our target of 50 SRs was primarily selected for feasibility reasons given our available resources to conduct all components of the ROBUST study [
      • Page MJ
      • Bero L
      • Kroeger CM
      • Dai Z
      • McDonald S
      • Forbes A
      • et al.
      Investigation of risk of bias due to unreported and selectively included results in meta-analyses of nutrition research: the ROBUST study protocol.
      ], which was informed by the time taken to conduct a previous similar study. Any discrepancies in screening decisions at each stage were resolved via discussion between investigators, or by consultation with another investigator (JM) where necessary. For each included SR, we retrieved the published protocol for the SR or registration record (e.g., PROSPERO record), if available, as cited or reported in the SR.
      From each SR meeting the inclusion criteria, one investigator (MJP) selected one pairwise meta-analysis of aggregate data for inclusion. The selected meta-analysis was the first meta-analytic result mentioned in the review (with no restrictions on its placement in the manuscipt); we refer to the selected meta-analysis as the “index meta-analysis.” We initially selected an index meta-analysis regardless of the outcome domain (e.g., weight, bladder cancer), effect measure (e.g., odds ratio, standardised mean difference), meta-analytic model and number and type of included studies (i.e., randomized or non-randomized study). However, following selection of 50 index meta-analyses and recording the number of studies included in each, we decided to restrict inclusion to only those systematic reviews with an index meta-analysis including fewer than 20 studies, again for reasons of feasibility. The 50 systematic reviews originally sampled included 553 studies, which is more than twice what we had anticipated based on our previous study (which included 44 systematic reviews with 210 studies) [
      • Page MJ
      • McKenzie JE
      • Chau M
      • Green SE
      • Forbes A
      Methods to select results to include in meta-analyses deserve more consideration in systematic reviews.
      ]. We did not replace the systematic reviews that included meta-analyses with more than 20 studies. For each included index meta-analysis with fewer than 20 studies, we retrieved the reports of all included studies (see the protocol for further details [
      • Page MJ
      • Bero L
      • Kroeger CM
      • Dai Z
      • McDonald S
      • Forbes A
      • et al.
      Investigation of risk of bias due to unreported and selectively included results in meta-analyses of nutrition research: the ROBUST study protocol.
      ]).

      2.2 Data collection and management

      A data collection form was developed in REDCap (see Supplementary Table S2) [
      • Harris PA
      • Taylor R
      • Minor BL
      • Elliott V
      • Fernandez M
      • O'Neal L
      • et al.
      The REDCap consortium: building an international community of software platform partners.
      ]. Seven investigators (RK, ZD, SM, CMK, EK, LB, and MJP) piloted the form on two randomly selected meta-analyses and their included studies. Discrepancies were discussed, and we modified the form accordingly. Following piloting, two investigators (RK and one of ZD, SM, CMK, EK, LB and MJP) independently collected data from a random sample of half of the index meta-analyses and their included studies, although one investigator (RK) collected data on the remaining index meta-analyses, and their included studies. Any discrepancies were resolved through discussions between two investigators or adjudication by a third investigator (JM) if necessary.
      An overview of the data items, and the sources these were obtained from (i.e., systematic review protocol, systematic review or study report), is presented in Table 1; further details are available in the protocol [
      • Page MJ
      • Bero L
      • Kroeger CM
      • Dai Z
      • McDonald S
      • Forbes A
      • et al.
      Investigation of risk of bias due to unreported and selectively included results in meta-analyses of nutrition research: the ROBUST study protocol.
      ]. In the case of data extracted from the reports of studies included in the index meta-analysis, we extracted all outcome data that were eligible for inclusion in the index meta-analysis. This was determined by the eligibility criteria and decision rules stated in the SR protocol if available, and if not available, how the comparison and outcome of the meta-analysis were specified in the SR. For example, if the systematic reviewers pre-defined in the SR protocol that the eligible intervention and comparator for the meta-analysis of weight gain was “highest vs. lowest intake of dairy products,” and pre-defined a decision rule stipulating that they would consider only data at 12 weeks follow-up when data were available at multiple time points, we only extracted data for that comparison and time point, regardless of whether study reports had data for other time points and other comparisons for the same outcome. In the absence of an SR protocol, we assumed that no eligibility criteria and decision rules were pre-specified (“worst-case scenario” assumption) and extracted all study outcome data based on how the outcome was specified in the SR. For example, if the systematic reviewers did not state in a protocol which results should be selected when multiple were available, yet defined the meta-analysis as “effect of dairy intake on weight at 6 months,” we extracted all data on weight at 6 months, regardless of the level of intake of dairy, whether results were unadjusted or adjusted, or what analysis sample was used.
      Table 1Data sources and data items (see protocol for further details)
      • Page MJ
      • Bero L
      • Kroeger CM
      • Dai Z
      • McDonald S
      • Forbes A
      • et al.
      Investigation of risk of bias due to unreported and selectively included results in meta-analyses of nutrition research: the ROBUST study protocol.
      SourceData items
      Systematic review protocolYear of publication/registration; eligibility criteria and decision rules to select results to include in the index meta-analysis
      Systematic reviewGeneral characteristics of the systematic review
      Journal name; year of publication; corresponding author's country and affiliation; conflicts of interest of review authors; source of funding for the review;
      General characteristics of the index meta-analysis
      Number of studies and participants; type of population investigated; type of interventions/exposures investigated; type of studies included in the meta-analysis; outcome domain (such as weight, cardiovascular function); outcome primacy label (primary or secondary or unlabelled); meta-analysis effect measure; meta-analysis model; eligibility criteria and decision rules to select results to include in the index meta-analysis; summary statistics, effect estimates and measures of precision (e.g confidence interval) for each included study; and the meta-analytic effect estimate and measure of precision.
      Study reportsOutcome data that could potentially be included in the index meta-analysis
      Outcome definition and measurement instrument; intervention/exposure desciption; comparator description; time point; analysis sample (e.g., intention-to-treat, per-protocol); summary statistics (e.g., number of events and sample sizes of both intervention/exposure and comparator); effect measure (e.g., risk ratio, mean difference); effect estimates and measures of precision (e.g., 95% confidence interval) and location of data in the report; whether results were unadjusted or adjusted for covariates; covariates that were adjusted for (if applicable).

      2.3 Data analysis

      We used descriptive statistics to summarise the characteristics of SR protocols, SRs, index meta-analyses, and included studies. For categorical variables, we present frequencies, and percentages. For continuous variables, we report medians (with interquartile ranges [IQR]). We computed the frequencies and percentages for different types of eligibility criteria and decision rules used to select results, differences in eligibility criteria or decision rules between the SR protocol and the SR, and studies with different types of multiplicity of results. We also calculated risk differences (with 95% confidence intervals [CIs]) to examine whether the percentages of different types of eligibility criteria and decision rules and studies with different types of multiplicity of results, differed between meta-analyses of continuous outcomes, and of non-continuous outcomes. Risk differences were calculated using the method of Mee with the Miettinen and Nurminen modification. Analyses were undertaken using the statistical packages Stata (College Station, Tx), except for the calculaton of risk differences, which was undertaken using the library PropCIs [

      Scherer R, Scherer MR. Package ‘PropCIs’. 2018.

      ] in R (Vienna, Austria) [

      Team RC. R: a language and environment for statistical computing. 2013.

      ].
      For the analysis of the frequency of different types of multiplicity of results, we restricted the sample to systematic reviews without any pre-specified decision rules to select results to include in meta-analyses. These reviews, and the studies within, were used to estimate the extent of multiplicity of results that can be expected when no decision rules to select results have been implemented.

      3. Results

      3.1 Results of search and screening

      Our search yielded a total of 7,167 references from the PubMed and Epistemonikos databases (Fig. 1). After removing duplicates (n = 908), we screened the titles and abstracts of the first 3,013 randomly sorted references, of which 2,777 were excluded, leaving 236 for full-text screening. Of these, 99 SRs met the inclusion criteria, including 25 SRs with a meta-analysis of a continuous outcome and 74 with meta-analysis of a non-continuous outcome. Initially, all SRs with a continuous outcome were included, and 25 of the SRs with a non-continuous outcome were randomly selected. From these 50 SRs, eight were excluded (6 continuous, 2 non-continuous), because the index meta-analysis had 20 or more studies, leaving 42 included SRs [
      • Ayoub-Charette S
      • Liu Q
      • Khan TA
      • Au-Yeung F
      • Mejia SB
      • de Souza RJ
      • et al.
      Important food sources of fructose-containing sugars and incident gout: a systematic review and meta-analysis of prospective cohort studies.
      ,
      • Choo VL
      • Viguiliouk E
      • Mejia SB
      • Cozma AI
      • Khan TA
      • Ha V
      • et al.
      Food sources of fructose-containing sugars and glycaemic control: systematic review and meta-analysis of controlled intervention studies.
      ,
      • Cheng S
      • Zheng Q
      • Ding G
      • Li G.
      Mediterranean dietary pattern and the risk of prostate cancer: a meta-analysis.
      ,
      • Bermejo LM
      • López-Plaza B
      • Santurino C
      • Cavero-Redondo I
      • Gómez-Candela C.
      Milk and dairy product consumption and bladder cancer risk: a systematic review and meta-analysis of observational studies.
      ,
      • de Magalhães Cunha C
      • Costa PR
      • de Oliveira LP
      • Valterlinda AdO
      • Pitangueira JC
      • Oliveira AM.
      Dietary patterns and cardiometabolic risk factors among adolescents: systematic review and meta-analysis.
      ,
      • Eaton JC
      • Rothpletz-Puglia P
      • Dreker MR
      • Iannotti L
      • Lutter C
      • Kaganda J
      • et al.
      Effectiveness of provision of animal-source foods for supporting optimal growth and development in children 6 to 59 months of age.
      ,
      • George ES
      • Marshall S
      • Mayr HL
      • Trakman GL
      • Tatucu-Babet OA
      • Lassemillante A-CM
      • et al.
      The effect of high-polyphenol extra virgin olive oil on cardiovascular risk factors: a systematic review and meta-analysis.
      ,
      • Ghaedi E
      • Mohammadi M
      • Mohammadi H
      • Ramezani-Jolfaie N
      • Malekzadeh J
      • Hosseinzadeh M
      • et al.
      Effects of a Paleolithic diet on cardiovascular disease risk factors: a systematic review and meta-analysis of randomized controlled trials.
      ,
      • Haider LM
      • Schwingshackl L
      • Hoffmann G
      • Ekmekcioglu C.
      The effect of vegetarian diets on iron status in adults: a systematic review and meta-analysis.
      ,
      • Hou R
      • Wei J
      • Hu Y
      • Zhang X
      • Sun X
      • Chandrasekar EK
      • et al.
      Healthy dietary patterns and risk and survival of breast cancer: a meta-analysis of cohort studies.
      ,
      • Huang Y
      • Zheng S
      • Wang T
      • Yang X
      • Luo Q
      • Li H
      Effect of oral nut supplementation on endothelium-dependent vasodilation–a meta-analysis.
      ,
      • Iguacel I
      • Miguel-Berges ML
      • Gómez-Bruton A
      • Moreno LA
      • Julián C.
      Veganism, vegetarianism, bone mineral density, and fracture risk: a systematic review and meta-analysis.
      ,
      • Kang K
      • Sotunde OF
      • Weiler HA.
      Effects of milk and milk-product consumption on growth among children and adolescents aged 6–18 years: a meta-analysis of randomized controlled trials.
      ,
      • Kibret KT
      • Chojenta C
      • Gresham E
      • Tegegne TK
      • Loxton D.
      Maternal dietary patterns and risk of adverse pregnancy (hypertensive disorders of pregnancy and gestational diabetes mellitus) and birth (preterm birth and low birth weight) outcomes: a systematic review and meta-analysis.
      ,
      • Kodama S
      • Horikawa C
      • Fujihara K
      • Ishii D
      • Hatta M
      • Takeda Y
      • et al.
      Relationship between intake of fruit separately from vegetables and triglycerides-A meta-analysis.
      ,
      • Kojima G
      • Avgerinou C
      • Iliffe S
      • Walters K.
      Adherence to Mediterranean diet reduces incident frailty risk: systematic review and meta-analysis.
      ,
      • Larsson SC
      • Drca N
      • Jensen-Urstad M
      • Wolk A.
      Chocolate consumption and risk of atrial fibrillation: two cohort studies and a meta-analysis.
      ,
      • Li R
      • Yu K
      • Li C.
      Dietary factors and risk of gout and hyperuricemia: a meta-analysis and systematic review.
      ,
      • Li W
      • Ruan W
      • Peng Y
      • Wang D.
      Soy and the risk of type 2 diabetes mellitus: a systematic review and meta-analysis of observational studies.
      ,
      • Li L
      • Lietz G
      • Seal C.
      Buckwheat and CVD risk markers: a systematic review and meta-analysis.
      ,
      • Lopez PD
      • Cativo EH
      • Atlas SA
      • Rosendorff C.
      The effect of vegan diets on blood pressure in adults: a meta-analysis of randomized controlled trials.
      ,
      • Maki KC
      • Palacios OM
      • Koecher K
      • Sawicki CM
      • Livingston KA
      • Bell M
      • et al.
      The relationship between whole grain intake and body weight: results of meta-analyses of observational studies and randomized controlled trials.
      ,
      • Malmir H
      • Saneei P
      • Larijani B
      • Esmaillzadeh A.
      Adherence to Mediterranean diet in relation to bone mineral density and risk of fracture: a systematic review and meta-analysis of observational studies.
      ,
      • Matía-Martín P
      • Torrego-Ellacuría M
      • Larrad-Sainz A
      • Fernández-Pérez C
      • Cuesta-Triana F
      • Rubio-Herrera MÁ.
      Effects of milk and dairy products on the prevention of osteoporosis and osteoporotic fractures in Europeans and non-Hispanic Whites from North America: a systematic review and updated meta-analysis.
      ,
      • Mena-Sánchez G
      • Becerra-Tomás N
      • Babio N
      • Salas-Salvadó J.
      Dairy product consumption in the prevention of metabolic syndrome: a systematic review and meta-analysis of prospective cohort studies.
      ,
      • Milajerdi A
      • Namazi N
      • Larijani B
      • Azadbakht L.
      The association of dietary quality indices and cancer mortality: a systematic review and meta-analysis of cohort studies.
      ,
      • Mishali M
      • Prizant-Passal S
      • Avrech T
      • Shoenfeld Y
      Association between dairy intake and the risk of contracting type 2 diabetes and cardiovascular diseases: a systematic review and meta-analysis with subgroup analysis of men versus women.
      ,
      • Mohseni R
      • Abbasi S
      • Mohseni F
      • Rahimi F
      • Alizadeh S
      Association between dietary inflammatory index and the risk of prostate cancer: a meta-analysis.
      ,
      • Musa-Veloso K
      • Poon T
      • Harkness LS
      • O'Shea M
      • Chu Y
      The effects of whole-grain compared with refined wheat, rice, and rye on the postprandial blood glucose response: a systematic review and meta-analysis of randomized controlled trials.
      ,
      • Namazi N
      • Larijani B
      • Azadbakht L
      Dietary inflammatory index and its association with the risk of cardiovascular diseases, metabolic syndrome, and mortality: a systematic review and meta-analysis.
      ,
      • Picasso MC
      • Lo-Tayraco JA
      • Ramos-Villanueva JM
      • Pasupuleti V
      • Hernandez AV.
      Effect of vegetarian diets on the presentation of metabolic syndrome or its components: a systematic review and meta-analysis.
      ,
      • Qin Z-Z
      • Xu J-Y
      • Chen G-C
      • Ma Y-X
      • Qin L-Q.
      Effects of fatty and lean fish intake on stroke risk: a meta-analysis of prospective cohort studies.
      ,
      • Rees K
      • Takeda A
      • Martin N
      • Ellis L
      • Wijesekara D
      • Vepa A
      • et al.
      Mediterranean-style diet for the primary and secondary prevention of cardiovascular disease.
      ,
      • Ren Y
      • Liu Y
      • Sun X-Z
      • Wang B-Y
      • Zhao Y
      • Liu D-C
      • et al.
      Chocolate consumption and risk of cardiovascular diseases: a meta-analysis of prospective studies.
      ,
      • Shab-Bidar S
      • Golzarand M
      • Hajimohammadi M
      • Mansouri S.
      A posteriori dietary patterns and metabolic syndrome in adults: a systematic review and meta-analysis of observational studies.
      ,
      • Shafiei F
      • Salari-Moghaddam A
      • Larijani B
      • Esmaillzadeh A.
      Adherence to the Mediterranean diet and risk of depression: a systematic review and updated meta-analysis of observational studies.
      ,
      • Teoh SL
      • Lai NM
      • Vanichkulpitak P
      • Vuksan V
      • Ho H
      • Chaiyakunapruk N.
      Clinical evidence on dietary supplementation with chia seed (Salvia hispanica L.): a systematic review and meta-analysis.
      ,
      • Voon PT
      • Lee ST
      • Ng TKW
      • Ng YT
      • Yong XS
      • Lee VKM
      • et al.
      Intake of palm olein and lipid status in healthy adults: a meta-analysis.
      ,
      • Wang L
      • Liu C
      • Zhou C
      • Zhuang J
      • Tang S
      • Yu J
      • et al.
      Meta-analysis of the association between the dietary inflammatory index (DII) and breast cancer risk.
      ,
      • Xiao Y
      • Ke Y
      • Wu S
      • Huang S
      • Li S
      • Lv Z
      • et al.
      Association between whole grain intake and breast cancer risk: a systematic review and meta-analysis of observational studies.
      ,
      • Xu Y
      • Yang J
      • Du L
      • Li K
      • Zhou Y.
      Association of whole grain, refined grain, and cereal consumption with gastric cancer risk: a meta-analysis of observational studies.
      ,
      • Zhang Z
      • Chen G-C
      • Qin Z-Z
      • Tong X
      • Li D-P
      • Qin L-Q.
      Poultry and fish consumption in relation to total cancer mortality: a meta-analysis of prospective studies.
      ].
      Fig 1
      Fig. 1PRISMA 2020 flow diagram of identification, screening and inclusion of systematic reviews.
      *Of the 6,259 unique titles and abstracts, we only needed to screen 3,013 randomly sorted titles and abstracts to reach our target sample size. **We initially drew a random sample of 50 systematic reviews, but post-hoc excluded eight systematic reviews with 20 or more included studies in the index meta-analysis to reduce workload.

      3.2 Characteristics of the included systematic reviews and their index meta-analyses

      Of the 42 SRs, 14 had accessible protocols (2 were published protocols and 12 were PROSPERO records) (Table 2). In most index meta-analyses the population was unclear (45%, 19 of 42), 33% (14 of 42) included only healthy participants and the remainder (22%, 9 of 42) included a mix of healthy people and people with a health condition. Most index meta-analyses included only non-randomized studies (62%, 26 of 42), 33% (14 of 42) included only randomized trials and the remaining two (5%) included studies of both designs (Table 3). Of the 19 meta-analyses of a continuous outcome, 14 included randomized trials only, three included non-randomized studies only, and two included both designs. Of the 23 meta-analyses of a non-continuous outcome, all included non-randomized studies only. The primacy of the outcome was not identified in most reviews (81%, 34 of 42). In nearly all meta-analyses, a random-effects model was fitted (90%, 38 of 42). The 42 index meta-analyses included a total of 325 studies (104 randomized, 221 non-randomized), with a median of seven studies (IQR 5-11; range 2-17) per meta-analysis.
      Table 2Characteristics of the systematic reviews (N = 42)
      Characteristicn (%)
      Focus of journal
      Restricted to nutrition research28 (67)
      Not restricted to nutrition research14 (33)
      Country of the corresponding author(s)
      China9 (21)
      Iran7 (17)
      United States of America6 (14)
      Others (Australia, Austria, Brazil, Canada, Israel, Japan, Malaysia, Spain, Sweden, Thailand, United Kingdon)20 (48)
      Affiliation of the corresponding author(s)
      Food industry2 (5)
      Non-industry37 (88)
      Mixed2 (5)
      Unclear1 (2)
      Source of funding
      Non-profit23 (55)
      For-profit3 (7)
      Mixed0
      No funding8 (19)
      Not reported8 (19)
      Conflict of interest
      Conflict of interest reported by at least one review author7 (17)
      All review authors stated they had no conflicts of interest29 (69)
      Missing/not reported6 (14)
      Protocol availability
      Both a protocol and registration record are available0
      Only a protocol is available2 (5)
      Only a registration record is available12 (29)
      Neither are available28 (67)
      Protocol published year
      Only two protocols published
      2012 & 2017
      Protocol registered year (median, [IQR])2018 (2017–2018)
      a Only two protocols published
      Table 3Characteristics of index meta-analyses (N = 42)
      Characteristicsn (%)
      Type of participants in included studies
      Healthy only14 (33)
      Mix of healthy people and people with a health condition9 (21)
      Unclear19 (45)
      Type of included studies
      Only randomized trials14 (33)
      Only non-randomized trials26 (62)
      Both2 (5)
      Total number of studies included (median [IQR])7 (5-11)
      Total number of participants (median [IQR])2,972 (857–44418)
      Only 17 of the 42 SRs reported the total number of participants included in the index meta-analysis
      Outcome labelling
      Primary6 (14)
      Secondary2 (5)
      Unlabelled34 (81)
      Outcome type
      Continuous19 (45)
      Non-continuous (e.g., binary, count, time-to-event)23 (55)
      Meta-analytic effect measure
      Risk ratio15 (36)
      Odds ratio6 (14)
      Hazard ratio2 (5)
      Mean difference18 (43)
      Standardised mean difference1 (2)
      Index meta-analysis model used
      Fixed-effect2 (5)
      Random-effects38 (90)
      Unclear2 (5)
      Type of intervention/exposure
      Dairy5 (12)
      Fruits2 (5)
      Pescatarian diet2 (5)
      Vegan diet1 (2)
      Vegetarian diet12 (28)
      Mediterranean diet5 (12)
      Non-vegetarian diet1 (2)
      Chocolates2 (5)
      Mixed
      Includes combinations of fruits, oils, grains, meat, egg, milk, fish, and vegetables etc.
      12 (28)
      a Only 17 of the 42 SRs reported the total number of participants included in the index meta-analysis
      b Includes combinations of fruits, oils, grains, meat, egg, milk, fish, and vegetables etc.

      3.3 Eligibility criteria and decision rules reported in SR protocols

      Of the SR protocols (n = 14), all included at least one eligibility criterion, and four (29%) reported at least one decision rule to select results (Table 4). Almost all protocols specified eligibility criteria based on interventions/exposures (93%, 13 of 14) (e.g., specifying which foods or dietary patterns were eligible), but few other types of eligibility criteria were specified (e.g., time points [29%, 4 of 14], information sources [13%, 3 of 14]). The most commonly pre-specified decision rule was based on interventions/exposures (reported in 3 of the 4 SR protocols with at least one decision rule to select results). See Supplementary Table S3 for the content of the decision rules.
      Table 4Number of systematic review protocols or registration entries reporting eligibility criteria and decision rules to select results (N = 14)
      CriteriaTotal n (%)

      (n = 14)
      SRs of continuous outcomes n (%) (n = 9)SRs of non-continuous outcomes n (%) (n = 5)Risk difference
      Risk difference calculated as the difference in percentage of SRs reporting the specified eligibility criteria/decision rule between SRs of continuous outcomes minus SRs of non-continuous outcomes. ITT, Intention to treat; NA, Not applicable;
      95% confidence interval
      Confidence limits for the difference in percentages calculated using the method of Mee with the Miettinen and Nurminen modification
      Total
      At least one eligibility criterion14 (100)9 (100)5 (100)0 (-31, 45)
      At least one decision rule4 (29)2 (22)2 (40)-18 (-62, 30)
      Measurement instruments
      Eligibility criteria1 (7)1 (11)011 (-36, 45)
      Decision rule0000 (-45, 31)
      Definitions/diagnostic criteria
      Eligibility criteria0000 (-45, 31)
      Decision rule0000 (-45, 31)
      Cut-points on a measurement instrument
      Eligibility criteria0000 (-45, 31)
      Decision rule0000 (-45, 31)
      Time points
      Eligibility criteria4 (29)2 (22)2 (40)-18 (-63, 30)
      Decision rule0000 (-45, 31)
      Interventions/exposures
      Eligibility criteria13 (93)9 (100)4 (80)20 (-16, 63)
      Decision rule3 (21)2 (22)1 (20)2 (-47, 43)
      Information sources
      Eligibility criteria3 (21)2 (22)1 (20)2 (-47, 43)
      Decision rule1 (7)01 (20)-20 (-63, 16)
      Analyses
      Eligibility criteria for any type of analysis2 (14)2 (22)022 (-27, 56)
      Decision rule for any type of analysis2 (14)02 (40)-40 (-78, 0)
      Rule for final vs. change from baseline values00NANA
      Rule for analyses undertaken on multiple samples (e.g., ITT vs. per-protocol)0000 (-45, 31)
      Rule for unadjusted vs. covariate-adjusted analyses1 (7)01 (20)-20 (-63, 16)
      Rule for period vs. paired analyses in crossover randomized trials0000 (-45, 31)
      Rule to handle results arising from overlapping samples of participants1 (7)01 (20)-20 (-63, 16)
      Other decision rule0000 (-45, 31)
      a Risk difference calculated as the difference in percentage of SRs reporting the specified eligibility criteria/decision rule between SRs of continuous outcomes minus SRs of non-continuous outcomes. ITT, Intention to treat; NA, Not applicable;
      b Confidence limits for the difference in percentages calculated using the method of Mee with the Miettinen and Nurminen modification

      3.4 Eligibility criteria and decision rules reported in SRs

      Of the SRs (n = 42), all included at least one eligibility criterion, and 69% reported at least one decision rule to select results (Table 5). Similar to the SR protocols, the most commonly reported eligibility criteria (95%, 40 of 42), and decision rule (40%, 17 of 42) in the SRs was based on interventions/exposures. Eligibility criteria and decision rules for the type of analysis were more freqently specified in SRs as compared with their protocols. The most commonly reported decision rules for analyses were rules to select from multiple unadjusted and covariate-adjusted analyses (24%) (Table 5). There were some discrepancies observed between SR protocols and their published SRs, with the most common type being the addition of a new decision rule to deal with multiple unadjusted and covariate-adjusted analyses in the included studies (Supplementary Table S4).
      Table 5Number of systematic reviews reporting eligibility criteria and decision rules to select results
      CriteriaTotal SRs n (%)

      (n = 42)
      SRs of continuous outcomes n (%) (n = 19)SRs of non-continuous outcomes n (%) (n = 23)Risk difference
      Risk difference calculated as the difference in percentage of SRs reporting the specified eligibility criteria/decision rule between SRs of continuous outcomes minus SRs of non-continuous outcomes. ITT, Intention to treat; NA, Not applicable;
      95% Confidence interval
      Confidence limits for the difference in percentages calculated using the method of Mee with the Miettinen and Nurminen modification
      Total
      At least one eligibility criterion42 (100)19 (100)23 (100)0 (-17, 14)
      At least one decision rule29 (69)14 (74)15 (65)9 (-20, 35)
      Measurement instruments
      Eligibility criteria1 (2)1 (5)05 (-9, 24)
      Decision rule2 (5)1 (5)1 (4)1 (-17, 21)
      Definitions/diagnostic criteria
      Eligibility criteria2 (5)02 (9)-9 (-27, 9)
      Decision rule1 (2)01 (4)-4 (-21, 13)
      Cut-points on a measurement instrument
      Eligibility criteria0000 (-15, 17)
      Decision rule0000 (-15, 17)
      Time points
      Eligibility criteria4 (10)2 (11)2 (9)2 (-18, 24)
      Decision rule4(10)3 (16)1 (4)12 (-8, 34)
      Interventions/exposures
      Eligibility criteria40 (95)19 (100)21 (91)9 (-9, 27)
      Decision rule17 (40)6 (32)11 (48)-16 (-43, 14)
      Information sources
      Eligibility criteria3 (7)1 (5)2 (9)-4 (-23, 17)
      Decision rule4 (10)3 (16)1 (4)12 (-8, 34)
      Analyses
      Eligibility criteria for any type of analysis11 (26)7 (37)4 (17)20 (-7, 45)
      Decision rule for any type of analysis16 (38)8 (42)8 (35)7 (-21, 35)
      Rule for final vs. change from baseline values3 (7)3 (16)0NA
      Rule for analyses undertaken on multiple samples (e.g., ITT vs. per-protocol)0000 (-15, 17)
      Rule for unadjusted vs. covariate-adjusted analyses10 (24)2 (11)8 (35)-24 (-47, 2)
      Rule for period vs. paired analyses in crossover randomized trials2 (5)2 (11)011 (-5, 31)
      Rule to handle results arising from overlapping samples of participants1 (2)01 (4)-4 (-21, 13)
      Other decision rule5 (12)1 (5)4 (17)-12 (-33, 10)
      a Risk difference calculated as the difference in percentage of SRs reporting the specified eligibility criteria/decision rule between SRs of continuous outcomes minus SRs of non-continuous outcomes. ITT, Intention to treat; NA, Not applicable;
      b Confidence limits for the difference in percentages calculated using the method of Mee with the Miettinen and Nurminen modification
      The percentage of reviews specifying different types of eligibility criteria and decision rules generally did not differ by outcome type. However, a larger percentage of SRs with an index meta-analysis of a continuous outcome specified eligibility criteria for any type of analysis compared with SRs with a non-continuous outcome (37% vs. 17%; risk difference [RD] 20%, 95% CI -7% to 45%). Conversely, a smaller percentage of SRs with an index meta-analysis of a continuous outcome specified a rule for selecting an adjusted/unadjusted result compared with SRs of a non-continuous outcome (11% vs. 35%; RD -24%, 95% CI -47% to 2%).

      3.5 Multiplicity of results in included studies

      Of the 325 studies, 296 studies were included in reviews (n = 38) without any pre-specified decision rules to select results to include in meta-analyses (Table 6). These reviews, and the studies within, are therefore used to estimate the extent of multiplicity of results that can be expected when no decision rules to select results have been implemented. The median (IQR) number of available effect estimates per study was 2 (1 to 4), and the largest number of effect estimates in a study was 41 [
      • Adjibade M
      • Assmann KE
      • Andreeva VA
      • Lemogne C
      • Hercberg S
      • Galan P
      • et al.
      Prospective association between adherence to the Mediterranean diet and risk of depressive symptoms in the French SU.VI.MAX cohort.
      ]. The most common types of multiplicity arose from multiple unadjusted and one or more covariate-adjusted analyses (which occurred in 39% of the included studies), followed by multiple intervention/control groups (24%). The least common types of multiplicity arose from multiple instruments (0%). The studies with continuous outcomes had less multiplicity than the studies with non-continuous outcomes (53% vs. 80%; RD -27%, 95% CI -37%, -16%).
      Table 6Number of studies with different types of multiplicity of results in systematic reviews without pre-specified decision rules to select results (N = 38)
      Type of multiplicityTotal n (%)

      (n = 296)
      Continuous outcomes n (%) (n = 125)Non-continuous outcomes n (%) (n = 171)Risk difference 95% confidence interval
      Confidence limits for the difference in percentages calculated using the method of Mee with the Miettinen and Nurminen modification; ITT, Intention to treat; NA, Not applicable;
      Any202 (68)66 (53)136 (80)-27 (-37, -16)
      Instruments0000 (-2, 3)
      Intervention/control groups70 (24)23 (18)47 (27)-9 (-18, 7)
      Time points14 (5)13 (10)1 (1)9 (5, 16)
      Final and change from baseline values18 (6)18 (14)NANA
      Analyses undertaken on multiple samples (e.g., ITT and per-protocol)2 (1)2 (2)02 (-1, 6)
      Unadjusted and one or more covariate-adjusted analyses115 (39)4 (3)111 (65)-62 (-69, -53)
      Period and paired analyses in crossover randomized trials0000 (-2, 3)
      Definitions of an event4 (1)04 (2)-2 (-6, 1)
      Subgroups24 (8)8 (6)16 (9)-3 (-9, 4)
      Other
      Multiple information sources and sample sizes
      12 (4)7 (6)5 (3)3 (-2, 8)
      a Confidence limits for the difference in percentages calculated using the method of Mee with the Miettinen and Nurminen modification; ITT, Intention to treat; NA, Not applicable;
      b Multiple information sources and sample sizes

      4. Discussion

      Our findings show that decision rules to select results were less frequently pre-specified in the protocols of a randomly selected sample of SRs in nutrition research. Multiplicity of results in the primary studies included in the index meta-analyses was common, with 68% having at least one type of multiplicity. The frequency and types of multiplicity in the included studies varied, arising from multiple intervention groups, time points, analyses, and subgroups.

      Comparison with previous research

      The findings of our study are in line with previously published studies, which have observed incomplete pre-specification of SR methods such as eligibility criteria, methods for collecting, handling and analysing data, and pre-specification of eligibility criteria and decision rules for selecting results in PROSPERO records [
      • Page MJ
      • Shamseer L
      • Tricco AC.
      Registration of systematic reviews in PROSPERO: 30,000 records and counting.
      ], Cochrane protocols [
      • Page MJ
      • McKenzie JE
      • Chau M
      • Green SE
      • Forbes A
      Methods to select results to include in meta-analyses deserve more consideration in systematic reviews.
      ,
      • Liu Z
      • Saldanha IJ
      • Margolis D
      • Dumville JC
      • Cullum NA.
      Outcomes in Cochrane systematic reviews related to wound care: an investigation into prespecification.
      ] and published SRs [
      • Page MJ
      • McKenzie JE
      • Chau M
      • Green SE
      • Forbes A
      Methods to select results to include in meta-analyses deserve more consideration in systematic reviews.
      ,
      • Saldanha IJ
      • Dickersin K
      • Wang X
      • Li T.
      Outcomes in Cochrane systematic reviews addressing four common eye conditions: an evaluation of completeness and comparability.
      ]. Zeraatkar et al. [
      • Zeraatkar D
      • Bhasin A
      • Morassut RE
      • Churchill I
      • Gupta A
      • Lawson DO
      • et al.
      Characteristics and quality of systematic reviews and meta-analyses of observational nutritional epidemiology: a cross-sectional study.
      ] examined the conduct and reporting of 150 systematic reviews of observational nutrition studies published from 2018-2019, identifying several flaws in the conduct and reporting, and also recommended that pre-defined rules should be specified for selecting one estimate from each study for inclusion in a particular meta-analysis.
      Two previous studies, on which the methodology of the present study is based, examined methods used to select results for inclusion in meta-analyses [
      • Tendal B
      • Nuesch E
      • Higgins JP
      • Juni P
      • Gotzsche PC.
      Multiplicity of data in trial reports and the reliability of meta-analyses: empirical study.
      ,
      • Page MJ
      • McKenzie JE
      • Chau M
      • Green SE
      • Forbes A
      Methods to select results to include in meta-analyses deserve more consideration in systematic reviews.
      ]. Page et al. [
      • Page MJ
      • McKenzie JE
      • Chau M
      • Green SE
      • Forbes A
      Methods to select results to include in meta-analyses deserve more consideration in systematic reviews.
      ] examined 44 SRs, nearly half of which were Cochrane reviews, and they included only randomized trials. In Page et al. [
      • Page MJ
      • McKenzie JE
      • Chau M
      • Green SE
      • Forbes A
      Methods to select results to include in meta-analyses deserve more consideration in systematic reviews.
      ] and our study, all protocols reported at least one eligibility criteria, but at least one decision rule was pre-specified more in the SR protocols included in Page et al. [
      • Page MJ
      • McKenzie JE
      • Chau M
      • Green SE
      • Forbes A
      Methods to select results to include in meta-analyses deserve more consideration in systematic reviews.
      ] than those in our study (81% vs. 29%). The frequency of discrepancies between the SR protocol and SR in the eligibility criteria or decision rules to select results was higher in our study. Tendal et al. [
      • Tendal B
      • Nuesch E
      • Higgins JP
      • Juni P
      • Gotzsche PC.
      Multiplicity of data in trial reports and the reliability of meta-analyses: empirical study.
      ] examined eighteen Cochrane reviews, all of which had protocols. Eight of the protocols mentioned eligible time points or periods, but only one provided decision rules to handle multiplicity of time points. Interestingly, all of the 18 protocols reported eligibility criteria for the control group, but none reported decision rules to handle multiple control groups in included studies.
      Three studies (Tendal et al. [
      • Tendal B
      • Nuesch E
      • Higgins JP
      • Juni P
      • Gotzsche PC.
      Multiplicity of data in trial reports and the reliability of meta-analyses: empirical study.
      ], Mayo-Wilson et al. [
      • Mayo-Wilson E
      • Fusco N
      • Li T
      • Hong H
      • Canner JK
      • Dickersin K
      Multiple outcomes and analyses in clinical trials create challenges for interpretation and research synthesis.
      ], and Page et al. [
      • Page MJ
      • McKenzie JE
      • Chau M
      • Green SE
      • Forbes A
      Methods to select results to include in meta-analyses deserve more consideration in systematic reviews.
      ]) assessed the multiplicity of results among the included studies of SRs. Compared to our study, Page et al. [
      • Page MJ
      • McKenzie JE
      • Chau M
      • Green SE
      • Forbes A
      Methods to select results to include in meta-analyses deserve more consideration in systematic reviews.
      ], and Tendal et al. [
      • Tendal B
      • Nuesch E
      • Higgins JP
      • Juni P
      • Gotzsche PC.
      Multiplicity of data in trial reports and the reliability of meta-analyses: empirical study.
      ] had fewer studies with multiple estimates that were available for inclusion in a particular meta-analysis. Multiplicity arising from multiple intervention/control groups was slightly less frequent in our study compared to Tendal et al. [
      • Tendal B
      • Nuesch E
      • Higgins JP
      • Juni P
      • Gotzsche PC.
      Multiplicity of data in trial reports and the reliability of meta-analyses: empirical study.
      ] (24% vs. 29%) but was more frequent in ours when compared with Page et al. [
      • Page MJ
      • McKenzie JE
      • Chau M
      • Green SE
      • Forbes A
      Methods to select results to include in meta-analyses deserve more consideration in systematic reviews.
      ] (24% vs. 17%). Our study findings showed less multiplicity in terms of time points (5%) and measurement instruments for outcomes (0%) compared to previous studies [
      • Page MJ
      • McKenzie JE
      • Chau M
      • Green SE
      • Forbes A
      Methods to select results to include in meta-analyses deserve more consideration in systematic reviews.
      ,
      • Tendal B
      • Nuesch E
      • Higgins JP
      • Juni P
      • Gotzsche PC.
      Multiplicity of data in trial reports and the reliability of meta-analyses: empirical study.
      ]. However, we observed greater multiplicity (39%) due to unadjusted, and one or more covariate-adjusted analyses. This difference was likely driven by inclusion of non-randomized studies in the present study, which often require adjustment for covariates to reduce risk of bias due to confounding [
      • Trutschel D
      • Palm R
      • Holle B
      • Simon M.
      Methodological approaches in analysing observational data: a practical example on how to address clustering and selection bias.
      ]. Similarly, Mayo-Wilson et al. [
      • Mayo-Wilson E
      • Fusco N
      • Li T
      • Hong H
      • Canner JK
      • Dickersin K
      Multiple outcomes and analyses in clinical trials create challenges for interpretation and research synthesis.
      ] assessed multiplicity in the clinical trials of publicly accessible reports and non-public reports (e.g., unpublished Clinical Study Reports) related to gabapentin for neuropathic pain (n = 21) and quetiapine for bipolar depression (n = 7). In 15 of 21 (71%) gabapentin trials and 7 of 7 (100%) quetiapine trials, there was multiplicity of results.

      Strengths and limitations

      The major strength of our study is that we have pre-specified methods to identify, select and collect data from eligible SRs and studies, and provided the modifications/deviations to our study protocol (supplementary Table S1). We also used extensive search strategies to identify SRs in nutrition research. In addition, all the study authors, who had different levels of expertise, undertook training and pilot-testing of data collection forms. Moreover, given we randomly selected the SRs, our findings are generalisable to SRs meeting this study's eligibility criteria.
      A limitation of our study is that we only retrieved reports of studies included in the index meta-analyses that the systematic reviewers cited. It is possible that other papers relating to the studies exist, which contain additional results that are compatible with the index meta-analysis (e.g., results of cohort studies or randomized trials at later time points may have been presented in other papers which were not cited by the systematic reviewers). For this reason, we may have underestimated the true extent of multiplicity of results within studies. Furthermore, our focus on SRs of food/diet-outcome associations means our results are not generalisable to other types of SRs of nutrition research. For example, it is possible that evidence of multiplicity is more apparent in studies included in SRs of nutrient-specific associations with outcomes, because there are more methods to measure nutrient-specific exposures (e.g., supplement intake, biomarkers) than foods and dietary patterns, which are typically measured via food frequency questionnaires. We made a post-hoc decision to restrict our sample to systematic reviews with fewer than 20 studies included in the index meta-analyses, for reasons of feasibility. However, we have no reason to believe that the extent, and types of multiplicity would differ by the number of included studies in a review. We were unable to translate and interpret the data from four included non-English language studies however, their absence is unlikely to have modified the observed extent of multiplicity. Finally, we only searched PubMed and Epistemonikos to find all SRs and included SRs written in English, so generalisation of our study findings to non-indexed SRs and non-English language SRs is potentially limited.

      Implications of this research for practice

      Our study, in common with previous research [
      • Page MJ
      • McKenzie JE
      • Chau M
      • Green SE
      • Forbes A
      Methods to select results to include in meta-analyses deserve more consideration in systematic reviews.
      ,
      • Tendal B
      • Nuesch E
      • Higgins JP
      • Juni P
      • Gotzsche PC.
      Multiplicity of data in trial reports and the reliability of meta-analyses: empirical study.
      ,
      • Mayo-Wilson E
      • Fusco N
      • Li T
      • Hong H
      • Canner JK
      • Dickersin K
      Multiple outcomes and analyses in clinical trials create challenges for interpretation and research synthesis.
      ], demonstrates that multiplicity of effect estimates in primary studies is very common in nutrition research, and is therefore an issue that systematic reviewers should prepare for when designing their review. Doing so will have multiple benefits. It will reduce post-hoc decision making, and in doing-so, provide greater assurance to a reader that the results have not been “cherry-picked” for inclusion in the meta-analyses. Furthermore, specification of eligibility criteria, and decision rules is likely to lessen the data extraction effort, requiring less information to be extracted from each primary study.
      Our results suggest that in nutrition research, specification of eligibility criteria, and decision rules to select results from among multiple unadjusted or covariate-adjusted analyses are most important to pre-specify in SRs that include non-randomized studies with non-continuous outcomes. On the other hand, methods used to select results arising from both final values and change from baseline values, and multiple time points, are most important to pre-specify in SRs that include randomized trials with continuous outcomes. Furthermore, in the SR, we recommend reporting any modifications to the specified rules, or any additions that were introduced to cover multiplicity scenarios that had not been anticipated when designing the review.
      The Cochrane Handbook for Systematic Reviews of Interventions version 6.2 [

      McKenzie JE, Brennan SE, Ryan RE, Thomson HJ, Johnston RV, Thomas J. Chapter 3: Defining the criteria for including studies and how they will be grouped for the synthesis. In: Higgins J.P.T., Thomas J., Chandler J., Cumpston M., Li T., Page M.J., Welch V.A. (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). Cochrane, 2021. Accessed on 1 December 2021 from: www.training.cochrane.org/handbook.

      ] provides updated guidance to systematic reviewers about how to group interventions with multiple components or co-interventions or how to select from multiple comparisons and handle multiplicity of outcomes when conducting meta-analyses. In addition, the recently updated Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement [
      • Page MJ
      • Moher D
      • Bossuyt P
      • Boutron I
      • Hoffmann T
      • Mulrow C
      • et al.
      PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews.
      ] includes a new item (10a), which recommends authors “List and define all outcomes for which data were sought. Specify whether all results that were compatible with each outcome domain in each study were sought (e.g., for all measures, time points, analyses), and if not, the methods used to decide which results to collect.” Implementation of recommendations from these sources will allow readers to understand the result selection process.

      5. Conclusion

      Our study found that in systematic reviews examining the effects of foods and diet, that multiplicity of results in the included primary studies was common. Yet, pre-specification of decision rules to select from multiple results was lacking. Systematic reviewers are encouraged to consider methods for dealing with multiplicity when designing their reviews. Doing so will limit the potential for selective inclusion of results, thus providing greater assurance to readers as to the trustworthiness of the review.

      6. Contributors

      All authors declare to meet the ICMJE conditions for authorship. MJP and JEM conceived the project. MJP, JEM, LB, ZD, SM, and CMK contributed to the design of the project. MJP, ZD, SM, and CMK screened articles for inclusion. RK, MJP, LB, ZD, SM, CMK, and EK collected data. RK analysed the data. RK wrote the first draft of the manuscript, which was revised in conjunction with MJP and JEM. JEM drafted sections of the manuscript. All authors were involved in revising the article critically for important intellectual content. All authors approved the final version of the article. MJP is the guarantor of this work.

      Funding

      This project was funded by an Australian National Health and Medical Research Council (NHMRC) project grant ( APP1139997 ). RK is supported by a Monash Graduate Scholarship and a Monash International Tuition Scholarship . MJP is supported by an Australian Research Council Discovery Early Career Researcher Award ( DE200101618 ). JEM is supported by an Australian NHMRC Career Development Fellowship ( 1143429 ). SM is supported by the Country Women's Association (NSW) and Edna Winifred Blackman Postgraduate Research Scholarship . The funders had no role in the study design, data collection and analysis, or preparation of the manuscript.

      Appendix. Supplementary materials

      References

        • GBD 2019 Risk Factors Collaborators
        Global burden of 87 risk factors in 204 countries and territories, 1990-2019: a systematic analysis for the global burden of disease study 2019.
        Lancet. 2020; 396: 1223-1249
        • Estruch R
        • Ros E
        • Salas-Salvadó J
        • Covas MI
        • Corella D
        • Arós F
        • et al.
        Primary prevention of cardiovascular disease with a mediterranean diet supplemented with extra-virgin olive oil or nuts.
        N Engl J Med. 2018; 378: e34
        • Taylor MK
        • Mahnken JD
        • Sullivan DK.
        NHANES 2011-2014 reveals cognition of US older adults may benefit from better adaptation to the mediterranean diet.
        Nutrients. 2020; 12: 1979
        • Dong X
        • Li S
        • Chen J
        • Li Y
        • Wu Y
        • Zhang D.
        Association of dietary ω-3 and ω-6 fatty acids intake with cognitive performance in older adults: national health and nutrition examination survey (NHANES) 2011-2014.
        Nutr J. 2020; 19: 25
        • Kelley GA
        • Kelley KS.
        Systematic reviews and meta-analysis in nutrition research.
        Br J Nutr. 2019; 122: 1279-1294
        • Blake P
        • Durão S
        • Naude CE
        • Bero L.
        An analysis of methods used to synthesize evidence and grade recommendations in food-based dietary guidelines.
        Nutr Rev. 2018; 76: 290-300
        • Dai Z
        • Kroeger CM
        • Lawrence M
        • Scrinis G
        • Bero L.
        Comparison of methodological quality between the 2007 and 2019 Canadian dietary guidelines.
        Public Health Nutr. 2020; 23: 2879-2885
        • Schunemann HJ
        • Wiercioch W
        • Etxeandia I
        • Falavigna M
        • Santesso N
        • Mustafa R
        • et al.
        Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise.
        CMAJ. 2014; 186: E123-E142
        • Page MJ
        • McKenzie JE
        • Chau M
        • Green SE
        • Forbes A
        Methods to select results to include in meta-analyses deserve more consideration in systematic reviews.
        J Clin Epidemiol. 2015; 68: 1282-1291
        • Lopez-Lopez JA
        • Page MJ
        • Lipsey MW
        • Higgins JPT.
        Dealing with effect size multiplicity in systematic reviews and meta-analyses.
        Res Synth Methods. 2018; 9: 336-351
        • Page MJ
        • Forbes A
        • Chau M
        • Green SE
        • McKenzie JE.
        Investigation of bias in meta-analyses due to selective inclusion of trial effect estimates: empirical study.
        BMJ open. 2016; 6: e011863
        • Tendal B
        • Nuesch E
        • Higgins JP
        • Juni P
        • Gotzsche PC.
        Multiplicity of data in trial reports and the reliability of meta-analyses: empirical study.
        BMJ. 2011; 343: d4829
        • Mayo-Wilson E
        • Fusco N
        • Li T
        • Hong H
        • Canner JK
        • Dickersin K
        Multiple outcomes and analyses in clinical trials create challenges for interpretation and research synthesis.
        J Clin Epidemiol. 2017; 86: 39-50
        • Page MJ
        • Bero L
        • Kroeger CM
        • Dai Z
        • McDonald S
        • Forbes A
        • et al.
        Investigation of risk of bias due to unreported and selectively included results in meta-analyses of nutrition research: the ROBUST study protocol.
        F1000Res. 2019; 8: 1760
        • Higgins JP
        • Thomas J
        • Chandler J
        • Cumpston M
        • Li T
        • Page MJ
        • et al.
        Cochrane handbook for systematic reviews of interventions.
        John Wiley & Sons, 2019
        • Rada G
        • Perez D
        • Araya-Quintanilla F
        • Avila C
        • Bravo-Soto G
        • Bravo-Jeria R
        • et al.
        Epistemonikos: a comprehensive database of systematic reviews for health decision-making.
        BMC Med Res Methodol. 2020; 20: 286
        • Harris PA
        • Taylor R
        • Minor BL
        • Elliott V
        • Fernandez M
        • O'Neal L
        • et al.
        The REDCap consortium: building an international community of software platform partners.
        J Biomed Inform. 2019; 95103208
      1. Scherer R, Scherer MR. Package ‘PropCIs’. 2018.

      2. Team RC. R: a language and environment for statistical computing. 2013.

        • Ayoub-Charette S
        • Liu Q
        • Khan TA
        • Au-Yeung F
        • Mejia SB
        • de Souza RJ
        • et al.
        Important food sources of fructose-containing sugars and incident gout: a systematic review and meta-analysis of prospective cohort studies.
        BMJ open. 2019; 9e024171
        • Choo VL
        • Viguiliouk E
        • Mejia SB
        • Cozma AI
        • Khan TA
        • Ha V
        • et al.
        Food sources of fructose-containing sugars and glycaemic control: systematic review and meta-analysis of controlled intervention studies.
        BMJ. 2018; 363: k4644
        • Cheng S
        • Zheng Q
        • Ding G
        • Li G.
        Mediterranean dietary pattern and the risk of prostate cancer: a meta-analysis.
        Medicine. 2019; 98: e16341
        • Bermejo LM
        • López-Plaza B
        • Santurino C
        • Cavero-Redondo I
        • Gómez-Candela C.
        Milk and dairy product consumption and bladder cancer risk: a systematic review and meta-analysis of observational studies.
        Adv Nutr. 2019; 10: S224-SS38
        • de Magalhães Cunha C
        • Costa PR
        • de Oliveira LP
        • Valterlinda AdO
        • Pitangueira JC
        • Oliveira AM.
        Dietary patterns and cardiometabolic risk factors among adolescents: systematic review and meta-analysis.
        Br J Nutr. 2018; 119: 859-879
        • Eaton JC
        • Rothpletz-Puglia P
        • Dreker MR
        • Iannotti L
        • Lutter C
        • Kaganda J
        • et al.
        Effectiveness of provision of animal-source foods for supporting optimal growth and development in children 6 to 59 months of age.
        Cochrane Database Syst Rev. 2019; 2: CD012818
        • George ES
        • Marshall S
        • Mayr HL
        • Trakman GL
        • Tatucu-Babet OA
        • Lassemillante A-CM
        • et al.
        The effect of high-polyphenol extra virgin olive oil on cardiovascular risk factors: a systematic review and meta-analysis.
        Crit Rev Food Sci Nutr. 2019; 59: 2772-2795
        • Ghaedi E
        • Mohammadi M
        • Mohammadi H
        • Ramezani-Jolfaie N
        • Malekzadeh J
        • Hosseinzadeh M
        • et al.
        Effects of a Paleolithic diet on cardiovascular disease risk factors: a systematic review and meta-analysis of randomized controlled trials.
        Adv Nutr. 2019; 10: 634-646
        • Haider LM
        • Schwingshackl L
        • Hoffmann G
        • Ekmekcioglu C.
        The effect of vegetarian diets on iron status in adults: a systematic review and meta-analysis.
        Crit Rev Food Sci Nutr. 2018; 58: 1359-1374
        • Hou R
        • Wei J
        • Hu Y
        • Zhang X
        • Sun X
        • Chandrasekar EK
        • et al.
        Healthy dietary patterns and risk and survival of breast cancer: a meta-analysis of cohort studies.
        Cancer Causes & Control. 2019; 30: 835-846
        • Huang Y
        • Zheng S
        • Wang T
        • Yang X
        • Luo Q
        • Li H
        Effect of oral nut supplementation on endothelium-dependent vasodilation–a meta-analysis.
        Vasa. 2018; 47: 203-207
        • Iguacel I
        • Miguel-Berges ML
        • Gómez-Bruton A
        • Moreno LA
        • Julián C.
        Veganism, vegetarianism, bone mineral density, and fracture risk: a systematic review and meta-analysis.
        Nutr Rev. 2019; 77: 1-18
        • Kang K
        • Sotunde OF
        • Weiler HA.
        Effects of milk and milk-product consumption on growth among children and adolescents aged 6–18 years: a meta-analysis of randomized controlled trials.
        Adv Nutr. 2019; 10: 250-261
        • Kibret KT
        • Chojenta C
        • Gresham E
        • Tegegne TK
        • Loxton D.
        Maternal dietary patterns and risk of adverse pregnancy (hypertensive disorders of pregnancy and gestational diabetes mellitus) and birth (preterm birth and low birth weight) outcomes: a systematic review and meta-analysis.
        Public Health Nutr. 2019; 22: 506-520
        • Kodama S
        • Horikawa C
        • Fujihara K
        • Ishii D
        • Hatta M
        • Takeda Y
        • et al.
        Relationship between intake of fruit separately from vegetables and triglycerides-A meta-analysis.
        Clin Nutr ESPEN. 2018; 27: 53-58
        • Kojima G
        • Avgerinou C
        • Iliffe S
        • Walters K.
        Adherence to Mediterranean diet reduces incident frailty risk: systematic review and meta-analysis.
        J Am Geriatr Soc. 2018; 66: 783-788
        • Larsson SC
        • Drca N
        • Jensen-Urstad M
        • Wolk A.
        Chocolate consumption and risk of atrial fibrillation: two cohort studies and a meta-analysis.
        Am Heart J. 2018; 195: 86-90
        • Li R
        • Yu K
        • Li C.
        Dietary factors and risk of gout and hyperuricemia: a meta-analysis and systematic review.
        Asia Pac J Clin Nutr. 2018; 27: 1344
        • Li W
        • Ruan W
        • Peng Y
        • Wang D.
        Soy and the risk of type 2 diabetes mellitus: a systematic review and meta-analysis of observational studies.
        Diabetes Res Clin Pract. 2018; 137: 190-199
        • Li L
        • Lietz G
        • Seal C.
        Buckwheat and CVD risk markers: a systematic review and meta-analysis.
        Nutrients. 2018; 10: 619
        • Lopez PD
        • Cativo EH
        • Atlas SA
        • Rosendorff C.
        The effect of vegan diets on blood pressure in adults: a meta-analysis of randomized controlled trials.
        Am J Med. 2019; 132: 875-83. e7
        • Maki KC
        • Palacios OM
        • Koecher K
        • Sawicki CM
        • Livingston KA
        • Bell M
        • et al.
        The relationship between whole grain intake and body weight: results of meta-analyses of observational studies and randomized controlled trials.
        Nutrients. 2019; 11: 1245
        • Malmir H
        • Saneei P
        • Larijani B
        • Esmaillzadeh A.
        Adherence to Mediterranean diet in relation to bone mineral density and risk of fracture: a systematic review and meta-analysis of observational studies.
        Eur J Nutr. 2018; 57: 2147-2160
        • Matía-Martín P
        • Torrego-Ellacuría M
        • Larrad-Sainz A
        • Fernández-Pérez C
        • Cuesta-Triana F
        • Rubio-Herrera MÁ.
        Effects of milk and dairy products on the prevention of osteoporosis and osteoporotic fractures in Europeans and non-Hispanic Whites from North America: a systematic review and updated meta-analysis.
        Adv Nutr. 2019; 10: S120-SS43
        • Mena-Sánchez G
        • Becerra-Tomás N
        • Babio N
        • Salas-Salvadó J.
        Dairy product consumption in the prevention of metabolic syndrome: a systematic review and meta-analysis of prospective cohort studies.
        Adv Nutr. 2019; 10: S144-SS53
        • Milajerdi A
        • Namazi N
        • Larijani B
        • Azadbakht L.
        The association of dietary quality indices and cancer mortality: a systematic review and meta-analysis of cohort studies.
        Nutr Cancer. 2018; 70: 1091-1105
        • Mishali M
        • Prizant-Passal S
        • Avrech T
        • Shoenfeld Y
        Association between dairy intake and the risk of contracting type 2 diabetes and cardiovascular diseases: a systematic review and meta-analysis with subgroup analysis of men versus women.
        Nutr Rev. 2019; 77: 417-429
        • Mohseni R
        • Abbasi S
        • Mohseni F
        • Rahimi F
        • Alizadeh S
        Association between dietary inflammatory index and the risk of prostate cancer: a meta-analysis.
        Nutr Cancer. 2019; 71: 359-366
        • Musa-Veloso K
        • Poon T
        • Harkness LS
        • O'Shea M
        • Chu Y
        The effects of whole-grain compared with refined wheat, rice, and rye on the postprandial blood glucose response: a systematic review and meta-analysis of randomized controlled trials.
        Am J Clin Nutr. 2018; 108: 759-774
        • Namazi N
        • Larijani B
        • Azadbakht L
        Dietary inflammatory index and its association with the risk of cardiovascular diseases, metabolic syndrome, and mortality: a systematic review and meta-analysis.
        Horm Metab Re. 2018; 50: 345-358
        • Picasso MC
        • Lo-Tayraco JA
        • Ramos-Villanueva JM
        • Pasupuleti V
        • Hernandez AV.
        Effect of vegetarian diets on the presentation of metabolic syndrome or its components: a systematic review and meta-analysis.
        Clin Nutr. 2019; 38: 1117-1132
        • Qin Z-Z
        • Xu J-Y
        • Chen G-C
        • Ma Y-X
        • Qin L-Q.
        Effects of fatty and lean fish intake on stroke risk: a meta-analysis of prospective cohort studies.
        Lipids Health Dis. 2018; 17: 1-7
        • Rees K
        • Takeda A
        • Martin N
        • Ellis L
        • Wijesekara D
        • Vepa A
        • et al.
        Mediterranean-style diet for the primary and secondary prevention of cardiovascular disease.
        Cochrane Database Syst Rev. 2019;
        • Ren Y
        • Liu Y
        • Sun X-Z
        • Wang B-Y
        • Zhao Y
        • Liu D-C
        • et al.
        Chocolate consumption and risk of cardiovascular diseases: a meta-analysis of prospective studies.
        Heart. 2019; 105: 49-55
        • Shab-Bidar S
        • Golzarand M
        • Hajimohammadi M
        • Mansouri S.
        A posteriori dietary patterns and metabolic syndrome in adults: a systematic review and meta-analysis of observational studies.
        Public Health Nutr. 2018; 21: 1681-1692
        • Shafiei F
        • Salari-Moghaddam A
        • Larijani B
        • Esmaillzadeh A.
        Adherence to the Mediterranean diet and risk of depression: a systematic review and updated meta-analysis of observational studies.
        Nutr Rev. 2019; 77: 230-239
        • Teoh SL
        • Lai NM
        • Vanichkulpitak P
        • Vuksan V
        • Ho H
        • Chaiyakunapruk N.
        Clinical evidence on dietary supplementation with chia seed (Salvia hispanica L.): a systematic review and meta-analysis.
        Nutr Rev. 2018; 76: 219-242
        • Voon PT
        • Lee ST
        • Ng TKW
        • Ng YT
        • Yong XS
        • Lee VKM
        • et al.
        Intake of palm olein and lipid status in healthy adults: a meta-analysis.
        Adv Nutr. 2019; 10: 647-659
        • Wang L
        • Liu C
        • Zhou C
        • Zhuang J
        • Tang S
        • Yu J
        • et al.
        Meta-analysis of the association between the dietary inflammatory index (DII) and breast cancer risk.
        Eur J Clin Nutr. 2019; 73: 509-517
        • Xiao Y
        • Ke Y
        • Wu S
        • Huang S
        • Li S
        • Lv Z
        • et al.
        Association between whole grain intake and breast cancer risk: a systematic review and meta-analysis of observational studies.
        Nutr j. 2018; 17: 1-12
        • Xu Y
        • Yang J
        • Du L
        • Li K
        • Zhou Y.
        Association of whole grain, refined grain, and cereal consumption with gastric cancer risk: a meta-analysis of observational studies.
        Food Sci Nutr. 2019; 7: 256-265
        • Zhang Z
        • Chen G-C
        • Qin Z-Z
        • Tong X
        • Li D-P
        • Qin L-Q.
        Poultry and fish consumption in relation to total cancer mortality: a meta-analysis of prospective studies.
        Nutr Cancer. 2018; 70: 204-212
        • Adjibade M
        • Assmann KE
        • Andreeva VA
        • Lemogne C
        • Hercberg S
        • Galan P
        • et al.
        Prospective association between adherence to the Mediterranean diet and risk of depressive symptoms in the French SU.VI.MAX cohort.
        Eur J Nutr. 2018; 57: 1225-1235
        • Page MJ
        • Shamseer L
        • Tricco AC.
        Registration of systematic reviews in PROSPERO: 30,000 records and counting.
        Systematic reviews. 2018; 7: 1-9
        • Liu Z
        • Saldanha IJ
        • Margolis D
        • Dumville JC
        • Cullum NA.
        Outcomes in Cochrane systematic reviews related to wound care: an investigation into prespecification.
        Wound Repair Regen. 2017; 25: 292-308
        • Saldanha IJ
        • Dickersin K
        • Wang X
        • Li T.
        Outcomes in Cochrane systematic reviews addressing four common eye conditions: an evaluation of completeness and comparability.
        PloS one. 2014; 9e109400
        • Zeraatkar D
        • Bhasin A
        • Morassut RE
        • Churchill I
        • Gupta A
        • Lawson DO
        • et al.
        Characteristics and quality of systematic reviews and meta-analyses of observational nutritional epidemiology: a cross-sectional study.
        Am J Clin Nutr. 2021; 113: 1578-1592
        • Trutschel D
        • Palm R
        • Holle B
        • Simon M.
        Methodological approaches in analysing observational data: a practical example on how to address clustering and selection bias.
        Int J Nurs Stud. 2017; 76: 36-44
      3. McKenzie JE, Brennan SE, Ryan RE, Thomson HJ, Johnston RV, Thomas J. Chapter 3: Defining the criteria for including studies and how they will be grouped for the synthesis. In: Higgins J.P.T., Thomas J., Chandler J., Cumpston M., Li T., Page M.J., Welch V.A. (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). Cochrane, 2021. Accessed on 1 December 2021 from: www.training.cochrane.org/handbook.

        • Page MJ
        • Moher D
        • Bossuyt P
        • Boutron I
        • Hoffmann T
        • Mulrow C
        • et al.
        PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews.
        BMJ. 2021; 372: n160