If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Australian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales 2113, AustraliaCharles Perkins Centre, School of Pharmacy, Faculty of Medicine and Health, The University of Sydney, Camperdown, New South Wales, 2006, Australia
Charles Perkins Centre, Central Clinical School, Faculty of Medicine and Health, The University of Sydney, Camperdown, New South Wales, 2006, Australia
To investigate how often review authors encounter multiple results from included studies that are eligible for inclusion in a particular meta-analysis, and how often methods to select results are specified.
Methods
MEDLINE and Epistemonikos were searched (January 2018–June 2019) to identify systematic reviews with meta-analysis of the association between food/diet and health-related outcomes. A random sample of these reviews was selected, and for the first presented (index) meta-analysis, rules used to select effect estimates to include in this meta-analysis were extracted from the reviews and their protocols. All effect estimates from the primary studies that were eligible for inclusion in the index meta-analyses were extracted (e.g., when a study report presented effect estimates for blood pressure at 3 weeks and 6 weeks, both unadjusted and adjusted for covariates, and all were eligible for inclusion in a meta-analysis of the effect of red meat consumption on blood pressure, we extracted all estimates, and classified the study as having “multiplicity of results”).
Results
Forty-two systematic reviews with 325 studies (104 randomized, 221 non-randomized) were included; 14 reviews had a protocol. In 29% of review protocols and 69% of reviews, authors specified at least one decision rule to select effect estimates when multiple were available. In 68% of studies included in the index meta-analyses, there was at least one type of multiplicity of results.
Conclusions
Authors of systematic reviews of nutrition studies should anticipate encountering multiplicity of results in the included primary studies. Specification of methods to handle multiplicity when designing reviews is therefore recommended.
Authors of systematic reviews of nutrition research should anticipate encountering multiplicity of results in the included primary studies (i.e., multiple effect estimates being eligible for inclusion in a particular meta-analysis).
•
Decision rules to select results for inclusion in meta-analyses of nutrition research were infrequently pre-specified.
•
What this adds to what was known?
•
Previous studies have found that multiplicity of results of continuous outcomes in studies included in systematic reviews was common, and methods used to select results to include in meta-analyses were infrequently pre-specified in systematic review protocols. However, none of the previous studies examined meta-analyses in nutrition research, inclusion of randomized or non-randomized studies, or where the outcome was non-continuous (e.g., binary, count or time-to-event); circumstances for which different forms of multiplicity might arise. Our study addressed this gap.
•
What is the implication and what should change now?
•
Pre-specification of decision rules to handle multiplicity when designing reviews is recommended. In the systematic review, we recommend reporting any modifications to the specified rules, or any additions that were introduced to cover multiplicity scenarios that had not been anticipated when designing the review.
1. Background
The Global Burden of Disease study 2019 reported that diet has a significant impact on health outcomes. Diet quality was found to be the fifth leading risk factor for disability-adjusted life years [
GBD 2019 Risk Factors Collaborators Global burden of 87 risk factors in 204 countries and territories, 1990-2019: a systematic analysis for the global burden of disease study 2019.
]. Large and long-term prospective observational studies and short-term clinical trials have found associations between particular dietary factors and non-communicable diseases [
Association of dietary ω-3 and ω-6 fatty acids intake with cognitive performance in older adults: national health and nutrition examination survey (NHANES) 2011-2014.
]. For example, a study report may present multiple effect estimates for the association between red meat consumption and gout, where these estimates may arise from the fitting of multiple statistical models with different outcome definitions of gout (e.g., diagnosed according to different criteria), different exposure levels (e.g., <1, 1-2, 2-3 or >3 servings/week), or where adjustment is made for different sets of confounders. Although inclusion of multiple effect estimates from a particular study in a meta-analysis is possible (using methods that adjust for statistical dependency) [
], more commonly only one of the available effect estimates is selected for inclusion. There are various methods that can be used to select a single effect estimate [
]. However, when this selection is based on the statistical significance, magnitude or direction of effect, this may introduce bias into the meta-analysis effect estimate [
]. We refer to this selection process as “selective inclusion of results.”
To help mitigate the potential for selective inclusion of results, it has been recommended that methods for dealing with multiplicity should be pre-specified [
]. This includes pre-specification of “eligibility criteria” for each meta-analysis, indicating which results are eligible for inclusion in the meta-analysis (e.g., intervention groups, measurement instruments, time points), and “decision rules,” which specify the methods that will be used to select a single result from a study when multiple results are eligible for inclusion in the same meta-analysis (see Box 1 for examples of eligibility criteria and decision rules).
Box 1. Examples of eligibility criteria and decision rules to select results
Example of eligibility criteria to select results: Systematic reviewers state that only study effect estimates that were adjusted for sex and age would be included in a meta-analysis of the association between fruit consumption and coronary heart disease.
Example of a decision rule to select results when multiple are available: Systematic reviewers state that if multiple effect estimates quantifying the association between different levels of fruit consumption and coronary heart disease were available, as would arise if intake was categorised into quartiles in a study report, only the contrast between the highest (e.g., quartile 4), and lowest (e.g., quartile 1) intake would be included in the meta-analysis.
Previous research has examined the extent of multiplicity of results, and the methods used to select results for inclusion in several meta-analyses [
]. These studies have focused on a range of conditions and examined multiplicity in meta-analyses of randomized trials with continuous outcomes. All studies found that multiplicity of results was common, and Page et al. [
] found that specification of methods to select results for inclusion were rarely reported. However, none of the studies have examined meta-analyses including randomized or non-randomized studies, or non-continuous outcomes (e.g., binary, count or time-to-event). Understanding whether the extent of multiplicity or the methods used to select results for inclusion vary by these factors is important for developing tailored guidance.
We aimed to address the identified gaps by investigating the i) extent of multiplicity of results in study reports of nutrition research, and ii) the methods specified in systematic reviews to select results for inclusion in meta-analyses of all outcome types, including randomized or non-randomized study designs. We focus our investigation on reviews of nutrition because of their critical role in informing public health policy and because there has been no previous investigation of multiplicity in this area. Findings from this investigation may indicate the need for development of specific guidance for reviews of nutrition that address multiplicity issues unique to this field.
]. Here, we provide an overview of the methods, with modifications to the protocol reported in Supplementary Table 1. Our manuscript describes one component of the ROBUST study [
]. The ROBUST study aims to explore the extent of multiplicity in study reports, bias due to selective inclusion of results, and bias due to missing results, in systematic reviews of food/diet-outcome relationships. The results of the other components of the ROBUST study will be described in subsequent manuscripts.
2.1 Eligibility criteria, search, and selection of SRs
SRs that satisfied the definition of an SR, as outlined in the 2019 edition of the Cochrane Handbook for Systematic Reviews of Interventions [
], and that had explicitly stated methods of study identification (e.g., a search strategy) and of study selection (e.g., eligibility criteria and selection process), and included a meta-analysis of study results, were eligible for inclusion in this study. We included such SRs with meta-analysis that:
•
included studies that enrolled, regardless of their age and background, (a) people who were generally healthy, (b) a mixture of generally healthy people and people with diet-related risk factors (e.g., overweight, high blood pressure) or a particular health condition (e.g., type II diabetes or cardiovascular disease), or (c) people with non-specified health status;
•
included randomized trials or non-randomized studies that assessed the effects of at least one type of food (e.g., eggs, fish) or at least one dietary pattern (e.g., Mediterranean diet) on any continuous (e.g., systolic blood pressure) or non-continuous (e.g., gout incidence) health-related outcome;
•
were published between January 1, 2018 and June 30, 2019 (i.e., within 18 months before the drafting of our study protocol);
•
were written in English;
•
provided citations for all included studies in the SR, and;
•
presented the summary statistics or effect estimate and its precision (e.g., standard error or 95% confidence interval) for each included study, and the meta-analytic summary effect estimate and its precision in the text or forest plot, for at least one meta-analysis of a continuous or non-continuous outcome.
We excluded
•
SRs that did not include any meta-analysis of a non-continuous or continuous outcome;
•
meta-analyses or pooled analyses of studies conducted outside the context of a SR;
•
SRs that only focused on nutrient-specific associations with outcomes (e.g., examining the effects of single nutrients such as folic acid, salt), as the focus of this study is to assess evidence on the effects of consuming whole foods or dietary patterns on health outcomes, in line with the food-based rather than a nutrient-based approach adopted by the Food and Agriculture Organisation of the United Nations and many other countries developing dietary guidelines;
•
SRs that included studies enrolling only participants with a health condition, or who were obese, or who were frail or elderly people at risk of malnutrition, and;
•
SRs that were co-authored by any of our research team members.
We searched for eligible SRs indexed in the PubMed and Epistemonikos [
] databases from January 1, 2018 to June 30, 2019 (search strategies reported in supplement 2). The search results were exported into Microsoft Excel, all duplicate records were removed, and the remaining records were randomly sorted. In the piloting phase, four investigators (MJP, CMK, ZD, and SM) independently assessed 50 abstracts against the inclusion criteria (rating each as “Eligible,” “Ineligible”, or “Unsure”), discussed any discrepancies, and made any necessary changes to the screening form. Following piloting, two investigators (MJP and one of CMK, ZD or SM) independently screened titles and abstracts of 450 records. Two investigators (MJP and one of CMK, ZD or SM) then independently assessed the full text of records that were rated as “Eligible” or “Unsure” against the eligibility criteria. This screening process was repeated (in batches of 500 records) until we reached the target sample of 50 SRs, including 25 meta-analyses of continuous, and 25 meta-analyses of non-continuous outcomes. If the total number of eligible SRs exceeded this target at the end of a batch, we planned to randomly sample 25 SRs of each type. Our target of 50 SRs was primarily selected for feasibility reasons given our available resources to conduct all components of the ROBUST study [
], which was informed by the time taken to conduct a previous similar study. Any discrepancies in screening decisions at each stage were resolved via discussion between investigators, or by consultation with another investigator (JM) where necessary. For each included SR, we retrieved the published protocol for the SR or registration record (e.g., PROSPERO record), if available, as cited or reported in the SR.
From each SR meeting the inclusion criteria, one investigator (MJP) selected one pairwise meta-analysis of aggregate data for inclusion. The selected meta-analysis was the first meta-analytic result mentioned in the review (with no restrictions on its placement in the manuscipt); we refer to the selected meta-analysis as the “index meta-analysis.” We initially selected an index meta-analysis regardless of the outcome domain (e.g., weight, bladder cancer), effect measure (e.g., odds ratio, standardised mean difference), meta-analytic model and number and type of included studies (i.e., randomized or non-randomized study). However, following selection of 50 index meta-analyses and recording the number of studies included in each, we decided to restrict inclusion to only those systematic reviews with an index meta-analysis including fewer than 20 studies, again for reasons of feasibility. The 50 systematic reviews originally sampled included 553 studies, which is more than twice what we had anticipated based on our previous study (which included 44 systematic reviews with 210 studies) [
]. We did not replace the systematic reviews that included meta-analyses with more than 20 studies. For each included index meta-analysis with fewer than 20 studies, we retrieved the reports of all included studies (see the protocol for further details [
]. Seven investigators (RK, ZD, SM, CMK, EK, LB, and MJP) piloted the form on two randomly selected meta-analyses and their included studies. Discrepancies were discussed, and we modified the form accordingly. Following piloting, two investigators (RK and one of ZD, SM, CMK, EK, LB and MJP) independently collected data from a random sample of half of the index meta-analyses and their included studies, although one investigator (RK) collected data on the remaining index meta-analyses, and their included studies. Any discrepancies were resolved through discussions between two investigators or adjudication by a third investigator (JM) if necessary.
An overview of the data items, and the sources these were obtained from (i.e., systematic review protocol, systematic review or study report), is presented in Table 1; further details are available in the protocol [
]. In the case of data extracted from the reports of studies included in the index meta-analysis, we extracted all outcome data that were eligible for inclusion in the index meta-analysis. This was determined by the eligibility criteria and decision rules stated in the SR protocol if available, and if not available, how the comparison and outcome of the meta-analysis were specified in the SR. For example, if the systematic reviewers pre-defined in the SR protocol that the eligible intervention and comparator for the meta-analysis of weight gain was “highest vs. lowest intake of dairy products,” and pre-defined a decision rule stipulating that they would consider only data at 12 weeks follow-up when data were available at multiple time points, we only extracted data for that comparison and time point, regardless of whether study reports had data for other time points and other comparisons for the same outcome. In the absence of an SR protocol, we assumed that no eligibility criteria and decision rules were pre-specified (“worst-case scenario” assumption) and extracted all study outcome data based on how the outcome was specified in the SR. For example, if the systematic reviewers did not state in a protocol which results should be selected when multiple were available, yet defined the meta-analysis as “effect of dairy intake on weight at 6 months,” we extracted all data on weight at 6 months, regardless of the level of intake of dairy, whether results were unadjusted or adjusted, or what analysis sample was used.
Table 1Data sources and data items (see protocol for further details)
Year of publication/registration; eligibility criteria and decision rules to select results to include in the index meta-analysis
Systematic review
General characteristics of the systematic review
Journal name; year of publication; corresponding author's country and affiliation; conflicts of interest of review authors; source of funding for the review;
General characteristics of the index meta-analysis
Number of studies and participants; type of population investigated; type of interventions/exposures investigated; type of studies included in the meta-analysis; outcome domain (such as weight, cardiovascular function); outcome primacy label (primary or secondary or unlabelled); meta-analysis effect measure; meta-analysis model; eligibility criteria and decision rules to select results to include in the index meta-analysis; summary statistics, effect estimates and measures of precision (e.g confidence interval) for each included study; and the meta-analytic effect estimate and measure of precision.
Study reports
Outcome data that could potentially be included in the index meta-analysis
Outcome definition and measurement instrument; intervention/exposure desciption; comparator description; time point; analysis sample (e.g., intention-to-treat, per-protocol); summary statistics (e.g., number of events and sample sizes of both intervention/exposure and comparator); effect measure (e.g., risk ratio, mean difference); effect estimates and measures of precision (e.g., 95% confidence interval) and location of data in the report; whether results were unadjusted or adjusted for covariates; covariates that were adjusted for (if applicable).
We used descriptive statistics to summarise the characteristics of SR protocols, SRs, index meta-analyses, and included studies. For categorical variables, we present frequencies, and percentages. For continuous variables, we report medians (with interquartile ranges [IQR]). We computed the frequencies and percentages for different types of eligibility criteria and decision rules used to select results, differences in eligibility criteria or decision rules between the SR protocol and the SR, and studies with different types of multiplicity of results. We also calculated risk differences (with 95% confidence intervals [CIs]) to examine whether the percentages of different types of eligibility criteria and decision rules and studies with different types of multiplicity of results, differed between meta-analyses of continuous outcomes, and of non-continuous outcomes. Risk differences were calculated using the method of Mee with the Miettinen and Nurminen modification. Analyses were undertaken using the statistical packages Stata (College Station, Tx), except for the calculaton of risk differences, which was undertaken using the library PropCIs [
For the analysis of the frequency of different types of multiplicity of results, we restricted the sample to systematic reviews without any pre-specified decision rules to select results to include in meta-analyses. These reviews, and the studies within, were used to estimate the extent of multiplicity of results that can be expected when no decision rules to select results have been implemented.
3. Results
3.1 Results of search and screening
Our search yielded a total of 7,167 references from the PubMed and Epistemonikos databases (Fig. 1). After removing duplicates (n = 908), we screened the titles and abstracts of the first 3,013 randomly sorted references, of which 2,777 were excluded, leaving 236 for full-text screening. Of these, 99 SRs met the inclusion criteria, including 25 SRs with a meta-analysis of a continuous outcome and 74 with meta-analysis of a non-continuous outcome. Initially, all SRs with a continuous outcome were included, and 25 of the SRs with a non-continuous outcome were randomly selected. From these 50 SRs, eight were excluded (6 continuous, 2 non-continuous), because the index meta-analysis had 20 or more studies, leaving 42 included SRs [
Effects of milk and milk-product consumption on growth among children and adolescents aged 6–18 years: a meta-analysis of randomized controlled trials.
Maternal dietary patterns and risk of adverse pregnancy (hypertensive disorders of pregnancy and gestational diabetes mellitus) and birth (preterm birth and low birth weight) outcomes: a systematic review and meta-analysis.
Adherence to Mediterranean diet in relation to bone mineral density and risk of fracture: a systematic review and meta-analysis of observational studies.
Effects of milk and dairy products on the prevention of osteoporosis and osteoporotic fractures in Europeans and non-Hispanic Whites from North America: a systematic review and updated meta-analysis.
Association between dairy intake and the risk of contracting type 2 diabetes and cardiovascular diseases: a systematic review and meta-analysis with subgroup analysis of men versus women.
The effects of whole-grain compared with refined wheat, rice, and rye on the postprandial blood glucose response: a systematic review and meta-analysis of randomized controlled trials.
Dietary inflammatory index and its association with the risk of cardiovascular diseases, metabolic syndrome, and mortality: a systematic review and meta-analysis.
*Of the 6,259 unique titles and abstracts, we only needed to screen 3,013 randomly sorted titles and abstracts to reach our target sample size. **We initially drew a random sample of 50 systematic reviews, but post-hoc excluded eight systematic reviews with 20 or more included studies in the index meta-analysis to reduce workload.
3.2 Characteristics of the included systematic reviews and their index meta-analyses
Of the 42 SRs, 14 had accessible protocols (2 were published protocols and 12 were PROSPERO records) (Table 2). In most index meta-analyses the population was unclear (45%, 19 of 42), 33% (14 of 42) included only healthy participants and the remainder (22%, 9 of 42) included a mix of healthy people and people with a health condition. Most index meta-analyses included only non-randomized studies (62%, 26 of 42), 33% (14 of 42) included only randomized trials and the remaining two (5%) included studies of both designs (Table 3). Of the 19 meta-analyses of a continuous outcome, 14 included randomized trials only, three included non-randomized studies only, and two included both designs. Of the 23 meta-analyses of a non-continuous outcome, all included non-randomized studies only. The primacy of the outcome was not identified in most reviews (81%, 34 of 42). In nearly all meta-analyses, a random-effects model was fitted (90%, 38 of 42). The 42 index meta-analyses included a total of 325 studies (104 randomized, 221 non-randomized), with a median of seven studies (IQR 5-11; range 2-17) per meta-analysis.
Table 2Characteristics of the systematic reviews (N = 42)
Characteristic
n (%)
Focus of journal
Restricted to nutrition research
28 (67)
Not restricted to nutrition research
14 (33)
Country of the corresponding author(s)
China
9 (21)
Iran
7 (17)
United States of America
6 (14)
Others (Australia, Austria, Brazil, Canada, Israel, Japan, Malaysia, Spain, Sweden, Thailand, United Kingdon)
20 (48)
Affiliation of the corresponding author(s)
Food industry
2 (5)
Non-industry
37 (88)
Mixed
2 (5)
Unclear
1 (2)
Source of funding
Non-profit
23 (55)
For-profit
3 (7)
Mixed
0
No funding
8 (19)
Not reported
8 (19)
Conflict of interest
Conflict of interest reported by at least one review author
7 (17)
All review authors stated they had no conflicts of interest
29 (69)
Missing/not reported
6 (14)
Protocol availability
Both a protocol and registration record are available
3.3 Eligibility criteria and decision rules reported in SR protocols
Of the SR protocols (n = 14), all included at least one eligibility criterion, and four (29%) reported at least one decision rule to select results (Table 4). Almost all protocols specified eligibility criteria based on interventions/exposures (93%, 13 of 14) (e.g., specifying which foods or dietary patterns were eligible), but few other types of eligibility criteria were specified (e.g., time points [29%, 4 of 14], information sources [13%, 3 of 14]). The most commonly pre-specified decision rule was based on interventions/exposures (reported in 3 of the 4 SR protocols with at least one decision rule to select results). See Supplementary Table S3 for the content of the decision rules.
Table 4Number of systematic review protocols or registration entries reporting eligibility criteria and decision rules to select results (N = 14)
Risk difference calculated as the difference in percentage of SRs reporting the specified eligibility criteria/decision rule between SRs of continuous outcomes minus SRs of non-continuous outcomes. ITT, Intention to treat; NA, Not applicable;
Confidence limits for the difference in percentages calculated using the method of Mee with the Miettinen and Nurminen modification
Total
At least one eligibility criterion
14 (100)
9 (100)
5 (100)
0 (-31, 45)
At least one decision rule
4 (29)
2 (22)
2 (40)
-18 (-62, 30)
Measurement instruments
Eligibility criteria
1 (7)
1 (11)
0
11 (-36, 45)
Decision rule
0
0
0
0 (-45, 31)
Definitions/diagnostic criteria
Eligibility criteria
0
0
0
0 (-45, 31)
Decision rule
0
0
0
0 (-45, 31)
Cut-points on a measurement instrument
Eligibility criteria
0
0
0
0 (-45, 31)
Decision rule
0
0
0
0 (-45, 31)
Time points
Eligibility criteria
4 (29)
2 (22)
2 (40)
-18 (-63, 30)
Decision rule
0
0
0
0 (-45, 31)
Interventions/exposures
Eligibility criteria
13 (93)
9 (100)
4 (80)
20 (-16, 63)
Decision rule
3 (21)
2 (22)
1 (20)
2 (-47, 43)
Information sources
Eligibility criteria
3 (21)
2 (22)
1 (20)
2 (-47, 43)
Decision rule
1 (7)
0
1 (20)
-20 (-63, 16)
Analyses
Eligibility criteria for any type of analysis
2 (14)
2 (22)
0
22 (-27, 56)
Decision rule for any type of analysis
2 (14)
0
2 (40)
-40 (-78, 0)
Rule for final vs. change from baseline values
0
0
NA
NA
Rule for analyses undertaken on multiple samples (e.g., ITT vs. per-protocol)
0
0
0
0 (-45, 31)
Rule for unadjusted vs. covariate-adjusted analyses
1 (7)
0
1 (20)
-20 (-63, 16)
Rule for period vs. paired analyses in crossover randomized trials
0
0
0
0 (-45, 31)
Rule to handle results arising from overlapping samples of participants
1 (7)
0
1 (20)
-20 (-63, 16)
Other decision rule
0
0
0
0 (-45, 31)
a Risk difference calculated as the difference in percentage of SRs reporting the specified eligibility criteria/decision rule between SRs of continuous outcomes minus SRs of non-continuous outcomes. ITT, Intention to treat; NA, Not applicable;
b Confidence limits for the difference in percentages calculated using the method of Mee with the Miettinen and Nurminen modification
3.4 Eligibility criteria and decision rules reported in SRs
Of the SRs (n = 42), all included at least one eligibility criterion, and 69% reported at least one decision rule to select results (Table 5). Similar to the SR protocols, the most commonly reported eligibility criteria (95%, 40 of 42), and decision rule (40%, 17 of 42) in the SRs was based on interventions/exposures. Eligibility criteria and decision rules for the type of analysis were more freqently specified in SRs as compared with their protocols. The most commonly reported decision rules for analyses were rules to select from multiple unadjusted and covariate-adjusted analyses (24%) (Table 5). There were some discrepancies observed between SR protocols and their published SRs, with the most common type being the addition of a new decision rule to deal with multiple unadjusted and covariate-adjusted analyses in the included studies (Supplementary Table S4).
Table 5Number of systematic reviews reporting eligibility criteria and decision rules to select results
Risk difference calculated as the difference in percentage of SRs reporting the specified eligibility criteria/decision rule between SRs of continuous outcomes minus SRs of non-continuous outcomes. ITT, Intention to treat; NA, Not applicable;
Confidence limits for the difference in percentages calculated using the method of Mee with the Miettinen and Nurminen modification
Total
At least one eligibility criterion
42 (100)
19 (100)
23 (100)
0 (-17, 14)
At least one decision rule
29 (69)
14 (74)
15 (65)
9 (-20, 35)
Measurement instruments
Eligibility criteria
1 (2)
1 (5)
0
5 (-9, 24)
Decision rule
2 (5)
1 (5)
1 (4)
1 (-17, 21)
Definitions/diagnostic criteria
Eligibility criteria
2 (5)
0
2 (9)
-9 (-27, 9)
Decision rule
1 (2)
0
1 (4)
-4 (-21, 13)
Cut-points on a measurement instrument
Eligibility criteria
0
0
0
0 (-15, 17)
Decision rule
0
0
0
0 (-15, 17)
Time points
Eligibility criteria
4 (10)
2 (11)
2 (9)
2 (-18, 24)
Decision rule
4(10)
3 (16)
1 (4)
12 (-8, 34)
Interventions/exposures
Eligibility criteria
40 (95)
19 (100)
21 (91)
9 (-9, 27)
Decision rule
17 (40)
6 (32)
11 (48)
-16 (-43, 14)
Information sources
Eligibility criteria
3 (7)
1 (5)
2 (9)
-4 (-23, 17)
Decision rule
4 (10)
3 (16)
1 (4)
12 (-8, 34)
Analyses
Eligibility criteria for any type of analysis
11 (26)
7 (37)
4 (17)
20 (-7, 45)
Decision rule for any type of analysis
16 (38)
8 (42)
8 (35)
7 (-21, 35)
Rule for final vs. change from baseline values
3 (7)
3 (16)
0
NA
Rule for analyses undertaken on multiple samples (e.g., ITT vs. per-protocol)
0
0
0
0 (-15, 17)
Rule for unadjusted vs. covariate-adjusted analyses
10 (24)
2 (11)
8 (35)
-24 (-47, 2)
Rule for period vs. paired analyses in crossover randomized trials
2 (5)
2 (11)
0
11 (-5, 31)
Rule to handle results arising from overlapping samples of participants
1 (2)
0
1 (4)
-4 (-21, 13)
Other decision rule
5 (12)
1 (5)
4 (17)
-12 (-33, 10)
a Risk difference calculated as the difference in percentage of SRs reporting the specified eligibility criteria/decision rule between SRs of continuous outcomes minus SRs of non-continuous outcomes. ITT, Intention to treat; NA, Not applicable;
b Confidence limits for the difference in percentages calculated using the method of Mee with the Miettinen and Nurminen modification
The percentage of reviews specifying different types of eligibility criteria and decision rules generally did not differ by outcome type. However, a larger percentage of SRs with an index meta-analysis of a continuous outcome specified eligibility criteria for any type of analysis compared with SRs with a non-continuous outcome (37% vs. 17%; risk difference [RD] 20%, 95% CI -7% to 45%). Conversely, a smaller percentage of SRs with an index meta-analysis of a continuous outcome specified a rule for selecting an adjusted/unadjusted result compared with SRs of a non-continuous outcome (11% vs. 35%; RD -24%, 95% CI -47% to 2%).
3.5 Multiplicity of results in included studies
Of the 325 studies, 296 studies were included in reviews (n = 38) without any pre-specified decision rules to select results to include in meta-analyses (Table 6). These reviews, and the studies within, are therefore used to estimate the extent of multiplicity of results that can be expected when no decision rules to select results have been implemented. The median (IQR) number of available effect estimates per study was 2 (1 to 4), and the largest number of effect estimates in a study was 41 [
]. The most common types of multiplicity arose from multiple unadjusted and one or more covariate-adjusted analyses (which occurred in 39% of the included studies), followed by multiple intervention/control groups (24%). The least common types of multiplicity arose from multiple instruments (0%). The studies with continuous outcomes had less multiplicity than the studies with non-continuous outcomes (53% vs. 80%; RD -27%, 95% CI -37%, -16%).
Table 6Number of studies with different types of multiplicity of results in systematic reviews without pre-specified decision rules to select results (N = 38)
Confidence limits for the difference in percentages calculated using the method of Mee with the Miettinen and Nurminen modification; ITT, Intention to treat; NA, Not applicable;
Any
202 (68)
66 (53)
136 (80)
-27 (-37, -16)
Instruments
0
0
0
0 (-2, 3)
Intervention/control groups
70 (24)
23 (18)
47 (27)
-9 (-18, 7)
Time points
14 (5)
13 (10)
1 (1)
9 (5, 16)
Final and change from baseline values
18 (6)
18 (14)
NA
NA
Analyses undertaken on multiple samples (e.g., ITT and per-protocol)
2 (1)
2 (2)
0
2 (-1, 6)
Unadjusted and one or more covariate-adjusted analyses
115 (39)
4 (3)
111 (65)
-62 (-69, -53)
Period and paired analyses in crossover randomized trials
a Confidence limits for the difference in percentages calculated using the method of Mee with the Miettinen and Nurminen modification; ITT, Intention to treat; NA, Not applicable;
Our findings show that decision rules to select results were less frequently pre-specified in the protocols of a randomly selected sample of SRs in nutrition research. Multiplicity of results in the primary studies included in the index meta-analyses was common, with 68% having at least one type of multiplicity. The frequency and types of multiplicity in the included studies varied, arising from multiple intervention groups, time points, analyses, and subgroups.
Comparison with previous research
The findings of our study are in line with previously published studies, which have observed incomplete pre-specification of SR methods such as eligibility criteria, methods for collecting, handling and analysing data, and pre-specification of eligibility criteria and decision rules for selecting results in PROSPERO records [
] examined the conduct and reporting of 150 systematic reviews of observational nutrition studies published from 2018-2019, identifying several flaws in the conduct and reporting, and also recommended that pre-defined rules should be specified for selecting one estimate from each study for inclusion in a particular meta-analysis.
Two previous studies, on which the methodology of the present study is based, examined methods used to select results for inclusion in meta-analyses [
] and our study, all protocols reported at least one eligibility criteria, but at least one decision rule was pre-specified more in the SR protocols included in Page et al. [
] than those in our study (81% vs. 29%). The frequency of discrepancies between the SR protocol and SR in the eligibility criteria or decision rules to select results was higher in our study. Tendal et al. [
] examined eighteen Cochrane reviews, all of which had protocols. Eight of the protocols mentioned eligible time points or periods, but only one provided decision rules to handle multiplicity of time points. Interestingly, all of the 18 protocols reported eligibility criteria for the control group, but none reported decision rules to handle multiple control groups in included studies.
] had fewer studies with multiple estimates that were available for inclusion in a particular meta-analysis. Multiplicity arising from multiple intervention/control groups was slightly less frequent in our study compared to Tendal et al. [
] (24% vs. 17%). Our study findings showed less multiplicity in terms of time points (5%) and measurement instruments for outcomes (0%) compared to previous studies [
]. However, we observed greater multiplicity (39%) due to unadjusted, and one or more covariate-adjusted analyses. This difference was likely driven by inclusion of non-randomized studies in the present study, which often require adjustment for covariates to reduce risk of bias due to confounding [
] assessed multiplicity in the clinical trials of publicly accessible reports and non-public reports (e.g., unpublished Clinical Study Reports) related to gabapentin for neuropathic pain (n = 21) and quetiapine for bipolar depression (n = 7). In 15 of 21 (71%) gabapentin trials and 7 of 7 (100%) quetiapine trials, there was multiplicity of results.
Strengths and limitations
The major strength of our study is that we have pre-specified methods to identify, select and collect data from eligible SRs and studies, and provided the modifications/deviations to our study protocol (supplementary Table S1). We also used extensive search strategies to identify SRs in nutrition research. In addition, all the study authors, who had different levels of expertise, undertook training and pilot-testing of data collection forms. Moreover, given we randomly selected the SRs, our findings are generalisable to SRs meeting this study's eligibility criteria.
A limitation of our study is that we only retrieved reports of studies included in the index meta-analyses that the systematic reviewers cited. It is possible that other papers relating to the studies exist, which contain additional results that are compatible with the index meta-analysis (e.g., results of cohort studies or randomized trials at later time points may have been presented in other papers which were not cited by the systematic reviewers). For this reason, we may have underestimated the true extent of multiplicity of results within studies. Furthermore, our focus on SRs of food/diet-outcome associations means our results are not generalisable to other types of SRs of nutrition research. For example, it is possible that evidence of multiplicity is more apparent in studies included in SRs of nutrient-specific associations with outcomes, because there are more methods to measure nutrient-specific exposures (e.g., supplement intake, biomarkers) than foods and dietary patterns, which are typically measured via food frequency questionnaires. We made a post-hoc decision to restrict our sample to systematic reviews with fewer than 20 studies included in the index meta-analyses, for reasons of feasibility. However, we have no reason to believe that the extent, and types of multiplicity would differ by the number of included studies in a review. We were unable to translate and interpret the data from four included non-English language studies however, their absence is unlikely to have modified the observed extent of multiplicity. Finally, we only searched PubMed and Epistemonikos to find all SRs and included SRs written in English, so generalisation of our study findings to non-indexed SRs and non-English language SRs is potentially limited.
], demonstrates that multiplicity of effect estimates in primary studies is very common in nutrition research, and is therefore an issue that systematic reviewers should prepare for when designing their review. Doing so will have multiple benefits. It will reduce post-hoc decision making, and in doing-so, provide greater assurance to a reader that the results have not been “cherry-picked” for inclusion in the meta-analyses. Furthermore, specification of eligibility criteria, and decision rules is likely to lessen the data extraction effort, requiring less information to be extracted from each primary study.
Our results suggest that in nutrition research, specification of eligibility criteria, and decision rules to select results from among multiple unadjusted or covariate-adjusted analyses are most important to pre-specify in SRs that include non-randomized studies with non-continuous outcomes. On the other hand, methods used to select results arising from both final values and change from baseline values, and multiple time points, are most important to pre-specify in SRs that include randomized trials with continuous outcomes. Furthermore, in the SR, we recommend reporting any modifications to the specified rules, or any additions that were introduced to cover multiplicity scenarios that had not been anticipated when designing the review.
The Cochrane Handbook for Systematic Reviews of Interventions version 6.2 [
McKenzie JE, Brennan SE, Ryan RE, Thomson HJ, Johnston RV, Thomas J. Chapter 3: Defining the criteria for including studies and how they will be grouped for the synthesis. In: Higgins J.P.T., Thomas J., Chandler J., Cumpston M., Li T., Page M.J., Welch V.A. (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). Cochrane, 2021. Accessed on 1 December 2021 from: www.training.cochrane.org/handbook.
] provides updated guidance to systematic reviewers about how to group interventions with multiple components or co-interventions or how to select from multiple comparisons and handle multiplicity of outcomes when conducting meta-analyses. In addition, the recently updated Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement [
] includes a new item (10a), which recommends authors “List and define all outcomes for which data were sought. Specify whether all results that were compatible with each outcome domain in each study were sought (e.g., for all measures, time points, analyses), and if not, the methods used to decide which results to collect.” Implementation of recommendations from these sources will allow readers to understand the result selection process.
5. Conclusion
Our study found that in systematic reviews examining the effects of foods and diet, that multiplicity of results in the included primary studies was common. Yet, pre-specification of decision rules to select from multiple results was lacking. Systematic reviewers are encouraged to consider methods for dealing with multiplicity when designing their reviews. Doing so will limit the potential for selective inclusion of results, thus providing greater assurance to readers as to the trustworthiness of the review.
6. Contributors
All authors declare to meet the ICMJE conditions for authorship. MJP and JEM conceived the project. MJP, JEM, LB, ZD, SM, and CMK contributed to the design of the project. MJP, ZD, SM, and CMK screened articles for inclusion. RK, MJP, LB, ZD, SM, CMK, and EK collected data. RK analysed the data. RK wrote the first draft of the manuscript, which was revised in conjunction with MJP and JEM. JEM drafted sections of the manuscript. All authors were involved in revising the article critically for important intellectual content. All authors approved the final version of the article. MJP is the guarantor of this work.
Funding
This project was funded by an Australian National Health and Medical Research Council (NHMRC) project grant (APP1139997). RK is supported by a Monash Graduate Scholarship and a Monash International Tuition Scholarship. MJP is supported by an Australian Research Council Discovery Early Career Researcher Award (DE200101618). JEM is supported by an Australian NHMRC Career Development Fellowship (1143429). SM is supported by the Country Women's Association (NSW) and Edna Winifred Blackman Postgraduate Research Scholarship. The funders had no role in the study design, data collection and analysis, or preparation of the manuscript.
Association of dietary ω-3 and ω-6 fatty acids intake with cognitive performance in older adults: national health and nutrition examination survey (NHANES) 2011-2014.
Effects of milk and milk-product consumption on growth among children and adolescents aged 6–18 years: a meta-analysis of randomized controlled trials.
Maternal dietary patterns and risk of adverse pregnancy (hypertensive disorders of pregnancy and gestational diabetes mellitus) and birth (preterm birth and low birth weight) outcomes: a systematic review and meta-analysis.
Adherence to Mediterranean diet in relation to bone mineral density and risk of fracture: a systematic review and meta-analysis of observational studies.
Effects of milk and dairy products on the prevention of osteoporosis and osteoporotic fractures in Europeans and non-Hispanic Whites from North America: a systematic review and updated meta-analysis.
Association between dairy intake and the risk of contracting type 2 diabetes and cardiovascular diseases: a systematic review and meta-analysis with subgroup analysis of men versus women.
The effects of whole-grain compared with refined wheat, rice, and rye on the postprandial blood glucose response: a systematic review and meta-analysis of randomized controlled trials.
Dietary inflammatory index and its association with the risk of cardiovascular diseases, metabolic syndrome, and mortality: a systematic review and meta-analysis.
McKenzie JE, Brennan SE, Ryan RE, Thomson HJ, Johnston RV, Thomas J. Chapter 3: Defining the criteria for including studies and how they will be grouped for the synthesis. In: Higgins J.P.T., Thomas J., Chandler J., Cumpston M., Li T., Page M.J., Welch V.A. (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). Cochrane, 2021. Accessed on 1 December 2021 from: www.training.cochrane.org/handbook.
Conflict of interest statement: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: MJP and LB are editorial board members for the Journal of Clinical Epidemiology.
00368-1&id=gr1.jpg){kind=link}