Attention should be given to multiplicity issues in systematic reviews

Abstract 

Objective

The objective of this paper is to describe the problem of multiple comparisons in systematic reviews and to provide some guidelines on how to deal with it in practice.

Study Design and Setting

We describe common reasons for multiplicity in systematic reviews, and present some examples. We provide guidance on how to deal with multiplicity when it is unavoidable.

Results

We identified six common reasons for multiplicity in systematic reviews: multiple outcomes, multiple groups, multiple time points, multiple effect measures, subgroup analyses, and multiple looks at accumulating data. The existing methods to deal with multiplicity in single trials can not always be applied in systematic reviews.

Conclusion

There is no simple and completely satisfactory solution to the problem of multiple comparisons in systematic reviews. More research is required to develop multiple comparison procedures for use in systematic reviews. Authors and consumers of systematic reviews should give serious attention to multiplicity in systematic reviews when presenting, interpreting and using the results of these reports.

Keywords: Cumulative meta-analysis, Meta-analysis, Multiple comparisons, Simultaneous statistical inference, Statistical data interpretation, Systematic reviews