Abstract
Multiplicity of data, hypotheses, and analyses is a common problem in biomedical and
epidemiological research. Multiple testing theory provides a framework for defining
and controlling appropriate error rates in order to protect against wrong conclusions.
However, the corresponding multiple test procedures are underutilized in biomedical
and epidemiological research. In this article, the existing multiple test procedures
are summarized for the most important multiplicity situations. It is emphasized that
adjustments for multiple testing are required in confirmatory studies whenever results
from multiple tests have to be combined in one final conclusion and decision. In case
of multiple significance tests a note on the error rate that will be controlled for
is desirable.
Keywords
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Journal of Clinical EpidemiologyAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- Statistical considerations for multiplicity in confirmatory protocols.Drug Inf J. 1996; 30: 523-534
- Some comments on frequently used multiple endpoint adjustment methods in clinical trials.Stat Med. 1997; 16: 2529-2542
- Biostatistical methodology in clinical trials in applications for marketing authorizations for medical products.Stat Med. 1995; 14: 1659-1682
- No adjustments are needed for multiple comparisons.Epidemiology. 1990; 1: 43-46
- Multiple comparisons and related issues in the interpretation of epidemiologic data.Am J Epidemiol. 1995; 142: 904-908
- Describing data requires no adjustment for multiple comparisons.Am J Epidemiol. 1998; 147: 813-814
- What's wrong with Bonferroni adjustments.BMJ. 1998; 316: 1236-1238
- Other method for adjustment of multiple testing exists.BMJ. 1999; 318 ([Letter]): 127
- Multiple test procedures other than Bonferroni's deserve wider use.BMJ. 1999; 318 ([Letter]): 600-601
- Multiple testing in clinical trials.Stat Med. 1991; 10: 871-890
- Invited commentary.Am J Epidemiol. 1990; 147: 801-806
- Multiple comparisons, explained.Am J Epidemiol. 1998; 147: 807-812
- The appropriateness of analysis of variance and multiple comparison procedures.Biometrics. 1983; 39: 787-794
- Simultaneous statistical inference. McGraw-Hill, New York1966
- Multiple comparison procedures. Wiley, New York1987
- Adjusted p-values for simultaneous inference.Biometrics. 1992; 48: 1005-1013
- Multiple significance tests.BMJ. 1995; 310: 170
- Annotation.Am J Public Health. 1996; 86: 628-629
- A simple sequentially rejective multiple test procedure.Scand J Stat. 1979; 6: 65-70
- Adjusting for multiple testing when reporting research results.Am J Public Health. 1996; 86: 726-728
- On closed testing procedures with special reference to ordered analysis of variance.Biometrika. 1976; 63: 655-660
- Resampling-based multiple testing. Wiley, New York1993
- Reader reaction.Biometrics. 1993; 49: 941-945
- Comparing several groups using analysis of variance.BMJ. 1996; 312: 1472-1473
- Comparing means of several groups.N Engl J Med. 1985; 313: 1450-1456
- Pairwise multiple comparison procedures.Psychol Bull. 1984; 96: 589-596
- New developments in pairwise multiple comparisons.Psychol Bull. 1991; 110: 577-586
- Practical nonparametric statistics. Wiley, New York1980
- Clinical trials with multiple outcomes.Contr Clin Trials. 1997; 18: 530-545
- Some statistical methods for multiple endpoints in clinical trials.Contr Clin Trials. 1997; 18: 204-221
- Procedures for comparing samples with multiple endpoints.Biometrics. 1984; 40: 1079-1087
- The analysis of multiple endpoints in clinical trials.Biometrics. 1987; 43: 487-498
- Exact t and F tests for analyzing studies with multiple endpoints.Biometrics. 1996; 52: 964-970
- Multivariate tests for multiple endpoints in clinical trials.Stat Med. 1995; 14: 1163-1175
- Procedures for two-sample comparisons with multiple endpoints controlling the experimentwise error rate.Biometrics. 1991; 47: 511-521
- P-value interpretation and alpha allocation in clinical trials.Ann Epidemiol. 1998; 8: 351-357
- Alpha calculus in clinical trials.Stat Med. 2000; 19: 767-779
- Tutorial in biostatistics.Stat Med. 1997; 16: 2349-2380
- Tutorial in biostatistics.Stat Med. 1998; 17: 1261-1291
- Analysis of serial measurements in medical research.BMJ. 1990; 300: 230-235
- Tutorial in biostatistics.Stat Med. 2000; 19: 861-877
- On multiple comparisons in the randomization analysis of growth and response curves.Biometrics. 1986; 42: 795-804
- A consumer's guide to subgroup analysis.Ann Intern Med. 1992; 116: 78-84
- The problem of cogent subgroups.J Clin Epidemiol. 1998; 51: 297-299
- Analysis of clinical trial outcomes.Contr Clin Trials. 1989; 10: 187S-194S
- Analysis and interpretation of treatment effects in subgroups of patients in randomised clinical trials.JAMA. 1991; 266: 93-98
- Clinical trials. Wiley, Chichester1983
- Design and analysis of randomised clinical trials requiring prolonged observation of each patient. I. Introduction and design.Br J Cancer. 1976; 34: 585-612
- A multiple testing procedure for clinical trials.Biometrics. 1979; 35: 549-556
- Overview of sequential methods and their applications in clinical trials.Commun Stat A. 1984; 13: 2315-2338
- Interim analyses in randomized clinical trials.Biometrics. 1987; 43: 213-223
- Statistical approaches to interim monitoring of medical trials.Stat Sci. 1990; 5: 299-317
- Statistical and ethical issues in monitoring clinical trials.Stat Med. 1993; 12: 1459-1469
- Group sequential testing in clinical trials with multivariate observations.Stat Med. 1994; 13: 101-111
- The management of interim analyses in drug development.Stat Med. 1998; 17: 1801-1809
- Repeated significance tests on accumulating survival data.J Clin Epidemiol. 1999; 52: 1083-1088
- U.K. Prospective Diabetes Study (UKPDS).Diabetologia. 1991; 34: 877-890
- Intensive blood-glucose control with sulphonylureas or insulin compared with conventional treatment and risk of complications in patients with type 2 diabetes (UKPDS 33).Lancet. 1998; 352: 837-853
- Effect of intensive blood-glucose control with metformin on complications in overweight patients with type 2 diabetes (UKPDS 34).Lancet. 1998; 352: 854-865
- Tight blood pressure control and risk of macrovascular and microvascular complications in type 2 diabetes.BMJ. 1998; 317: 703-713
- Efficacy of atenolol and captopril in reducing risk of macrovascular and microvascular complications in type 2 diabetes.BMJ. 1998; 317: 713-720
- The UK Prospective Diabetes Study.Lancet. 1998; 352 ([Letter]): 1933
- Tutorial in biostatistics.Stat Med. 1997; 16: 1413-1430
- Empirical Bayes estimates of age-standardized relative risks for use in disease mapping.Biometrics. 1987; 43: 671-681
- Empirical-Bayes adjustments for multiple comparisons are sometimes useful.Epidemiology. 1991; 2: 244-251
Article info
Publication history
Accepted:
August 2,
2000
Received in revised form:
July 31,
2000
Received:
September 9,
1999
Identification
Copyright
© 2001 Elsevier Science Inc. Published by Elsevier Inc. All rights reserved.
