Credibility and methodology of effectiveness research keep us busy
Article Outline
Although the randomized controlled trial (RCT) is considered as the paradigm for all study design and its methodology is highly crystallized and established, it is and will stay a vital method that is continuously being developed, extended, and improved [1]. At the same time, when a good trial is difficult or not feasible, observational alternatives ask our attention [2], [3]. In this issue of the Journal a number of articles address topics related to credibility and methodology of effectiveness research.
Regarding credibility, “industry bias,” described as association between funding of a trial by industry and a favorable outcome for that industry's product, is a well-described and quite extensively documented phenomenon [4], and more research on how to safeguard academic freedom has been recommended [5]. As the starting point of a Variance and Dissent section, in a systematic review Gartlehner et al. report that industry bias may play a lesser role in comparative effectiveness reviews than in reviews of placebo-controlled trials. Nevertheless, bias may still be introduced by industry funding [6], and in a subsequent discussion between these authors and Paul Shekelle important points are made to enhance the credibility of reported trial results, such as registration and publication of trial protocols in advance and making trial data openly available. Moreover, the more general subject of how to overcome a “broken system” is elaborated, to optimally safeguard the independence of the research; for example, by disentanglement of sponsorship and research as has also been proposed by Garattini and Chalmers [7].
In observational research on treatment effects, survivor bias is often overlooked. In a cohort study on the impact of valve surgery on mortality in infective endocarditis, Tleyeh et al. address this problem by applying propensity score analysis [8] while comparing a time-fixed and a time-dependent approach aiming to adjusted for survivor bias. This is the basis for a second series of debate in this JCE issue, focused on whether propensity score analysis adjusted for survivor bias is a good method to evaluate treatment effectiveness. Austin and Platt comment on this approach.
Another vital issue in treatment evaluation is how to handle baseline covariates. In a systematic review, Austin et al. show that there is substantial variation between published studies in how this is achieved, and they make an important plea for greater editorial consistency across journals in reporting RCTs in this respect.
In a more general sense, Singh et al., also in a systematic review, studied whether reported trial quality is associated with trial outcomes. An interesting finding is that a larger sample size is an independent predictor of trial outcome and implications are discussed.
Also in this issue, the methodology of outcome assessment is represented in various ways. Based on two RCTs, Staples et al. report on the validity and responsiveness of shoulder-specific outcome measures. And from their longitudinal research on chemotherapy in cancer patients, Kemmler et al. suggest to incorporate the type one and possibly the type two error when defining relevant change in quality of life for individual subjects.
Additional important methodological contributions are made by Smith et al. on supplementing Bland–Altman plots with bar charts, especially when the number of unique values is limited, and by Vergouwe et al. on how to deal with missing predictor values in the development and validation of a clinical prediction model.
Equally important is the work by Jaspers et al. showing the limited value of retrospectively collected information on lifestyle during pregnancy and early childhood, and by Myint et al. and Adair et al. highlighting, respectively, prediction of mortality based on subjective functional health, and the influence of certification practices on reported diabetes mortality. Finally the groups of Vega and Severinsen nicely demonstrate both the usefulness of evaluation and the need for appropriate interpretation of data from population-based cohorts and hospital discharge diagnosis registries.
References
- . A systematic review of trends in the methodological quality of randomized controlled trials in various research fields. J Clin Epidemiol. 2009;62:227–231
- . Evaluation of interventions. J Clin Epidemiol. 2002;55:1183–1190
- . Focus on nonexperimental approaches. J Clin Epidemiol. 2009;62:1225
- . Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ. 2003;326:1167–1170
- . “Experimental” institutional models for corporate funding of academic research: unknown effects on the research enterprise. J Clin Epidemiol. 2008;61:629–633
- . Factors associated with findings of published trials of drug-drug comparisons: why some statins appear more efficacious than others. PLoS Med. 2007;4(6):e184
- . Patients and the public deserve big changes in evaluation of drugs. BMJ. 2009;338:b1025
- . The performance of different propensity-score methods for estimating relative risks. J Clin Epidemiol. 2008;61:537–545
PII: S0895-4356(09)00355-2
doi:10.1016/j.jclinepi.2009.11.003
© 2010 Elsevier Inc. All rights reserved.
