Real world research
Article Outline
The design and performance of studies that can really make a difference in improving practice is a topic that will never stop with keeping us busy. That is a good thing because there is no plausible reason that, although biomedical knowledge has a much higher turnover rate in continuously innovating and replacing itself, the methodology for clinical and health research would be stable over time. Indeed, the methodologic debate and development will never end.
In this issue, Freemantle starts out with a very instructive overview of the key characteristics of, as he calls them, “Real World Trials” (RWTs). These characteristics include the design choices that should be made in order to assure that external (clinical) validity will be optimal in addition to, of course, internal validity as basic requirement. The RWT concept has much in common with what has been designated as pragmatic trial, which was further elaborated in the Journal [1] and to which Freemantle presents useful additional considerations. At the same time, we must recognize that like ‘conventional’ randomized controlled trials, also RWTs are not always possible. In those situations, observational, that is, quasi-experimental studies are sometimes considered to enhance the body of therapeutic knowledge, and this remains a controversial issue [2], [3]. In this context, the study by Müeller et al. is very interesting. They used the availability of randomized and non-randomized studies carried out on the same issue (laparoscopic vs. open cholecystectomy) as an opportunity to make a comparative review. While in none of the examined outcome variables the results of both design types were in the opposite direction, the two approaches yielded significantly different results on a substantial proportion of relevant outcome variables. These results justify awareness, especially in the many cases that such comparisons are not possible in the absence of randomized studies. But Müeller et al. also emphasizes that non-randomized studies – which can generally be larger – may be in a better position to detect infrequent complications.
In this connection, the report of Sampson et al. is also interesting. Composite outcomes, in addition to being an opportunity to improve our understanding of various types of treatment effects and to obtain a more comprehensive view on them, are often assumed to increase statistical power in a trial. However, Sampson et al. could not confirm this assumption in a trial of cardiovascular therapy. Therefore, the added value of composite scores in giving a more complete picture of the real world clinical outcomes may be the most useful and justified reason to apply them. Speaking of a more complete picture, one problem is that not all important adverse effects occurring in clinical studies are always reported and published. In a systematic review of methodologic studies, Golder et al. show that detection and inclusion of unpublished studies in reviews can provide additional information on adverse effects and more precise pooled risk estimates, but there is insufficient evidence as to whether this would have a major influence on the results of meta-analyses. Because retrieving unpublished data is difficult and the balance between added value and extra effort should be reasonable, we underline the authors’ suggestions as to increasing the accessibility of adverse effect data from unpublished sources (including regulatory authorities and pharmaceutical companies). Another useful check on whether theoretically justified methodologic improvement is worth the required additional efforts is reported by Parmar et al. In a prospective study, they compared the assessment of suspected cardiovascular outcomes with or without blinding for important patient characteristics. Interestingly, in this case, blinding did not seem to have relevant added value.
Real world research will also be promoted by better and more representative participation by the relevant target groups [4]. In this area, an under-researched and too often neglected issue is the specific methodologic requirements and challenges in representatively sampling for studies on the health of people with intellectual disabilities [5]. In an international comparative study, Veenstra et al., also addressing ethical questions, make an important contribution to achieving further progress toward international consensus on respectful consent procedures and tailored patient information. Sustained participation is a well known concern in long-term follow-up studies, as is demonstrated by Hill and colleagues for the UK Collaborative HIV Cohort study. They show that, when confronted with a substantial proportion of patients being potentially lost to follow-up, data linkage and identifying predictors of loss to follow-up may help investigators to reduce bias in such observational databases.
To get a better grip on participation of subjects eligible for clinical trials, Frew et al. developed an instrument to assess a patient’s willingness to participate (the Clinical Research Involvement Scales [CRIS]), and conclude that this instrument, while further replication and reliability studies are needed, may indeed have supportive value. Because in addition to good participation, comparability of groups is essential in trials, Perry et al. studied a technique that is sometimes applied with the objective of improving the strength of a trial (minimization), aiming to achieve a better balance for prognostic factors [6]. However, because minimization may introduce selection bias, the authors describe and evaluate a new method of treatment allocation (study wise minimization) to both avoid imbalance and reduce selection bias, which may be particularly useful in small trials. They report promising results from a simulation study.
Interesting studies in evaluating instruments to measure relevant health outcome are presented by the groups of Fletcher and Nosyk. Fletcher et al. compared prospective and retrospective measures of health-related quality of life in patients with chronic cough, concluding that the first approach is likely to be more useful and accurate. In a comparison of not less that 8 health status measures applied among chronic opioid dependent patients, Nosyk et al. showed that none of those instruments was obviously the best or worst, and that their results provide an evidence base to inform selection and development of appropriate instruments in opioid-dependent populations. Comparison of methods also turned out to be useful in a study by Austin and co-workers, who evaluated the performance of logistic regression models and regression trees in predicting in-hospital mortality in a large group of heart failure patients. They report that logistic regression predicted more accurately, although the findings may not be applicable to patients with other diseases or for other outcomes.
Earlier in this editorial, a contribution on the comparison of randomized and non-randomized treatment studies was mentioned. One may have the impression that inferences from observational studies applied to increase therapeutic insights are more critically encountered than observational studies on other (e.g., etiological or prognostic) issues. However, it is not evident that biases in the latter are less likely than in the first. Understandably, observational studies into etiology or prognosis are probably more readily accepted because random assignment of etiologic and prognostic factors is even more difficult than randomization in treatment research. However, for the debate on methodologic quality, there is no convincing specific reason not to apply the same rigorous methodologic ambitions to both alike, even though the direct implications of translating study results in clinical treatment are often, whether or not justified, considered more risky than the implications of translating study results into etiologic or prognostic insights. At least, the various domains can learn a lot from one another, although scientific cultures may have developed quite differently given the different types of research questions and applications [7]. Therefore, the contributions of Shamliyan et al. and Groenwold & Rovers on the appraisal of observational studies on the occurrence of and risk factors for diseases are welcomed. Shamiliyan evaluated checklists and scales used for quality assessment of such studies, and came to the conclusion that there was substantial variation as to content, format, validity, and applicability, and that development and validation were not consistently reported. Also, the tools do not discriminate poor reporting from quality of the studies themselves. While Shamiliyan recommends transparent quality assessments in the future, in their Commentary Groenwold & Rover state that a comprehensive checklist on the quality of observational studies seems impossible, and recommend to closely adhere to the STROBE statement on clear reporting, to make sure that published studies can be appropriately judged [8]. We are interested in a follow-up debate on this issue from our readers perspective.
References
- . A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. J Clin Epidemiol. 2009;62:464–475
- . The role of observational studies in the evaluation of therapy. Stat Med. 1984;3:341–345
- . Randomized trials and observational studies: still mostly similar results, still crucial differences. J Clin Epidemiol. 2008;61:207–208
- . Medicine based evidence, a prerequisite for evidence based medicine. BMJ. 1997;315(7116):1109–1110
- . People with intellectual disability in general practice: case definition and case finding. J Intellect Disabil Res. 1997;41:373–379
- . On stratification, minimization and protection against types 1 and 2 error. J Clin Epidemiol. 2001;54:104–105
- . Double standards, scientific methods, and epidemiologic research. N Engl J Med. 1982;307:1611–1617
- . STROBE Initiative. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol. 2008;61:344–349
PII: S0895-4356(10)00265-9
doi:10.1016/j.jclinepi.2010.08.001
© 2010 Elsevier Inc. All rights reserved.
