Journal of Clinical Epidemiology

Editorial Board

2010-10-01

Real world research

J. André Knottnerus, Peter Tugwell — 2010-10-01

The design and performance of studies that can really make a difference in improving practice is a topic that will never stop with keeping us busy. That is a good thing because there is no plausible reason that, although biomedical knowledge has a much higher turnover rate in continuously innovating and replacing itself, the methodology for clinical and health research would be stable over time. Indeed, the methodologic debate and development will never end.

Real-world effectiveness of new medicines should be evaluated by appropriately designed clinical trials

Nick Freemantle, Thomas Strack — 2009-11-02

Abstract: Objectives: Health care providers, policy makers, and importantly patients themselves are increasingly interested in the outcomes of clinical trials yet often expect different questions to be addressed than those commonly asked in conventional phase 3 trials.Study Design and Setting: Review of methodological articles.Results: Conventional randomized controlled trials (RCTs) emphasize internal validity through standardization and control but by design reduce external validity, that is, generalizability of results and conclusions. Ongoing uncertainty about effectiveness or safety of medical interventions in the real world is the major driver for developing improved phase 3b and phase 4 study designs. Factors that should improve the relevance of these real-world trials (RWTs) include choice of endpoints; investigator specialty, appropriate patient selection criteria; emphasis on patient–physician interaction; admittance of relevant interventions in all study groups; and more flexible, simple, and possibly event-driven study visits and procedures, while maintaining randomization as a critical element to address confounders.Conclusion: Although we do not believe that RWTs will supplant conventional RCTs, properly designed RWTs will enrich our understanding of the effectiveness of new health care interventions and better inform patients and health care providers alike.

The Catch-22 of appraisals on the quality of observational studies

R.H.H. Groenwold, M.M. Rovers — 2010-10-01

Over the last two decades, several statements on the reporting of randomized trials (CONSORT), diagnostic studies (STARD), observational studies (STROBE), and systematic reviews (PRISMA) have been developed. There appears to be large consensus on these statements, which are the result of the joined effort of epidemiologists, statisticians, and journal editors. For example, the STROBE statement on the reporting of observational studies appears to be generally agreed on, and many biomedical journals ask their contributors to adhere to this statement . Apparently, consensus can be reached on how observational studies should be reported. Although the STROBE statement is intended to assess the quality of the reporting, it is explicitly not developed to judge the quality (validity) of observational studies .

A systematic review of tools used to assess the quality of observational studies that examine incidence or prevalence and risk factors for diseases

Tatyana Shamliyan, Robert L. Kane, Stacy Dickinson — 2010-10-01

Abstract: Objective: To create a comprehensive evaluation of checklists and scales used to evaluate observational studies that examine incidence or prevalence and risk factors for diseases.Study Design: We did a literature search of several databases to abstract format, content, development, and validation of the tools.Results: We identified 46 scales and 51 checklists. Forty-seven of these tools were created for therapeutic studies, 48 for risk factors, and 5 for incidence studies. Forty-seven percent were modifications of previously published peer-reviewed appraisals, 18% were developed based on methodological standards, and 35% did not report development. Twenty-two percent reported reliability and 10% the validation procedure. Tools did not discriminate poor reporting vs. methodological quality of studies or external vs. internal validity; 35% categorize quality by the presence of predefined major flaws in design or by total score from the scale. Level of evidence was proposed in 22% of the tools by criteria of causality or internal validity of the studies. Evaluation required different degrees of subjectivity.Conclusions: Format, length, and content varied substantially across available checklists and scales. Development, validation, and reliability were not consistently reported. Transparent objective quality assessments should be developed in the future.

Unpublished data can be of value in systematic reviews of adverse effects: methodological overview

Su Golder, Yoon K. Loke, Martin Bland — 2010-05-11

Abstract: Objective: To assess the impact of including unpublished data on adverse effects in systematic reviews.Study Design and Setting: We carried out a systematic review of methodological evaluations that compared the quantitative reporting of adverse effects data between published and unpublished sources, in particular, the frequency, rate, or risk of reported adverse effects. Included studies were sought from 10 databases as well as by checking references, handsearching, searching citations, and contacting experts.Results: We identified 6,218 potential articles yielding 10 relevant methodological evaluations.One evaluation found that adverse effects were reported more often in unpublished trials. For anecdotal case reports, two evaluations found a higher frequency of unpublished cases, whereas one study identified a greater number of published cases. Another evaluation indicated that differences in frequency of published and unpublished case reports were topic dependent.A comparison of relative risk estimates from five studies suggested no major systematic variation in risk estimates from published and unpublished studies.Conclusion: Inclusion of unpublished studies can provide additional adverse effects information and more precise risk estimates. However, there is insufficient evidence to indicate whether inclusion of unpublished studies has a major influence on the pooled risk estimates in meta-analyses of adverse effects.

Reported effects in randomized controlled trials were compared with those of nonrandomized trials in cholecystectomy

Dirk Müeller, Stefan Sauerland, Edmund A.M. Neugebauer, Marc Immenroth — 2010-03-26

Abstract: Objectives: Because external validity of randomized controlled trials (RCTs) may be insufficient, the performance of nonrandomized controlled trials (nRCTs) is increasingly debated. RCTs and nRCTs were compared using the example of laparoscopic vs. open cholecystectomy (LC vs. OC).Study Design and Setting: RCTs and nRCTs comparing LC and OC were identified by searching PubMed. To assess internal and external validity of the studies, patient characteristics, relative risks, and mean differences of RCTs and nRCTs were compared by meta-analytic techniques.Results: In total, 162 studies were analyzed (136 nRCTs and 26 RCTs). Significant discrepancies between RCT- and nRCT-based results were revealed for 3 of 15 variables: overall complications (P<0.021), wound infections (P<0.014), and length of hospital stay (P<0.005). In RCTs and in nRCTs, length of hospital stay and return to work were significantly reduced when using LC compared with OC. The results of nRCTs were more often heterogeneous among themselves (11 of 15) as compared with RCTs (4 of 15).Conclusion: The results of RCTs and nRCTs differ significantly in at least 20% of the variables. External validities of RCTs and nRCTs in LC vs. OC appear to be similar. Between-study heterogeneity was larger in nRCTs than in RCTs of cholecystectomy.

Sampling and ethical issues in a multicenter study on health of people with intellectual disabilities

2010-03-22

Abstract: Objectives: To study health inequalities in persons with intellectual disabilities, representative and unbiased samples are needed. Little is known about sample recruitment in this vulnerable group. This study aimed to determine differences in ethical procedures and sample recruitment in a multicenter research on health of persons with intellectual disabilities. Study questions regarded the practical sampling procedure, how ethical consent was obtained in each country, and which person gave informed consent for each study participant.Study Design and Setting: Exploratory, as part of a multicenter study, in 14 European countries. After developing identical guidelines for all countries, partners collected data on health indicators by orally interviewing 1,269 persons with intellectual disabilities. Subsequently, semistructured interviews were carried out with partners and researchers.Results: Identification of sufficient study participants proved feasible. Sampling frames differed from nationally estimated proportions of persons with intellectual disabilities living with families or in residential settings. Sometimes, people with intellectual disabilities were hard to trace. Consent procedures and legal representation varied broadly. Nonresponse data proved unavailable.Conclusion: To build representative unbiased samples of vulnerable groups with limited academic capacities, international consensus on respectful consent procedures and tailored patient information is necessary.

Data linkage reduces loss to follow-up in an observational HIV cohort study

2010-03-29

Abstract: Objective: To ascertain the degree of loss to follow-up in a cohort and to identify its predictors.Study Design and Setting: Human immunodeficiency virus (HIV)–infected individuals without CD4 cell counts for a year or more were defined as potentially lost to follow-up (LFU). Multivariable Poisson regression models identified the risk factors for potential LFU. Multivariable logistic regression models compared demographic and clinical characteristics of those who returned for care and those permanently LFU.Results: Of 16,595 patients under follow-up, 43.6% were potentially LFU at least once. Of these, 39.8% were considered permanently LFU and 60.2% were seen again after 1 year. Of 9,766 episodes when patients were potentially LFU, 59% resulted in the patient returning for care at the same clinic or at a different clinic. Compared with those permanently LFU, patients returning were more likely to have started highly active antiretroviral therapy, to have higher CD4 counts and viral loads, to be younger, and to have had more CD4 tests before LFU. They were less likely to have had a previous episode of potential LFU.Conclusions: A substantial proportion of patients in the UK Collaborative HIV Cohort study are potentially LFU. Data linkage identifies patients returning for care at different centers. Recognition of factors associated with LFU may help reduce this important source of bias in observational databases.

The likelihood of participation in clinical trials can be measured: the Clinical Research Involvement Scales

2010-03-22

Abstract: Objective: We developed the Clinical Research Involvement Scales (CRIS) to assess the willingness to participate in a clinical trial.Study Design and Setting: Diverse populations (N=919) aged 18 years or older from Atlanta, Georgia, were included in comprehensive testing of the 41-item CRIS instrument. The formative phase focused on item content for the new measures (n=54). Questionnaires from potential vaccine trial participants (n=865), collected at multiple time points, resulted in the evaluation of scale reliability and validity (i.e., attitudes, behavioral and normative beliefs, perceived social support for clinical research participation, social norm compliance, perceptions of the clinical research organization, and perceived relevance of the research endeavor).Results: Qualitative testing revealed adequate comprehension and content validity of the initial item set. The subjective norms domain (n=3) initially exhibited poor internal consistency in pilot testing (Cronbach's α=0.525); yet, rewording of the items resulted in consistently stable measurement improvement (Cronbach's α=0.850). Each of the CRIS subscales demonstrated extremely high reliability, ranging from 0.734 to 0.918. Confirmatory factor analysis verified item–factor relationships and determined construct and convergent validity (root mean square error of approximation=0.068; comparative fit index=0.835).Conclusions: CRIS is a reliable instrument for measuring community attitudes toward participation in biomedical research studies. Results of this study support the use of these scales to recruit diverse populations to clinical trials.

Studywise minimization: A treatment allocation method that improves balance among treatment groups and makes allocation unpredictable

2010-03-22

Abstract: Objectives: In randomized controlled trials with many potential prognostic factors, serious imbalance among treatment groups regarding these factors can occur. Minimization methods can improve balance but increase the possibility of selection bias. We described and evaluated the performance of a new method of treatment allocation, called studywise minimization, that can avoid imbalance by chance and reduce selection bias.Study Design and Setting: The studywise minimization algorithm consists of three steps: (1) calculate the imbalance for all possible allocations, (2) list all allocations with minimum imbalance, and (3) randomly select one of the allocations with minimum imbalance. We carried out a simulation study to compare the performance of studywise minimization with three other allocation methods: randomization, biased-coin minimization, and deterministic minimization. Performance was measured, calculating maximal and average imbalance as a percentage of the group size.Results: Independent of trial size and number of prognostic factors, the risk of serious imbalance was the highest in randomization and absent in studywise minimization. The largest differences among the allocation methods regarding the risk of imbalance were found in small trials.Conclusion: Studywise minimization is particularly useful in small trials, where it eliminates the risk of serious imbalances without generating the occurrence of selection bias.

A prospective global measure, the Punum Ladder, provides more valid assessments of quality of life than a retrospective transition measure

2010-03-22

Abstract: Objectives: We assessed the reliability and validity of two measures of change, one retrospective (the Global Rating of Change Scale [GRCS]) and one prospective (the Punum Ladder), and the relative utility of the two methods of assessing change and establishing the minimal important difference (MID) of the Cough Quality of Life Questionnaire (CQLQ), a reliable and valid cough-specific quality-of-life (QoL) instrument.Study Design and Setting: A prospective, longitudinal study assessing the change in cough-related QoL over 6 months in participants with chronic cough was carried out in a tertiary care cough clinic. Before seeing a physician, subjects completed eight Punum Ladders and the CQLQ. At 1 and 6 months, eight Punum Ladders, the CQLQ, and seven GRCSs were completed. Punum Ladders and GRCSs were psychometrically tested, and MIDs were calculated.Results: Reliability and validity of GRCSs and Punum Ladders were acceptable. However, closer analysis of the relation between change scores and CQLQ pretest and posttest scores showed that the GRCS was only related to patient's present state, whereas the Punum Ladder was associated with both initial and present states. This compromises the validity of the GRCS. Crosstab comparisons revealed that GRCS ratings made more liberal estimates of change in the CQLQ than the Punum Ladder; this is reflected in their respective MIDs (10.58±10.63 vs. 21.89±15.38).Conclusion: The prospective Punum Ladder is likely to be more useful, because it reflects the actual change in QoL over time in a less biased and more accurate way than the retrospective GRCS.

The quality of eight health status measures were compared for chronic opioid dependence

2010-03-18

Abstract: Objective: To provide a comparative analysis of the psychometric properties of eight measures of health status among chronic opioid-dependent patients.Study Design and Setting: Longitudinal data were analyzed for 251 patients enrolled in the North American Opiate Medication Initiative randomized controlled trial, conducted in Vancouver, British Columbia and Montreal, Quebec, Canada. Content validity, evidence of floor and ceiling effects, internal consistency, construct validity, and responsiveness were assessed for the Addiction Severity Index (ASI) medical and psychiatric (ASImed and ASIpsych) composite scores, the Maudesley Addiction Profile (MAP) physical and mental health scores (MAP-physical health score [MAP-PHS], MAP-mental health score [MAP-MHS]), the World Health Organization Disability Assessment Schedule-II, the EuroQol Group's EQ-5D index score and visual analog scale, EuroQol visual analog scale (EQ-VAS), and the Short Form SF-6D index score.Results: ASImed was best able to discriminate among patients with and without chronic conditions. The MAP-PHS and MAP-MHS were not unidimensional. ASImed and ASIpsych had prominent ceiling effects. ASImed, MAP-MHS, MAP–PHS, EQ-VAS, and EQ-5D were all responsive to decreases in illicit drug use.Conclusion: None of the instruments performed uniformly as “best” or “worst.” The EQ-5D appeared to be the preferable generic, indirect utility measure. Our results provide an evidence base to inform selection and further development of health status measures in opioid-dependent populations.

Logistic regression had superior performance compared with regression trees for predicting in-hospital mortality in patients hospitalized with heart failure

Peter C. Austin, Jack V. Tu, Douglas S. Lee — 2010-03-22

Abstract: Objective: To compare the predictive accuracy of regression trees with that of logistic regression models for predicting in-hospital mortality in patients hospitalized with heart failure.Study Design and Setting: Models were developed in 8,236 patients hospitalized with heart failure between April 1999 and March 2001. Models included the Enhanced Feedback for Effective Cardiac Treatment and Acute Decompensated Heart Failure National Registry (ADHERE) regression models and tree. Predictive accuracy was assessed using 7,608 patients hospitalized between April 2004 and March 2005.Results: The area under the receiver operating characteristic curve for five different logistic regression models ranged from 0.747 to 0.775, whereas the corresponding values for three different regression trees ranged from 0.620 to 0.651. For the regression trees grown in 1,000 random samples drawn from the derivation sample, the number of terminal nodes ranged from 1 to 6, whereas the number of variables used in specific trees ranged from 0 to 5. Three different variables (blood urea nitrogen, dementia, and systolic blood pressure) were used for defining the first binary split when growing regression trees.Conclusion: Logistic regression predicted in-hospital mortality in patients hospitalized with heart failure more accurately than did the regression trees. Regression trees grown in random samples from the same data set can differ substantially from one another.

Composite outcomes: weighting component events according to severity assisted interpretation but reduced statistical power

2010-06-17

Abstract: Objective: In trials of chronic disease therapy, each patient may experience several nonfatal illnesses and death. “Composite” outcome measures combine information from these different components of disease burden. Most common is the binary distinction between patients undergoing one or more events and those undergoing no events. We compare this approach with a composite score that preserves information on the number and severity of events.Study Design and Setting: The binary composite measure and composite score were derived for each patient in a trial of cardiovascular therapy. All nonfatal events contributed to the composite score according to their severity: recurrent myocardial infarction (weight 0.5), congestive heart failure that required the use of open-label angiotensin-converting enzyme (ACE) inhibitors (weight 0.2), and hospitalization to treat congestive heart failure (weight 0.5).Results: In the example data set, the composite score required a 10% larger sample size to achieve the same power as the binary measure. However, the composite score suggested that the treatment impacted on the first nonfatal event and mortality only.Conclusions: The composite score provides a more informative measure of disease burden and may avoid overestimating the evidence supporting a treatment effect when that evidence is largely from less severe early events.

Cardiovascular outcome ascertainment was similar using blinded and unblinded adjudicators in a national prospective study

2010-04-30

Abstract: Objective: Observational studies can avoid biases by blinding medical records to characteristics of interest before outcome adjudication. However, blinding is costly. We assessed the effect of blinding race and geography on outcome ascertainment.Study Design and Setting: The Reasons for Geographic and Racial Differences in Stroke - Myocardial Infarction (REGARDS-MI) Study is an ancillary study to the REGARDS national prospective cohort study including 30,228 participants. The primary characteristics of interest are race and geography, and the prespecified acceptable agreement rate between adjudicators is set at less than 80%. We selected 116 suspected cardiovascular events that underwent adjudication with usual blinding. At least 3 months later, cases were readjudicated without blinding race and geographic location of the patient. We assessed differences in outcome ascertainment using Cohen's κ statistic and ARs.Results: Agreement between blinded and unblinded reviews was good to excellent for all four outcomes. κ statistics were 0.80 (chest pain), 0.85 (heart failure), 0.86 (revascularization), and 0.74 (MI) (P<0.0001 for all). Within each outcome, ARs were similar for race and geographic groups (agreement: 83–100%).Conclusion: In observational studies, blinding medical record review for outcome ascertainment for some types of patient characteristics may cause an unwarranted expense.

2010-10-01

Journal of Clinical Epidemiology

Editorial Board

Real world research

Real-world effectiveness of new medicines should be evaluated by appropriately designed clinical trials

The Catch-22 of appraisals on the quality of observational studies

A systematic review of tools used to assess the quality of observational studies that examine incidence or prevalence and risk factors for diseases

Unpublished data can be of value in systematic reviews of adverse effects: methodological overview

Reported effects in randomized controlled trials were compared with those of nonrandomized trials in cholecystectomy

Sampling and ethical issues in a multicenter study on health of people with intellectual disabilities

Data linkage reduces loss to follow-up in an observational HIV cohort study

The likelihood of participation in clinical trials can be measured: the Clinical Research Involvement Scales

Studywise minimization: A treatment allocation method that improves balance among treatment groups and makes allocation unpredictable

A prospective global measure, the Punum Ladder, provides more valid assessments of quality of life than a retrospective transition measure

The quality of eight health status measures were compared for chronic opioid dependence

Logistic regression had superior performance compared with regression trees for predicting in-hospital mortality in patients hospitalized with heart failure

Composite outcomes: weighting component events according to severity assisted interpretation but reduced statistical power

Cardiovascular outcome ascertainment was similar using blinded and unblinded adjudicators in a national prospective study

Table of Contents