Journal of Clinical Epidemiology

Editorial Board

2010-04-01

Expectation and (un)predictability

J. André Knottnerus, Peter Tugwell — 2010-04-01

Clinical and scientific expectation and prediction, in a context of uncertainty, are basic concepts for developing, designing, and performing clinical research. This is reflected in formulating appropriate and promising research hypotheses while also facing the ethical dilemmas related to required equipoise, in estimating prior probabilities, in designing adequate procedures for decision making during studies, and in defining the framework for correctly interpreting the results. Moreover, for the implementation of results in clinical practice, more individualized predictions would be welcome . Given the context of uncertainty, which is in fact the justification of doing the research, these are not easy tasks, and we can always learn from one another's experiences from clinical trials, and etiologic, diagnostic, and prognostic research.

A probiotics trial on trial: the problem of timely detection of adverse advents in therapeutic trials

Raimond W.M. Giard — 2009-09-18

The outcome of a therapeutic trial may be frustrating. In a recent Dutch randomized, double-blind, placebo-controlled multicenter trial, the effects of probiotics were studied to investigate their potential to diminish infectious complications in patients with predicted severe acute pancreatitis . Desolately, the results of this study were quite contrary to expectations: there was no diminution of infectious complications, and furthermore, patients taking the probiotics had more than double the relative mortality risk. Finally, 33 of the 297 patients included in this trial died with an excess of 15 deaths occurring in the treatment group ().

When should an interim analysis be unblinded to the data monitoring committee?

J. André Knottnerus, Mark G. Spigt — 2009-09-18

When designing a trial, an interim analysis of the trial results may be considered, for example, halfway to the maximal follow-up time, or after a prespecified number of inclusions or events. The objectives of such an analysis include (1) avoiding exposure of further patients to harm if detected in the interim analysis; (2) not continuing exposure of further patients to the trial if unneeded for answering the study question; (3) avoiding unnecessary research costs; (4) providing an answer to the study question as soon as possible. Interim analysis is especially relevant if the tested intervention is invasive, potentially hazardous, burdensome, or expensive, or if the condition or outcome under study represents severe pathology . Another use is when prior knowledge is lacking on the probability of the outcomes and adverse effects of interest in the study population.

Interim analysis in randomized trials: DAMOCLES' sword?

Marc G.H. Besselink, Yolanda van der Graaf, Hein G. Gooszen — 2009-12-04

Dr. Giard effectively summarizes the difficulties a Data Safety Monitoring Committee (DSMC) faces when deciding whether or not to terminate a trial early. We can comply with the suggested decision matrix by Knottnerus and Spigt, but feel that this matrix should be predefined in the study protocol. Furthermore, we and many others feel that a DSMC should take additional information into account as well. Pocock stated: “Negative stopping decisions cannot simply be based on statistical guidelines. For instance, if negative interim data are in contradiction with previous, more positive evidence, then the quality, extent, and relevance of that evidence will influence one's judgment” . In 2005, the DAMOCLES guidelines on clinical trial data monitoring committees stated: “Statistical issues should be only one of several considerations that a DMC needs to take into account. Other considerations include the balance of primary risks and benefits, the internal consistency of results, the consistency with, and nature of, external evidence, and the likelihood that the results would affect clinical practice. Statistical criteria (often called stopping rules) should be agreed in advance and regarded as guidelines for recommending stopping rather than rules” .

Methods to elicit beliefs for Bayesian priors: a systematic review

2009-08-28

Abstract: Objective: Bayesian analysis can incorporate clinicians' beliefs about treatment effectiveness into models that estimate treatment effects. Many elicitation methods are available, but it is unclear if any confer advantages based on principles of measurement science. We review belief-elicitation methods for Bayesian analysis and determine if any of them had an incremental value over the others based on its validity, reliability, and responsiveness.Study Design and Setting: A systematic review was performed. MEDLINE, EMBASE, CINAHL, Health and Psychosocial Instruments, Current Index to Statistics, MathSciNet, and Zentralblatt Math were searched using the terms (prior OR prior probability distribution) AND (beliefs OR elicitation) AND (Bayes OR Bayesian). Studies were evaluated on: design, question stem, response options, analysis, consideration of validity, reliability, and responsiveness.Results: We identified 33 studies describing methods for elicitation in a Bayesian context. Elicitation occurred in cross-sectional studies (n=30, 89%), to derive point estimates with individual-level variation (n=19; 58%). Although 64% (n=21) considered validity, 24% (n=8) reliability, 12% (n=4) responsiveness of the elicitation methods, only 12% (n=4) formally tested validity, 6% (n=2) tested reliability, and none tested responsiveness.Conclusions: We have summarized methods of belief elicitation for Bayesian priors. The validity, reliability, and responsiveness of elicitation methods have been infrequently evaluated. Until comparative studies are performed, strategies to reduce the effects of bias on the elicitation should be used.

A valid and reliable belief elicitation method for Bayesian priors

2009-11-19

Abstract: Objective: Bayesian inference has the advantage of formally incorporating prior beliefs about the effect of an intervention into analyses of treatment effect through the use of prior probability distributions or “priors.” Multiple methods to elicit beliefs from experts for inclusion in a Bayesian study have been used; however, the measurement properties of these methods have been infrequently evaluated. The objectives of this study were to evaluate the feasibility, validity, and reliability of a belief elicitation method for Bayesian priors.Study Design and Setting: A single-center, cross-sectional study using a sample of academic specialists who treat pulmonary hypertension patients was conducted to test the feasibility, face and construct validity, and reliability of a belief elicitation method. Using this method, participants expressed the probability of 3-year survival with and without warfarin. Applying adhesive dots or “chips,” each representing 5% probability, in “bins” on a line, participants expressed their uncertainty and weight of belief about the effect of warfarin on 3-year survival.Results: Of the 12 participants, 11 (92%) reported that the belief elicitation method had face validity, 10 (83%) found the questions clear, and 11 (92%) found the response option easy to use. The median time to completion was 10 minutes (5–15 minutes). Internal validity testing found moderate agreement (weighted kappa=0.54–0.57). The intraclass correlation coefficient for test–retest reliability was 0.93.Conclusion: This method of belief elicitation for Bayesian priors is feasible, valid, and reliable. It can be considered for application in Bayesian clinical studies.

Challenges in the validation of triage systems at emergency departments

Henriëtte A. Moll — 2009-10-30

Abstract: Objectives: Triage systems, developed by consensus of experts and based on decision rules, are typically not validated. The objective is to discuss the challenges to evaluate the reliability and validity of triage systems.Study Design and Setting: Theoretical–conceptual approach to validate triage systems.Results: The consensus-based triage systems have to be applied to a broad population with a variety of signs and symptoms. For the individual patient-specific decision, rules are used and the outcome measure is, typically, one of five prognosis-specific urgency categories. In contrast, prediction rules in diagnostic research are developed for a narrow specific subpopulation and based on a combination of parameters to predict presence of a specific diagnosis. Reliability is based on case scenario and simultaneous triage studies. The first step in triage validation is to decide on the best proxy for prognosis, “the reference standard” for the urgency classification. The next step is modification of the triage decision rules, including a multivariate approach. The final step is the validation in different settings and to evaluate the impact in clinical practice.Conclusion: Triage should be viewed as diagnostic research and would benefit if it would use the available methodology in diagnostic research.

Quantifying the magnitude of risk for balance impairment on falls in community-dwelling older adults: a systematic review and meta-analysis

Susan W. Muir, Katherine Berg, Bert Chesworth, Neil Klar, Mark Speechley — 2009-09-10

Abstract: Objectives: To evaluate and summarize the evidence linking balance impairment as a risk factor for falls in community-dwelling older adults.Study Design and Setting: Systematic review and meta-analysis. English language articles in MEDLINE, EMBASE, CINAHL (1988–2009), under keywords of accidental falls, aged, risk factors, and hip, radius, ulna, and humerus fractures; and bibliographies of retrieved articles. Community-dwelling older adults in a prospective study, at least 1-year duration, age more than 60 years, and samples not specific to a single disease-defined population were included. Sample size, inclusion/exclusion criteria, demographics, clinical balance measurement scale, type of fall outcome, method of fall ascertainment, length of follow-up, and odds ratio (OR) or risk ratio (RR) were extracted. Studies must have reported adjustment for confounders. Random effects meta-analysis to generate summary risk estimate was used. A priori evaluation of sources of heterogeneity was performed.Results: Twenty-three studies met the selection criteria. A single summary measure could not be calculated because of the nonequivalence of the OR and RR, producing an overall fall risk of RR of 1.42 (1.08, 1.85) and OR of 1.98 (1.60, 2.46).Conclusions: Balance impairment imparts a moderate increase on fall risk in community-dwelling older adults. The type of fall outcome, the length of follow-up, and the balance measurement tool impact the magnitude of the association. Specific balance measurement scales were identified with associations for an increased fall risk, but further research is required to refine recommendations for their use in clinical practice.

Outpatient urticaria diagnosis codes have limited predictive value for same-day influenza vaccine adverse event detection

2009-11-05

Abstract: Objectives: To assess the predictive value of claims-based outpatient urticaria diagnosis codes to identify potential vaccine-related adverse events (AEs) when recorded on the same day as influenza vaccination.Study Design and Setting: Health plan members with outpatient claims for influenza vaccination and urticaria on the same day between October 1, 2002, and December 31, 2007, were eligible for inclusion. Electronic medical records (EMRs) for 50 eligible patients with the most recent visits of interest occurring at a large group practice were sampled for review.Results: EMRs were available and reviewed for 42 of 50 patients. An influenza vaccination was confirmed in all reviewed medical charts. Urticaria occurring on the day of influenza vaccination was confirmed for 40% of participants (17/42); 3 confirmed urticaria diagnoses were potential AEs and 14 urticaria events occurred before vaccination. Among those with unconfirmed diagnoses, 17 had no evidence of urticaria on physical examination on the day of interest (4 had evidence of a nonurticarial rash and 13 had no evidence of rash on examination) and 8 had insufficient information to make a clinical determination.Conclusion: Outpatient diagnosis codes for urticaria found in health insurance claims data are limited in their predictive value to identify same-day vaccine AEs.

The predictors of self-rated health and the relationship between self-rated health and health service needs are similar across socioeconomic groups in Canada

Peter M. Smith, Richard H. Glazier, Lyn M. Sibley — 2009-11-18

Abstract: Objective: To examine if there are systematic differences in the predictors of self-rated health (SRH) and to examine the relationship between SRH and health care utilization across socioeconomic groups.Study Design and Setting: We used cross-sectional data from the Canadian Community Health Survey linked to the Ontario Health Insurance Plan (N=17,109). We examined relative differences in the factors associated with different levels of SRH across socioeconomic groups (as assessed by education and household income) using probit models separately for men and women. We then examined differences in expected health care costs, as assessed by adjusted clinical group weights using administrative health care records, between socioeconomic groups within the same level of SRH.Results: We found limited differences across the predictive ability of a broad range of physical, mental, health service/care utilization, and health behavior variables on SRH across socioeconomic groups. In addition, no differences were found in the expected health care utilization costs across socioeconomic groups within the same level of SRH.Conclusions: The results of this study suggest that SRH assesses a broad variety of factors, including physical health status, mental health status, health service/care utilization, and health behaviors, relatively equally across socioeconomic groups, measured as either education or income.

Construction of drug treatment episodes from drug-dispensing histories is influenced by the gap length

2009-11-02

Abstract: Objectives: When constructing drug treatment episodes using drug-dispensing databases, duration and the number of prescriptions belonging to a single treatment episode need to be defined. We investigated how different methods used to construct antidepressant treatment episodes influence their median estimated length.Study Design and Setting: A follow-up study among adult antidepressant drug users, identified from the Dutch PHARMO RLS, starting selective serotonin reuptake inhibitor (SSRI) use in 2001 was conducted. The influence of varying lengths of the prescription overlap and the gap between prescriptions (number of days or percentage of prescription duration) on the median antidepressant treatment episode length were investigated.Results: Of the 16,053 SSRI starters, 65.1% were female and mean age was 45.7 (SD: 17.2) years. Median antidepressant treatment episode length doubled when the gap length was expanded from 0 to 10 days. For short gap lengths the episode interquartile range was 40% to 200% larger when overlap was accounted for and when percentage of prescription duration gap length was used.Conclusion: Differences in median episode length exist between methods that account for or disregard prescription overlap. These differences are of importance for studies that focus on drug exposure-outcome relationships and could have consequences for epidemiological analysis.

Older Australians' medication use: self-report by phone showed good agreement and accuracy compared with home visit

Sabrina Pit, Julie Byles — 2009-09-29

Abstract: Objective: To ascertain the accuracy of telephone-interview method for measuring older people's medication use (“self-report by phone”) by determining agreement between results from this method and from a home visit (“home inventory”).Study Design and Setting: An agreement study involving community-dwelling patients aged more than 65 years, selected from four general practices in the Hunter Region of Australia. Commonly used classes of drugs were selected for comparison.Results: Of 154 patients, 14 participants were ineligible, because they had hearing problems (9) or did not use any medicines (5). The response rate was 70% (98 of 140). The observed overall agreement and prevalence-adjusted and bias-adjusted kappa coefficients were very high for all prescribed drug categories, but lower for over-the-counter (OTC) and complementary medicines. Specificity of the self-report by phone compared with home inventory was consistently high across all drug classes. Sensitivity values were more than 89% for all drug classes but were lower for OTC and complementary medicines (74%) and paracetamol (78%). Similar patterns were found for negative predictive values. Positive predictive values were lower for drugs used on an as-needed basis.Conclusion: Measuring patient's medication use by telephone is an accurate and relatively inexpensive alternative to home-inventory methods, and has merit for use in future studies of older patients' drug use.

Proxy reports of physical activity were valid in older people with and without cognitive impairment

Laura E. Middleton, Susan A. Kirkland, Arnold Mitnitski, Kenneth Rockwood — 2009-09-14

Abstract: Objective: To determine the validity of proxy reports of physical activity in people with symptoms of cognitive impairment.Study Design and Setting: In the Canadian Study of Health and Aging, a multicenter prospective cohort study, someone close to the participant (proxy) reported exercise levels for people who screened positive for cognitive impairment or were institutionalized (n=2421), some of whom were subsequently diagnosed with cognitive impairment (n=1612) and some of whom were diagnosed as having no cognitive impairment (n=809). The reliability and validity of proxy reports of physical activity were examined by agreement with self-reports of physical activity (intraclass correlation coefficient) and by association with adverse health markers (Mantel–Haenzel χ2) and survival time (Cox proportional hazards).Results: Proxy reports of physical activity had moderate interrater reliability (0.55, 95% confidence interval: 0.49–0.61, P<0.001). People in higher physical activity group had fewer adverse health outcomes than those in lower physical activity groups. Predictive validity was confirmed as people who had higher proxy-reported physical activity survived longer that those with lower physical activity.Conclusion: Proxy-reported physical activity appears to be a valid estimate of physical activity in people with symptoms or a diagnosis of cognitive impairment.

Decreased accuracy in interpretation of community-based screening mammography for women with multiple clinical risk factors

2009-09-10

Abstract: Objective: To assess the impact of women's breast cancer risk factors (use of hormone therapy, family history of breast cancer, previous breast biopsy) on radiologists' mammographic interpretive performance and whether the influence of risk factors varies according to radiologist characteristics.Study Design and Setting: Screening mammograms (n=638,947) performed from 1996 to 2005 by 134 radiologists from three Breast Cancer Surveillance Consortium registries was linked to cancer outcomes, radiologist surveys, and patient questionnaires. Interpretive performance measures were modeled using marginal and conditional logistic regression.Results: Having one or more clinical risk factors was associated with higher recall rates (1 vs. 0 risk factors: odds ratio [OR]=1.17, 95% confidence interval [CI]=1.15–1.19; ≥2 vs. 0: OR=1.43, 95% CI=1.40–1.47) and lower specificity (1 vs. 0: OR=0.86 [95% CI=0.84–0.88]; ≥2 vs. 0: OR=0.70 [95% CI=0.68–0.72]) without a corresponding improvement in sensitivity and only a small increase in positive predictive value (1 vs. 0: OR=1.08 [95% CI=0.99–1.19]; ≥2 vs. 0: OR=1.12 [95% CI=0.99–1.26]). There was no indication that influence of risk factors varied by radiologist characteristics.Conclusion: Women with clinical risk factors who undergo screening mammography are more likely recalled for false-positive evaluation without an associated increase in cancer detection. Radiologists and patients with risk factors should be aware of this increased risk of adverse screening events.

Cascade effects of laboratory testing are found to be rare in low disease probability situations: prospective cohort study

Paul H.H. Houben, Trudy van der Weijden, Ron A.G. Winkens, Richard P.T.M. Grol — 2009-11-02

Abstract: Objectives: (1) To investigate the frequency of cascades of further diagnostic investigations and referrals after abnormal laboratory results in situations of low disease probability; (2) to investigate pretest and posttest determinants; and (3) to describe the cascades that occur.Study Design and Setting: Prospective cohort study in primary care in The Netherlands. Numbers of investigations/referrals were recorded during 6 months of follow-up for 256 patients with normal and abnormal laboratory results. The influences of the reason for ordering tests, interpretation of results, and pretest/posttest disease probability were examined.Results: After receiving the laboratory results, the physicians ordered further investigations for 22 (17.3%) patients with abnormal results and for two (1.6%) patients with normal results (P<0.001). They referred 12 (9.4%) patients with abnormal results and eight (6.2%) patients with normal results (P=0.33). Six patients had two investigations and/or referrals, and one patient had three referrals. There were significantly more investigations/referrals for results interpreted as abnormal (P=0.004) and for cases with a high posttest disease probability (P=0.001).Conclusion: This study suggests that cascade processes after laboratory testing in situations of low disease probability are limited in magnitude and frequency. Improving interpretations may help improve the appropriateness of further investigations and referrals.

A comparison of ad hoc methods to account for non-cancer AIDS and deaths as competing risks when estimating the effect of HAART on incident cancer AIDS among HIV-infected men

2009-11-02

Abstract: Objective: To compare three ad hoc methods to estimate the marginal hazard of incident cancer acquired immune deficiency syndrome (AIDS) in a highly active antiretroviral therapy (1996–2006) relative to a monotherapy/combination therapy (1990–1996) calendar period, accounting for other AIDS events and deaths as competing risks.Study Design and Setting: Among 1,911 human immunodeficiency virus (HIV)-positive men from the Multicenter AIDS Cohort Study, 228 developed cancer AIDS and 745 developed competing risks in 14,202 person-years from 1990 to 2006. Method 1 censored competing risks at the time they occurred, method 2 excluded competing risks, and method 3 censored competing risks at the date of analysis.Results: The age, race, and infection duration adjusted hazard ratios (HRs) for cancer AIDS were similar for all methods (HR≈0.15). We estimated bias and confidence interval coverage of each method with Monte Carlo simulation. On average, across 24 scenarios, method 1 produced less-biased estimates than methods 2 or 3.Conclusions: When competing risks are independent of the event of interest, only method 1 produced unbiased estimates of the marginal HR, although independence cannot be verified from the data. When competing risks are dependent, method 1 generally produced the least-biased estimates of the marginal HR for the scenarios explored; however, alternative methods may be preferred.

Allocation concealment continues to be misunderstood

Vance W. Berger, Anh-Chi Do — 2009-12-11

Foley et al. lamented the fact that allocation concealment is described inadequately in two-thirds of the studies, but the flip side of this is the solace that one can take in knowing that at least allocation concealment is described adequately in one-third of the studies. Of course, Foley et al. also pointed out that allocation concealment continues to be misunderstood by many investigators, and we would like to echo this sentiment. In fact, it seems to be true even more than Foley et al. themselves recognize, and there is a particular irony here, a point to which we will return shortly. Foley et al. considered the reporting of allocation concealment to be adequate “if the authors clearly reported a mechanism through which it could be reasonably ascertained that the investigators had no foreknowledge of the treatment assignments.”

Concealed allocation: an under-reported and misunderstood component of trial methodology in stroke rehabilitation - reply

Norine C. Foley, Mark Speechley — 2009-12-08

We would like to thank Drs. Berger and Do for making us aware of Berger's work, which includes approaches for the prevention, detection, and statistical adjustment of the effects of third-order residual selection bias because of lack of concealed allocation (CA) in randomized trials . Having now acquainted ourselves with his work, we believe that our two groups probably share many values and beliefs. At a general level, we share a profound respect for the scientific method, including the necessity of observing proper methodology. More specifically, we share an interest in the problematic issue of CA in clinical trials. And finally, we would likely agree that CA is frequently misunderstood by authors and, by implication, editors and reviewers.

Journal of Clinical Epidemiology

Editorial Board

Table of Contents

Expectation and (un)predictability

A probiotics trial on trial: the problem of timely detection of adverse advents in therapeutic trials

When should an interim analysis be unblinded to the data monitoring committee?

Interim analysis in randomized trials: DAMOCLES' sword?

Methods to elicit beliefs for Bayesian priors: a systematic review

A valid and reliable belief elicitation method for Bayesian priors

Challenges in the validation of triage systems at emergency departments

Quantifying the magnitude of risk for balance impairment on falls in community-dwelling older adults: a systematic review and meta-analysis

Outpatient urticaria diagnosis codes have limited predictive value for same-day influenza vaccine adverse event detection

The predictors of self-rated health and the relationship between self-rated health and health service needs are similar across socioeconomic groups in Canada

Construction of drug treatment episodes from drug-dispensing histories is influenced by the gap length

Older Australians' medication use: self-report by phone showed good agreement and accuracy compared with home visit

Proxy reports of physical activity were valid in older people with and without cognitive impairment

Decreased accuracy in interpretation of community-based screening mammography for women with multiple clinical risk factors

Cascade effects of laboratory testing are found to be rare in low disease probability situations: prospective cohort study

A comparison of ad hoc methods to account for non-cancer AIDS and deaths as competing risks when estimating the effect of HAART on incident cancer AIDS among HIV-infected men

Allocation concealment continues to be misunderstood

Concealed allocation: an under-reported and misunderstood component of trial methodology in stroke rehabilitation - reply