A valid and reliable belief elicitation method for Bayesian priors

Johnson, Sindhu R.; Tomlinson, George A.; Hawker, Gillian A.; Granton, John T.; Grosbein, Haddas A.; Feldman, Brian M.

doi:10.1016/j.jclinepi.2009.08.005

« Previous Next »Journal of Clinical Epidemiology
Volume 63, Issue 4 , Pages 370-383, April 2010

A valid and reliable belief elicitation method for Bayesian priors

Sindhu R. Johnson
- Division of Rheumatology, Department of Medicine, Toronto Western Hospital, University Health Network, Toronto, Ontario, Canada
- Department of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada
- Corresponding author. Division of Rheumatology, Ground Floor, East Wing, Toronto Western Hospital, 399 Bathurst Street, Toronto, Ontario M5T 2S8, Canada. Tel.: +416-603-6417; fax: +416-603-4348.
George A. Tomlinson
- Department of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada
- Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada
- Division of Clinical Decision Making and Health Care, Toronto General Research Institute, Toronto, Ontario, Canada
Gillian A. Hawker
- Department of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada
- Division of Rheumatology, Department of Medicine, Women's College Hospital, Toronto, Ontario, Canada
John T. Granton
- Divisions of Respirology and Critical Care Medicine, Department of Medicine, Toronto General Hospital, Toronto, Ontario, Canada
Haddas A. Grosbein
- Division of Rheumatology, Department of Pediatrics, The Hospital for Sick Children, Toronto, Ontario, Canada
Brian M. Feldman
- Department of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada
- Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada
- Division of Rheumatology, Department of Pediatrics, The Hospital for Sick Children, Toronto, Ontario, Canada

Accepted 6 August 2009. published online 19 November 2009.

Abstract

Objective

Bayesian inference has the advantage of formally incorporating prior beliefs about the effect of an intervention into analyses of treatment effect through the use of prior probability distributions or “priors.” Multiple methods to elicit beliefs from experts for inclusion in a Bayesian study have been used; however, the measurement properties of these methods have been infrequently evaluated. The objectives of this study were to evaluate the feasibility, validity, and reliability of a belief elicitation method for Bayesian priors.

Study Design and Setting

A single-center, cross-sectional study using a sample of academic specialists who treat pulmonary hypertension patients was conducted to test the feasibility, face and construct validity, and reliability of a belief elicitation method. Using this method, participants expressed the probability of 3-year survival with and without warfarin. Applying adhesive dots or “chips,” each representing 5% probability, in “bins” on a line, participants expressed their uncertainty and weight of belief about the effect of warfarin on 3-year survival.

Results

Of the 12 participants, 11 (92%) reported that the belief elicitation method had face validity, 10 (83%) found the questions clear, and 11 (92%) found the response option easy to use. The median time to completion was 10 minutes (5–15 minutes). Internal validity testing found moderate agreement (weighted kappa=0.54–0.57). The intraclass correlation coefficient for test–retest reliability was 0.93.

Conclusion

This method of belief elicitation for Bayesian priors is feasible, valid, and reliable. It can be considered for application in Bayesian clinical studies.

Keywords: Belief elicitation, Bayesian, Validity, Reliability, Priors, Pulmonary hypertension