Advertisement

A valid and reliable belief elicitation method for Bayesian priors

  • Sindhu R. Johnson
    Correspondence
    Corresponding author. Division of Rheumatology, Ground Floor, East Wing, Toronto Western Hospital, 399 Bathurst Street, Toronto, Ontario M5T 2S8, Canada. Tel.: +416-603-6417; fax: +416-603-4348.
    Affiliations
    Division of Rheumatology, Department of Medicine, Toronto Western Hospital, University Health Network, Toronto, Ontario, Canada

    Department of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada
    Search for articles by this author
  • George A. Tomlinson
    Affiliations
    Department of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada

    Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada

    Division of Clinical Decision Making and Health Care, Toronto General Research Institute, Toronto, Ontario, Canada
    Search for articles by this author
  • Gillian A. Hawker
    Affiliations
    Department of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada

    Division of Rheumatology, Department of Medicine, Women's College Hospital, Toronto, Ontario, Canada
    Search for articles by this author
  • John T. Granton
    Affiliations
    Divisions of Respirology and Critical Care Medicine, Department of Medicine, Toronto General Hospital, Toronto, Ontario, Canada
    Search for articles by this author
  • Haddas A. Grosbein
    Affiliations
    Division of Rheumatology, Department of Pediatrics, The Hospital for Sick Children, Toronto, Ontario, Canada
    Search for articles by this author
  • Brian M. Feldman
    Affiliations
    Department of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada

    Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada

    Division of Rheumatology, Department of Pediatrics, The Hospital for Sick Children, Toronto, Ontario, Canada
    Search for articles by this author
Published:November 19, 2009DOI:https://doi.org/10.1016/j.jclinepi.2009.08.005

      Abstract

      Objective

      Bayesian inference has the advantage of formally incorporating prior beliefs about the effect of an intervention into analyses of treatment effect through the use of prior probability distributions or “priors.” Multiple methods to elicit beliefs from experts for inclusion in a Bayesian study have been used; however, the measurement properties of these methods have been infrequently evaluated. The objectives of this study were to evaluate the feasibility, validity, and reliability of a belief elicitation method for Bayesian priors.

      Study Design and Setting

      A single-center, cross-sectional study using a sample of academic specialists who treat pulmonary hypertension patients was conducted to test the feasibility, face and construct validity, and reliability of a belief elicitation method. Using this method, participants expressed the probability of 3-year survival with and without warfarin. Applying adhesive dots or “chips,” each representing 5% probability, in “bins” on a line, participants expressed their uncertainty and weight of belief about the effect of warfarin on 3-year survival.

      Results

      Of the 12 participants, 11 (92%) reported that the belief elicitation method had face validity, 10 (83%) found the questions clear, and 11 (92%) found the response option easy to use. The median time to completion was 10 minutes (5–15 minutes). Internal validity testing found moderate agreement (weighted kappa=0.54–0.57). The intraclass correlation coefficient for test–retest reliability was 0.93.

      Conclusion

      This method of belief elicitation for Bayesian priors is feasible, valid, and reliable. It can be considered for application in Bayesian clinical studies.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Berry D.A.
        Bayesian clinical trials.
        Nat Rev Drug Discov. 2006; 5: 27-36
        • Chaloner K.
        • Rhame F.S.
        Quantifying and documenting prior beliefs in clinical trials.
        Stat Med. 2001; 20: 581-600
      1. Johnson SR, Tomlinson GA, Hawker GA, Granton JT, Feldman BM. Methods to elicit beliefs for Bayesian priors. A systematic review. J Clin Epidemiol. In press.

        • Moye L.A.
        Bayesians in clinical trials: asleep at the switch.
        Stat Med. 2008; 27: 469-482
        • Hughes M.D.
        Practical reporting of Bayesian analyses of clinical trials.
        Drug Inf J. 1991; 25: 381-393
        • Parmar M.K.
        • Spiegelhalter D.J.
        • Freedman L.S.
        The CHART trials: Bayesian design and monitoring in practice. CHART Steering Committee.
        Stat Med. 1994; 13: 1297-1312
        • Parmar M.K.B.
        • Ungerleider R.S.
        • Simon R.
        Assessing whether to perform a confirmatory randomized clinical trial.
        J Natl Cancer Inst. 1996; 88: 1645-1651
        • Tan S.-B.
        • Chung Y.-F.
        • Tai B.-C.
        • Cheung Y.-B.
        • Machin D.
        Elicitation of prior distributions for a phase III randomized controlled trial of adjuvant therapy with surgery for hepatocellular carcinoma.
        Control Clin Trials. 2003; 24: 110-121
        • Johnson S.R.
        • Tomlinson G.A.
        • Hawker G.A.
        • Granton J.T.
        • Feldman B.M.
        Methods to elicit beliefs for Bayesian priors: a systematic review.
        J Clin Epidemiol. 2010; 63: 355-369
        • Hogarth R.M.
        Cognitive processes and the assessment of subjective probability distributions.
        J Am Stat Assoc. 1975; 70: 271-294
        • Wallsten T.S.
        • Budescu D.V.
        Encoding subjective probabilities: a psychological and psychometric review.
        Manage Sci. 1983; 29: 151-173
        • Murphy A.H.
        • Winkler R.L.
        Reliability of subjective probability forecasts of precipitation and temperature.
        Appl Statist. 1977; 26: 41-47
        • Bergus G.R.
        • Chapman G.B.
        • Gjerde C.
        • Elstein A.S.
        Clinical reasoning about new symptoms despite preexisting disease: sources of error and order effects.
        Fam Med. 1995; 27: 314-320
        • Evans J.S.
        • Brooks P.
        • Pollard P.
        Prior beliefs and statistical inference.
        Brit J Psychiatry. 1985; 76: 469-477
        • Evans J.S.
        • Handley S.J.
        • Over D.E.
        • Perham N.
        Background beliefs in Bayesian inference.
        Mem Cognit. 2002; 30: 179-190
        • White I.R.
        • Pocock S.J.
        • Wang D.
        Eliciting and using expert opinions about influence of patient characteristics on treatment effects: a Bayesian analysis of the CHARM trials.
        Stat Med. 2005; 24: 3805-3821
        • Winkler R.L.
        The assessment of prior distributions in Bayesian analysis.
        J Am Stat Assoc. 1967; 62: 776-800
        • Van der Fels-Klerx I.H.
        • Goossens L.H.
        • Saatkamp H.W.
        • Horst S.H.
        Elicitation of quantitative data from a heterogeneous expert panel: formal process and application in animal health.
        Risk Anal. 2002; 22: 67-81
        • Li Y.
        • Krantz D.H.
        Experimental tests of subjective Bayesian methods.
        Psychol Rec. 2005; 55: 251-277
        • Chaloner K.
        Elicitation of prior distributions.
        in: Berry D.A. Stangl D.K. Bayesian biostatistics. Marcel Dekker Inc, New York1996: 141-156
        • O'Hagan A.
        Eliciting expert beliefs in substantial practical application.
        Statistician. 1998; 47: 21-35
        • Winkler R.L.
        The quantification of judgement: some methodological suggestions.
        J Am Stat Assoc. 1967; 62: 1105-1120
        • Normand S.L.
        • Frank R.G.
        • McGuire T.G.
        Using elicitation techniques to estimate the value of ambulatory treatments for major depression.
        Med Decis Making. 2002; 22: 245-261
        • Johnson S.R.
        • Granton J.T.
        • Mehta S.
        Thrombotic arteriopathy and anticoagulation in pulmonary hypertension.
        Chest. 2006; 130: 545-552
        • Koh E.T.
        • Lee P.
        • Gladman D.D.
        • Abu-Shakra M.
        Pulmonary hypertension in systemic sclerosis: an analysis of 17 patients.
        Br J Rheumatol. 1996; 35: 989-993
        • D'Alonzo G.E.
        • Barst R.J.
        • Ayres S.M.
        • Bergofsky E.H.
        • Brundage B.H.
        • Detre K.M.
        • et al.
        Survival in patients with primary pulmonary hypertension. Results from a national prospective registry.
        Ann Intern Med. 1991; 115: 343-349
        • Johnson S.R.
        • Mehta S.
        • Granton J.T.
        Anticoagulation in pulmonary arterial hypertension: a qualitative systematic review.
        Eur Respir J. 2006; 28: 999-1004
        • Weijer C.
        • Shapiro S.H.
        • Cranley G.K.
        For and against: clinical equipoise and not the uncertainty principle is the moral underpinning of the randomised controlled trial.
        BMJ. 2000; 321: 756-758
        • Trochim W.M.
        The research methods knowledge base.
        Second edition. Atomic Dog Publishing, Cincinnati, Ohio2006
        • Dillman D.S.
        • Smyth J.D.
        • Christian L.M.
        Internet, mail, and mixed-mode surveys. The tailored design method.
        3rd edition. John Wiley & Sons, Inc, Hoboken, NJ2009
        • Streiner D.L.
        • Norman G.R.
        Health measurement scales. A practical guide to their development and use.
        3rd edition. Oxford University Press, New York2003
        • Cicchetti D.V.
        • Allison T.
        A new procedure for assessing reliability of scoring EEG sleep recordings.
        Am J EEG Technol. 1971; 11: 101-109
        • Walter S.D.
        • Eliasziw M.
        • Donner A.
        Sample size and optimal designs for reliability studies.
        Stat Med. 1998; 17: 101-110
        • Shrout P.E.
        • Fleiss J.L.
        Intraclass correlations: uses in assessing rater reliability.
        Psychol Bull. 1979; 86: 420-428
        • O'Hagan A.
        • Buck C.E.
        • Daneshkhah A.
        • Eiser J.R.
        • Garthwaite P.H.
        • Jenkinson D.J.
        • et al.
        Uncertain judgements. Eliciting experts' probabilities.
        John Wiley & Sons Ltd, Chichester, UK2006
        • Johnson S.R.
        • Hawker G.A.
        • Davis A.M.
        The health assessment questionnaire disability index and scleroderma health assessment questionnaire in scleroderma trials: an evaluation of their measurement properties.
        Arthritis Rheum. 2005; 53: 256-262
        • McHorney C.A.
        • Tarlov A.R.
        Individual-patient monitoring in clinical practice: are available health status surveys adequate?.
        Qual Life Res. 1995; 4: 293-307
        • Landis J.
        • Koch G.
        The measurement of observer agreement for categorical data.
        Biometrics. 1977; 33: 159-174
        • Clemen R.T.
        • Wolmark N.
        Combining probability distributions from experts in risk analysis.
        Risk Anal. 1999; 19: 187-203
        • Nunnally J.C.
        Psychometric theory.
        McGraw-Hill, New York1978
        • Liang M.H.
        Longitudinal construct validity: establishment of clinical meaning in patient evaluative instruments.
        Med Care. 2000; 38: II84-II90
        • Frank H.
        • Mlczoch J.
        • Huber K.
        • Schuster E.
        • Gurtner H.P.
        • Kneussl M.
        The effect of anticoagulant therapy in primary and anorectic drug-induced pulmonary hypertension.
        Chest. 1997; 112: 714-721
        • Rich S.
        • Kaufmann E.
        • Levy P.S.
        The effect of high doses of calcium-channel blockers on survival in primary pulmonary hypertension.
        N Engl J Med. 1992; 327: 76-81
        • Fuster V.
        • Steele P.M.
        • Edwards W.D.
        • Gersh B.J.
        • McGoon M.D.
        • Frye R.L.
        Primary pulmonary hypertension: natural history and the importance of thrombosis.
        Circulation. 1984; 70: 580-587
        • Ogata M.
        • Ohe M.
        • Shirato K.
        • Takishima T.
        Effects of a combination therapy of anticoagulant and vasodilator on the long-term prognosis of primary pulmonary hypertension.
        Jpn Circ J. 1993; 57: 63-69