Triage-weighted kappa: toward a more precise reflection of the reliability of emergency department triage systems - Reply
Article Outline
With this letter we respond to the comments of Twomey on our recent publication “Adjusting weighted kappa for severity of mistriage decreases reported reliability of emergency department triage systems: a comparative study”.
Triage-weighted kappa is a weighting scheme that can be applied in triage reliability studies, which measure the reliability by means of consistency. With this weighting scheme, we have introduced a new approach to triage reliability studies, that is, to critically view the height of the weights in relation to the distance from perfect agreement and their consequences for patient safety in the emergency department (ED) [1]. This approach should be applied to other reliability studies. We encourage researchers to develop an own weighting scheme that fits with the measurement scale instead of using existing weighting schemes to compare their results with the results of former studies.
In response to our article, Twomey encouraged the development of triage-weighted kappa but questioned whether the algorithms for over- and undertriage appropriately account for differences among triage systems and differences among ED settings. In addition, it is suggested to further develop triage-weighted kappa by taking into account such qualitative and quantitative factors. The suggestion to design weighting schemes that account for differences in mistriage among different triage systems is certainly an option and triage-weighted kappa can be used as a basis for this extension. However, a criterion for these weighting schemes should be that they must reflect the content of the triage system, as the purpose of the weighting scheme is to measure the reliability of a triage system. Furthermore, to make a quantitative distinction between 30 (Australasian Triage Scale category 3) and 60 (Manchester Triage System category 3) minutes in the weighting scheme, it should be known how this difference in waiting time influences the outcome.
To account for factors such as ED setting is, from a methodological point of view, not recommended because such weighting schemes do not reflect the reliability of a triage system. Twomey proposed that because of fewer available resources in one ED compared with another, the weights for overtriage should be accounted for more strictly in EDs with scarce resources compared with other EDs. In this case, one is interested in the quality of triage instead of the reliability of a triage system. A triage system should be reliable irrespective of ED setting. Although the latter is an important aspect of patient safety in the ED, we would not recommend accounting for ED setting when the purpose of the study is to measure the reliability of triage systems.
In conclusion, triage-weighted kappa is a first step toward a more precise reflection of the reliability of triage systems in relation to what is accepted in clinical practice. The next step should be to base the weightings on the impact of mistriage on outcome, which can be done separately for the existing triage systems. As triage systems ought to be reliable irrespective of ED setting, we think that weighting schemes should not account for factors, such as ED settings.
References
PII: S0895-4356(09)00357-6
doi:10.1016/j.jclinepi.2009.11.004
© 2010 Elsevier Inc. All rights reserved.
