Advertisement

Citation analysis may well have a role to play in study identification, but more evaluation and system development are required

Published:November 11, 2017DOI:https://doi.org/10.1016/j.jclinepi.2017.11.001
      I would like to thank Houcemeddine Turki [
      • Turki H.
      Citation analysis is also useful to assess the eligibility of biomedical research works for inclusion in living systematic reviews.
      ] for his interest in the paper that I and my colleagues recently published [
      • Thomas J.
      • Noel-Storr A.
      • Marshall I.
      • Wallace B.
      • McDonald S.
      • Mavergames C.
      • et al.
      Living systematic reviews: 2. Combining human and machine effort.
      ], and for drawing our attention to the potential utility of citation networks for identifying eligible studies for inclusion in systematic reviews.
      I agree that citation analysis may well have a role to play in the future along the lines suggested but feel that further research and development is needed. Most significantly, no study has evaluated these approaches at scale (eg, by looking across all systematic reviews in the Cochrane Library), so we do not know yet how generalizable these techniques are. One of the reasons for this is that it has been quite difficult to generate citation networks automatically. This factor has not only limited the evaluation work that has hitherto been possible but also points to a weakness in our information architecture more broadly: that, even if citation networks might be useful, we cannot generate them in a reliable and automated way. (This last limitation may be changing with the emergence of the new Microsoft Academic Graph API, which is designed to support the creation of citation networks automatically.)
      When using citation data, the reviewer needs to be always cautious that they are not simply reinforcing the effect of publication bias. We know that researchers tend to operate as communities, publishing with one another, attending the same conferences, and citing each other's papers. An over-reliance on snowballing techniques can miss important ‘clusters’ of studies in situations where they are not connected to the papers already identified. I agree with Mr Turki, that citation data might be considered useful when combined with semantic classification, rather than on its own, and looking forward to see more evaluative studies in this area, using services which enable citation networks to be created quickly, reliably, and at scale.

      Reference

        • Turki H.
        Citation analysis is also useful to assess the eligibility of biomedical research works for inclusion in living systematic reviews.
        J Clin Epidemiol. 2018; 97: 135-136
        • Thomas J.
        • Noel-Storr A.
        • Marshall I.
        • Wallace B.
        • McDonald S.
        • Mavergames C.
        • et al.
        Living systematic reviews: 2. Combining human and machine effort.
        J Clin Epidemiol. 2017; 91: 31-37

      Linked Article

      • Citation analysis is also useful to assess the eligibility of biomedical research works for inclusion in living systematic reviews
        Journal of Clinical EpidemiologyVol. 97
        • Preview
          I have read with much interest the JCE series advocating the use of human efforts and machine automation to create and update living systematic reviews (LSRs) [1]. I recognize that the series provides important information on how biomedical research works are verified as eligible for inclusion in LSRs using semantic classification and crowdsourcing techniques [1]. However, this paper has not dealt with another technique that has been recently shown to be useful (when jointly used with semantic classification and crowdsourcing techniques) in assessing the eligibility of papers for inclusion in LSRs: This important technique is citation analysis [2–6].
        • Full-Text
        • PDF