Advertisement

Citation analysis is also useful to assess the eligibility of biomedical research works for inclusion in living systematic reviews

Published:November 11, 2017DOI:https://doi.org/10.1016/j.jclinepi.2017.11.002
      I have read with much interest the JCE series advocating the use of human efforts and machine automation to create and update living systematic reviews (LSRs) [
      • Thomas J.
      • Noel-Storr A.
      • Marshall I.
      • Wallace B.
      • McDonald S.
      • Mavergames C.
      • et al.
      Living systematic reviews: 2. Combining human and machine effort.
      ]. I recognize that the series provides important information on how biomedical research works are verified as eligible for inclusion in LSRs using semantic classification and crowdsourcing techniques [
      • Thomas J.
      • Noel-Storr A.
      • Marshall I.
      • Wallace B.
      • McDonald S.
      • Mavergames C.
      • et al.
      Living systematic reviews: 2. Combining human and machine effort.
      ]. However, this paper has not dealt with another technique that has been recently shown to be useful (when jointly used with semantic classification and crowdsourcing techniques) in assessing the eligibility of papers for inclusion in LSRs: This important technique is citation analysis [
      • Chen T.T.
      The development and empirical study of a literature review aiding system.
      ,
      • Choong M.K.
      • Galgani F.
      • Dunn A.G.
      • Tsafnat G.
      Automatic evidence retrieval for systematic reviews.
      ,
      • Octaviano F.R.
      • Felizardo K.R.
      • Maldonado J.C.
      • Fabbri S.C.
      Semi-automatic selection of primary studies in systematic literature reviews: is it reasonable?.
      ,
      • Octaviano F.
      • Silva C.
      • Fabbri S.
      Using the SCAS strategy to perform the initial selection of studies in systematic reviews: an experimental study.
      ,
      • Wohlin C.
      Guidelines for snowballing in systematic literature studies and a replication in software engineering.
      ].
      It has been shown that evidence identified by traditional methods that are neither cited by nor citing other initial search results (not included in the citation network of the initially selected papers) do not relate to the topic of the LSR and had been included in the initial set of search results by error. Such papers can be easily eliminated without having to semantically analyze them [
      • Chen T.T.
      The development and empirical study of a literature review aiding system.
      ,
      • Octaviano F.R.
      • Felizardo K.R.
      • Maldonado J.C.
      • Fabbri S.C.
      Semi-automatic selection of primary studies in systematic literature reviews: is it reasonable?.
      ,
      • Octaviano F.
      • Silva C.
      • Fabbri S.
      Using the SCAS strategy to perform the initial selection of studies in systematic reviews: an experimental study.
      ].
      Furthermore, citation analysis based on two easy-to-do snowballing techniques can be useful to add the papers that are eligible for the LSR but that were missed by semantic classification and crowdsourcing techniques
      Less than 1% of the total number of papers that should be considered for the LSR according to [
      • Thomas J.
      • Noel-Storr A.
      • Marshall I.
      • Wallace B.
      • McDonald S.
      • Mavergames C.
      • et al.
      Living systematic reviews: 2. Combining human and machine effort.
      ]. This fact can be explained by the nonindexation of these few papers in the analyzed reference database or by the use of a language other than English when writing them.
      1Less than 1% of the total number of papers that should be considered for the LSR according to [
      • Thomas J.
      • Noel-Storr A.
      • Marshall I.
      • Wallace B.
      • McDonald S.
      • Mavergames C.
      • et al.
      Living systematic reviews: 2. Combining human and machine effort.
      ]. This fact can be explained by the nonindexation of these few papers in the analyzed reference database or by the use of a language other than English when writing them.
      .
      Backward snowballing consists of analyzing the references of the initial search results and of all the papers that were cited by the initial search results and then include the papers that cite or are cited by a significant number of initial search results in the LSR. Forward snowballing consists of analyzing the references of all the papers indexed in the database and include in the LSR the papers that cite a significant number of initial search results [
      • Choong M.K.
      • Galgani F.
      • Dunn A.G.
      • Tsafnat G.
      Automatic evidence retrieval for systematic reviews.
      ,
      • Wohlin C.
      Guidelines for snowballing in systematic literature studies and a replication in software engineering.
      ].
      Moreover, papers that are co-cited with the initial search results in the analyzed database (and that are not included in the initial set of considered papers for the LSR) are very likely to be related to the topic of the LSR [
      • Chen T.T.
      The development and empirical study of a literature review aiding system.
      ,
      • Small H.
      Co-citation in the scientific literature: a new measure of the relationship between two documents.
      ]. These papers are easy to find and to include in the LSR using the analysis of the references of all the papers that cite initial search results.
      For these reasons, citation analysis should be used for creating and updating LSRs.

      References

        • Thomas J.
        • Noel-Storr A.
        • Marshall I.
        • Wallace B.
        • McDonald S.
        • Mavergames C.
        • et al.
        Living systematic reviews: 2. Combining human and machine effort.
        J Clin Epidemiol. 2017; 91: 31-37
        • Chen T.T.
        The development and empirical study of a literature review aiding system.
        Scientometrics. 2012; 92: 105-116
        • Choong M.K.
        • Galgani F.
        • Dunn A.G.
        • Tsafnat G.
        Automatic evidence retrieval for systematic reviews.
        J Med Internet Res. 2014; 16: e223
        • Octaviano F.R.
        • Felizardo K.R.
        • Maldonado J.C.
        • Fabbri S.C.
        Semi-automatic selection of primary studies in systematic literature reviews: is it reasonable?.
        Empirical Softw Eng. 2015; 20: 1898-1917
        • Octaviano F.
        • Silva C.
        • Fabbri S.
        Using the SCAS strategy to perform the initial selection of studies in systematic reviews: an experimental study.
        (Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering)2016: 25 (ACM. Available at https://dl.acm.org/citation.cfm?doid=2915970.2916000)
        • Wohlin C.
        Guidelines for snowballing in systematic literature studies and a replication in software engineering.
        (Proceedings of the 18th international conference on evaluation and assessment in software engineering)2014: 38 (ACM. Available at https://dl.acm.org/citation.cfm?id=2601268)
        • Small H.
        Co-citation in the scientific literature: a new measure of the relationship between two documents.
        J Assoc Inf Sci Technology. 1973; 24: 265-269

      Linked Article

      • Living systematic reviews: 2. Combining human and machine effort
        Journal of Clinical EpidemiologyVol. 91
        • Preview
          New approaches to evidence synthesis, which use human effort and machine automation in mutually reinforcing ways, can enhance the feasibility and sustainability of living systematic reviews. Human effort is a scarce and valuable resource, required when automation is impossible or undesirable, and includes contributions from online communities (“crowds”) as well as more conventional contributions from review authors and information specialists. Automation can assist with some systematic review tasks, including searching, eligibility assessment, identification and retrieval of full-text reports, extraction of data, and risk of bias assessment.
        • Full-Text
        • PDF
        Open Access
      • Citation analysis may well have a role to play in study identification, but more evaluation and system development are required
        Journal of Clinical EpidemiologyVol. 97
        • Preview
          I would like to thank Houcemeddine Turki [1] for his interest in the paper that I and my colleagues recently published [2], and for drawing our attention to the potential utility of citation networks for identifying eligible studies for inclusion in systematic reviews.
        • Full-Text
        • PDF