Advertisement

A validation study revealed differences in design and performance of search filters for qualitative research in PsycINFO and CINAHL

Open AccessPublished:September 25, 2020DOI:https://doi.org/10.1016/j.jclinepi.2020.09.031

      Abstract

      Objectives

      Search filters can support qualitative evidence of information retrieval. Various search filters are available for the bibliographic databases PsycINFO and CINAHL. To date, no comparative overview of validation results of search filters verified with an independent gold standard exists.

      Study Design and Setting

      Identified search filters for PsycINFO and CINAHL were tested for plausibility. Gold standards were generated according to the relative recall approach using references included in an overview of systematic reviews of qualitative studies. All included references were collected and checked for indexing in PsycINFO and CINAHL. Validation tests for each search filter were conducted in both databases to determine whether the references of the gold standards could be retrieved or not.

      Results

      Twelve search filters for PsycINFO and fifteen for CINAHL were validated. The complexity and design of these search filters vary, as well as the validation results for the databases. When locating primary studies of qualitative research, the best sensitivity and precision ratio (among filters with a sensitivity of >80%) was achieved with a filter by McKibbon et al. for PsycINFO and a filter by Wilczynski et al. for CINAHL.

      Conclusion

      Project-specific requirements and resources influence the choice of a specific search filter for PsycINFO and CINAHL.

      Keywords

      What is new?

        Key findings

      • Search filters for qualitative research in PsycINFO and CINAHL differ greatly in design and performance.
      • It is not possible to suggest a single search filter to identify qualitative research in these databases.

        What this adds to what is known?

      • First comparative overview of validation measurements of search filters for PsycINFO and CINAHL using a gold standard irrespective of a medical or health-related topic and with literature published in several years.

        What is the implication, what should change now?

      • Project-specific requirements and resources drive the decision for or against a search filter, and for which one.
      • To increase the likelihood of identifying all relevant articles, several bibliographic databases need to be searched, especially in the field of qualitative research.

      1. Introduction

      The implication of qualitative research in evidence-based health care has increased. Patients’ preferences and experiences need to be considered in high-quality systematic reviews and clinical practice guidelines [
      • Booth A.
      Searching for qualitative research for inclusion in systematic reviews: a structured methodological review.
      ,
      • Noyes J.
      • Booth A.
      • Cargo M.
      • Flemming K.
      • Harden A.
      • Harris J.
      • et al.
      Qualitative evidence. Chapter 21.
      ,
      AGREE Next Steps Consortium
      Appraisal of guidelines for research & evaluation II. AGREE II Instrument. AGREE Research Trust.
      ].
      Various research methods and approaches are used in qualitative research. However, most research papers are less structured and not standardized in their vocabulary. That makes relevant qualitative evidence information retrieval difficult and complex [
      • Booth A.
      Searching for qualitative research for inclusion in systematic reviews: a structured methodological review.
      ,
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ,
      • Evans D.
      Database searches for qualitative research.
      ]. Lefebvre et al. [
      • Lefebvre C.
      • Glanville J.
      • Beale S.
      • Boachie C.
      • Duffy S.
      • Fraser C.
      • et al.
      Assessing the performance of methodological search filters to improve the efficiency of evidence information retrieval: five literature reviews and a qualitative study.
      ] asked information professionals to name “Areas where filters are needed/existing filters need to be improved” (p. 70). Interviewees mentioned, among other things, qualitative research filters.
      In recent years, search filters for identifying qualitative research in PsycINFO and CINAHL have been published but not adequately validated with an independent set of references (gold standard, GS).
      The first aim of this paper is to provide a comprehensive overview of published filters for locating qualitative research in bibliographic databases PsycINFO and CINAHL. Second, this study compares the validation performances of search filters. Validations were conducted using a newly generated GS irrespective of a medical or health-related topic and with references published over a longer period of time. Given these results, it is possible to determine the search filter with the best sensitivity, best precision ratio, and/or best balance of sensitivity and precision.
      The aims and methods of this paper are in accordance with a previous work by Wagner et al. [
      • Wagner M.
      • Rosumeck S.
      • Küffmeier C.
      • Döring K.
      • Euler U.
      A validation study revealed differences in design and performance of MEDLINE search filters for qualitative research.
      ] focusing on MEDLINE.

      2. Methods

      Similar to the aforementioned publication [
      • Wagner M.
      • Rosumeck S.
      • Küffmeier C.
      • Döring K.
      • Euler U.
      A validation study revealed differences in design and performance of MEDLINE search filters for qualitative research.
      ], a four-step approach was used: 1. identification of search filters to locate qualitative research in the bibliographic databases PsycINFO and CINAHL, 2. verification and plausibility checks of identified search filters, 3. generation of a GS irrespective of a medical or health-related topic, 4. validation of identified search filters against the newly created GS in PsycINFO and CINAHL.

      2.1 Identification of search filters

      For identifying published search filters for qualitative research a comprehensive literature search was carried out in PubMed. Additionally, other well-known sources like the Cochrane Handbook for Systematic Reviews of Interventions [
      • Higgins J.P.T.
      • Green S.
      Qualitative evidence: searching for qualitative evidence. Chapter 21, section 21.7.
      ], and websites, such as those of the InterTASC Information Specialists’ Subgroup (ISSG) [

      Glanville J, Lefebvre C, Wright K, editors. Filters to identify qualitative research. Update date: 4 February 2020. York, GB: ISSG [InterTASC Information Specialists' Sub-Group]. Available at https://sites.google.com/a/york.ac.uk/issg-search-filters-resource/filters-to-identify-qualitative-research. Accessed April 9, 2020.

      ] and the McMaster University’s Department of Health Information Research Unit [
      HiRU [Health Information Research Unit]
      Health information research unit. Our mission... Last modified February 9, 2016. Hamilton, CA-ON: HiRU.
      ], were checked. Published search filters with complete search strategy, irrespective of their publication date, were eligible.

      2.2 Verification and plausibility checks of search filters

      All filters created to run in the bibliographic databases PsycINFO or CINAHL, irrespective of the host platform, were taken into account. Newly developed filters, as well as adaptations of already published search filters, were included and considered for validation.
      Search strategies must be plausible and reproducible in terms of database syntax, e.g., to just list a number of keywords and state that these were searched with truncation is insufficient. Hence, all identified search filters were checked. In case of considerable syntax errors or content inconsistencies (e.g., including search terms that restrict the search to a thematic topic), the respective filter was excluded from the validation process. For reasons of feasibility, developers of search filters and authors were not contacted.

      2.3 Generation of the gold standard

      Different methods to generate a GS were reported, e.g., topic-related hand searching of specific journals [
      • McKibbon K.A.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
      ,
      • Wilczynski N.L.
      • Marks S.
      • Haynes R.B.
      Search strategies for identifying qualitative studies in CINAHL.
      ,
      • Wong S.S.-L.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.
      ] or combining all studies included in various systematic reviews of a topic (relative recall approach) [
      • Durão S.
      • Kredo T.
      • Volmink J.
      Validation of a search strategy to identify nutrition trials in PubMed using the relative recall method.
      ,
      • Sampson M.
      • Zhang L.
      • Morrison A.
      • Barrowman N.J.
      • Clifford T.J.
      • Platt R.W.
      • et al.
      An alternative to the hand searching gold standard: validating methodological search filters using relative recall.
      ,
      • Waffenschmidt S.
      • Hermanns T.
      • Gerber-Grote A.
      • Mostardt S.
      No suitable precise or optimized epidemiologic search filters were available for bibliographic databases.
      ]. In comparison to generating a GS with relevant articles published over a long period of time, the relative recall approach is more time saving and was already successfully applied in a previous project [
      • Wagner M.
      • Rosumeck S.
      • Küffmeier C.
      • Döring K.
      • Euler U.
      A validation study revealed differences in design and performance of MEDLINE search filters for qualitative research.
      ]. Thus, this approach was used to generate the GS for PsycINFO and CINAHL.
      Dalton et al. [
      • Dalton J.
      • Booth A.
      • Noyes J.
      • Sowden A.J.
      Potential value of systematic reviews of qualitative evidence in informing user-centered health and social care: findings from a descriptive overview.
      ] aimed at providing an overview of systematic reviews of qualitative research indexed in the Database of Abstracts of Reviews of Effects (DARE) from 2009 to 2014 regardless of the medical or health-related topic. Included systematic reviews were used as a starting point for the new GS. This can be considered as comprehensive, as DARE itself incorporates hits from MEDLINE, Embase, PsycINFO, PubMed and CINAHL. For more details on systematic literature searches used in the systematic reviews, see Supplemental File.
      First, all 145 systematic reviews included by Dalton et al. [
      • Dalton J.
      • Booth A.
      • Noyes J.
      • Sowden A.J.
      Potential value of systematic reviews of qualitative evidence in informing user-centered health and social care: findings from a descriptive overview.
      ] were obtained in full-text. Second, all primary studies (journal articles or dissertations) included in these systematic reviews were obtained and managed in EndNote. We assumed that all included articles addressed qualitative research, so we did not investigate whether each study applied qualitative methods. In a final step, duplicates were removed, and only articles indexed in PsycINFO or CINAHL were added to the respective database-specific GS.

      2.4 Validation of search filters

      The validation of search filters followed a method described elsewhere [
      • Wagner M.
      • Rosumeck S.
      • Küffmeier C.
      • Döring K.
      • Euler U.
      A validation study revealed differences in design and performance of MEDLINE search filters for qualitative research.
      ]. For each reference of the GS, it was determined whether it could be located with the particular search filter in PsycINFO or CINAHL. Performance measurements, such as sensitivity, precision, and number needed to read (NNR), were determined based on the number of hits revealed (Table 1). For overcoming the disadvantage that the relative recall approach does not allow for an estimation of the true precision, a precision ratio was calculated based on Waffenschmidt et al. [
      • Waffenschmidt S.
      • Hermanns T.
      • Gerber-Grote A.
      • Mostardt S.
      No suitable precise or optimized epidemiologic search filters were available for bibliographic databases.
      ]. Accordingly, the precision ratio was set to 1 for the search filter with the highest sensitivity. Precision ratios for the other filters were then calculated by the precision of each filter divided by the precision of the most sensitive filter. This allows a better interpretation of validation results and enhances the comparability of the performance of the investigated filters. For example, irrespective of specificity, a precision ratio of 2 indicates that a filter is twice as precise and would require half the number of records to be screened to find one qualitative article, compared to the search filter with the highest sensitivity.
      Table 1Calculations
      Definitionsa - relevant articles of the gold standard retrieved with search filter
      b - irrelevant articles
      Could be articles relating to qualitative research and relevant but are not part of the gold standard.
      retrieved with search filter (hits in the database for each search filter minus references of the gold standard)
      c - relevant articles of the gold standard not retrieved with the search filter
      Sensitivity (%)a/(a+c)∗100
      Precision (%)a/(a+b)∗100
      NNR(a+b)/a
      Precision ratePrecision/precision of the search filter with the highest sensitivity
      Abbreviation: NNR, number needed to read.
      # Could be articles relating to qualitative research and relevant but are not part of the gold standard.

      3. Results

      3.1 Identification of search filters

      A systematic search in PubMed for literature on qualitative research search filters (Box 1) revealed 377 hits. One person screened all titles and abstracts. Then, the full-texts of eight articles were obtained and screened by one person [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ,
      • Lefebvre C.
      • Glanville J.
      • Beale S.
      • Boachie C.
      • Duffy S.
      • Fraser C.
      • et al.
      Assessing the performance of methodological search filters to improve the efficiency of evidence information retrieval: five literature reviews and a qualitative study.
      ,
      • McKibbon K.A.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
      ,
      • Wilczynski N.L.
      • Marks S.
      • Haynes R.B.
      Search strategies for identifying qualitative studies in CINAHL.
      ,
      • DeJean D.
      • Giacomini M.
      • Simeonov D.
      • Smith A.
      Finding qualitative research evidence for health technology assessment.
      ,
      • Flemming K.
      • Briggs M.
      Electronic searching to locate qualitative research: evaluation of three strategies.
      ,
      • Gorecki C.A.
      • Brown J.M.
      • Briggs M.
      • Nixon J.
      Evaluation of five search strategies in retrieving qualitative patient-reported electronic data on the impact of pressure ulcers on quality of life.
      ,
      • Shaw R.L.
      • Booth A.
      • Sutton A.J.
      • Miller T.
      • Smith J.A.
      • Young B.
      • et al.
      Finding qualitative research: an evaluation of search strategies.
      ]. Five articles presented search strategies for locating qualitative research in PsycINFO [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ,
      • McKibbon K.A.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
      ] and CINAHL [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ,
      • Wilczynski N.L.
      • Marks S.
      • Haynes R.B.
      Search strategies for identifying qualitative studies in CINAHL.
      ,
      • DeJean D.
      • Giacomini M.
      • Simeonov D.
      • Smith A.
      Finding qualitative research evidence for health technology assessment.
      ,
      • Shaw R.L.
      • Booth A.
      • Sutton A.J.
      • Miller T.
      • Smith J.A.
      • Young B.
      • et al.
      Finding qualitative research: an evaluation of search strategies.
      ].
      Search strategy (PubMed), search date: December, 10th 2019
      Tabled 1
      (Information Storage and Retrieval[Mesh:NoExp] OR Databases as Topic[Mesh:NoExp] OR Databases, Bibliographic[Mesh:NoExp] OR PsycINFO[tiab] OR PsychINFO[tiab] OR PsyINFO[tiab] OR CINAHL[tiab] OR CINHAL[tiab]) AND (Qualitative Research[Mesh] OR qualitative[tiab]) AND (“search filter”[tiab] OR “search filters”[tiab] OR “search strategy”[tiab] OR “search strategies”[tiab] OR hedge[tiab] OR hedges[tiab])
      Scanning the chapter on searching for qualitative evidence of the Cochrane Handbook of Systematic Reviews of Intervention [
      • Higgins J.P.T.
      • Green S.
      Qualitative evidence: searching for qualitative evidence. Chapter 21, section 21.7.
      ], as well as the websites of the ISSG [

      Glanville J, Lefebvre C, Wright K, editors. Filters to identify qualitative research. Update date: 4 February 2020. York, GB: ISSG [InterTASC Information Specialists' Sub-Group]. Available at https://sites.google.com/a/york.ac.uk/issg-search-filters-resource/filters-to-identify-qualitative-research. Accessed April 9, 2020.

      ] and of the McMaster University [
      HiRU [Health Information Research Unit]
      Search strategies for PsycINFO in Ovid syntax. Last modified February 9, 2016. Hamilton, CA-ON: HiRU.
      ], yielded one additional search strategy for locating qualitative research in PsycINFO [
      UTHealth [University of Texas Health Science Center at Houston]
      Search filters for various databases: Ovid PsycINFO. Last Updated: Sep 26, 2019. Houston, US-TX: UTHealth.
      ] and CINAHL [
      • Nesbit K.
      Evidence-based filters for Ovid CINAHL. Date Last Updated 2012-09-26. University of Rochester.
      ], respectively.

      3.1.1 PsycINFO

      In total, 14 different strategies for locating qualitative research in PsycINFO were identified [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ,
      • McKibbon K.A.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
      ,
      UTHealth [University of Texas Health Science Center at Houston]
      Search filters for various databases: Ovid PsycINFO. Last Updated: Sep 26, 2019. Houston, US-TX: UTHealth.
      ], including two index term searches.
      McKibbon et al. [
      • McKibbon K.A.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
      ] (McMaster University Hedges Team) developed and evaluated six different filters, including single-term and multiple-term strategies. Different variations of the keywords interview, qualitative, themes, and experiences were tested. Rogers et al. [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ] evaluated five filters consisting of subject headings, simple free-text terms, complex free-text terms, and two broad-based strategies. All these strategies are adaptations of search filters published before. Another nonvalidated search filter, which incorporates the index term “Qualitative Research”, was identified on the website of the University of Texas – School of Public Health (UTHealth) [
      UTHealth [University of Texas Health Science Center at Houston]
      Search filters for various databases: Ovid PsycINFO. Last Updated: Sep 26, 2019. Houston, US-TX: UTHealth.
      ]. The index term was introduced in 2003 and changed to “Qualitative Methods” in 2019 [
      APA [American Psychological Association]
      Thesaurus of psychological index terms 2019 Update.
      ]. Exploded and unexploded searches of this term were considered.

      3.1.2 CINAHL

      In total, 19 different strategies for locating qualitative research in CINAHL were identified [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ,
      • Wilczynski N.L.
      • Marks S.
      • Haynes R.B.
      Search strategies for identifying qualitative studies in CINAHL.
      ,
      • DeJean D.
      • Giacomini M.
      • Simeonov D.
      • Smith A.
      Finding qualitative research evidence for health technology assessment.
      ,
      • Shaw R.L.
      • Booth A.
      • Sutton A.J.
      • Miller T.
      • Smith J.A.
      • Young B.
      • et al.
      Finding qualitative research: an evaluation of search strategies.
      ,
      • Nesbit K.
      Evidence-based filters for Ovid CINAHL. Date Last Updated 2012-09-26. University of Rochester.
      ], including two subject heading searches.
      Shaw et al. [
      • Shaw R.L.
      • Booth A.
      • Sutton A.J.
      • Miller T.
      • Smith J.A.
      • Young B.
      • et al.
      Finding qualitative research: an evaluation of search strategies.
      ] evaluated a thesaurus term, a free-text term, and a broad-based terms strategy. Rogers et al. [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ] compared five different strategies, and four of them were eligible for validation: subject headings, simple free-text terms, broad-based terms, and a complex free-text strategy. The Hedges Team of McMaster University published eight search filters for CINAHL via Ovid (different search terms and combinations) [
      • Wilczynski N.L.
      • Marks S.
      • Haynes R.B.
      Search strategies for identifying qualitative studies in CINAHL.
      ]. Several years ago, the Edward G. Miner Library at the University of Rochester Medical Center published various search filters for CINAHL via Ovid. One of them focused on locating qualitative research [
      • Nesbit K.
      Evidence-based filters for Ovid CINAHL. Date Last Updated 2012-09-26. University of Rochester.
      ]. Although it is marked as “to be used for historical purposes only”, it is cited on the ISSG website. Therefore, we decided to consider this comprehensive search filter for validation. In addition, a hybrid filter by DeJean et al. [
      • DeJean D.
      • Giacomini M.
      • Simeonov D.
      • Smith A.
      Finding qualitative research evidence for health technology assessment.
      ] was identified, which just combined existing search filters. Of the controlled vocabulary in CINAHL, the subject heading “Qualitative Studies” was considered as an exploded and unexploded search string.

      3.2 Verification and plausibility checks of search filters for qualitative research

      3.2.1 PsycINFO

      Two (UTHealth [
      UTHealth [University of Texas Health Science Center at Houston]
      Search filters for various databases: Ovid PsycINFO. Last Updated: Sep 26, 2019. Houston, US-TX: UTHealth.
      ], Rogerse [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ]) of the 14 identified search filters failed the test for plausibility and replicability and were excluded from validation (Table 2).
      Table 2Search strategies of identified search filters for PsycINFO via Ovid
      Search filterLabeled by the filter author(s)Search strategy for PsycINFO (via Ovid)
      Please refer to database guide for more details on search syntax.
      Index Terma-Qualitative Methods/
      Index Termb-exp Qualitative Methods/
      McKibbona [
      • McKibbon K.A.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
      ]
      Single term – best sensitivity and single term - best optimization of sensitivity & specificityinterview.tw.
      McKibbonb [
      • McKibbon K.A.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
      ]
      Single term – best specificityinterviews.tw.
      McKibbonc [
      • McKibbon K.A.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
      ]
      Combination of terms – best sensitivityexperience∗.mp. OR interview∗.tw. OR qualitative∗.tw.
      McKibbond [
      • McKibbon K.A.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
      ]
      Combination of terms – best specificityqualitative∗.tw. OR themes.tw.
      McKibbone [
      • McKibbon K.A.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
      ]
      Combination of terms – small drop in specificity with a substantive gain in sensitivityinterviews.tw. OR qualitative∗.tw.
      McKibbonf [
      • McKibbon K.A.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
      ]
      Combination of terms – best optimization of sensitivity & specificityexperiences.tw. OR interview∗.tw. OR qualitative.tw.
      Rogersa [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ]
      Subject headings termsQualitative Research/ OR exp Questionnaires/ OR Interviewing/ OR Attitudes/ OR Ethnology/ OR Phenomenology/ OR Observation Methods/ OR Discourse Analysis/ OR Content Analysis/
      Rogersb [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ]
      Simple free-text terms(((“semi-structured” OR semistructured OR unstructured OR informal OR “in-depth” OR indepth OR “face-to-face” OR structured OR guide) ADJ3 (interview∗ OR discussion∗ OR questionnaire∗)) OR (focus group∗ OR qualitative OR ethnograph∗ OR fieldwork OR “field work” OR “key informant”)).ti,ab.
      Rogersc [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ]
      Broad terms 1 (based on Wong [
      • Wong S.S.-L.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.
      ])
      (interview∗ OR experience∗).af. OR qualitative.tw.
      Rogersd [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ]
      Broad terms 2 (based on Shaw [
      • Shaw R.L.
      • Booth A.
      • Sutton A.J.
      • Miller T.
      • Smith J.A.
      • Young B.
      • et al.
      Additional File 2 – Qualitative methodology search strategies. Finding qualitative research: an evaluation of search strategies.
      ])
      (findings OR interview∗ OR qualitative).af.
      Rogerse [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      ]
      Complex free-text termsexcluded from validation
      UTHealth [
      UTHealth [University of Texas Health Science Center at Houston]
      Search filters for various databases: Ovid PsycINFO. Last Updated: Sep 26, 2019. Houston, US-TX: UTHealth.
      ]
      Qualitative studiesexcluded from validation
      # Please refer to database guide for more details on search syntax.
      The filter UTHealth consisting of free-text terms and controlled vocabulary is not executable. Due to syntax errors regarding the placement of parenthesis, this search filter could not be compiled and validated. Rogerse was adapted from Shaw’s “free-text terms’ filter, which was probably developed for MEDLINE via Ovid (according to displayed syntax), to run in PsycINFO via Ovid. Shaw et al. [
      • Shaw R.L.
      • Booth A.
      • Sutton A.J.
      • Miller T.
      • Smith J.A.
      • Young B.
      • et al.
      Finding qualitative research: an evaluation of search strategies.
      ] developed it primarily to locate qualitative research on breastfeeding support. Therefore, it includes search terms like “women’s stor$” or “feminis$”. Regarding Rogerse, some inconsistencies, such as missing field notations, were corrected by the authors, but some weaknesses remained, so it was excluded, too.
      For replicating the “subject heading terms” search by Rogers et al. [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ] (Rogersa), all reported terms were combined with the Boolean operator “OR” and the obviously missing slash for the last subject heading was added. All terms of the broad terms search (Rogersd, based on Shaw et al. [
      • Shaw R.L.
      • Booth A.
      • Sutton A.J.
      • Miller T.
      • Smith J.A.
      • Young B.
      • et al.
      Finding qualitative research: an evaluation of search strategies.
      ]) were connected with Boolean “OR” to be executable.
      Altogether, twelve filters by McKibbon et al. [
      • McKibbon K.A.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
      ] (McKibbona-f) and Rogers et al. [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ] (Rogersa-d), as well as the index term searches (Index Terma,b) were included for validation. Three of these filters comprise solely controlled vocabulary (Index Terma,b, Rogersa). Six filters consist of free-text term searches with one or multiple words in a specific text field (McKibbona,b,d-f, Rogersb). The other three filters combine both text field searches and controlled vocabulary (McKibbonc, Rogersc,d).

      3.2.2 CINAHL

      Four (DeJean [
      • DeJean D.
      • Giacomini M.
      • Simeonov D.
      • Smith A.
      Finding qualitative research evidence for health technology assessment.
      ], Rogersb [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ], Rogersd [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ], Shawb [
      • Shaw R.L.
      • Booth A.
      • Sutton A.J.
      • Miller T.
      • Smith J.A.
      • Young B.
      • et al.
      Finding qualitative research: an evaluation of search strategies.
      ]) out of nineteen identified search filters failed the test for plausibility and replicability. Those filters were excluded from validation (see Supplemental File).
      One of the search filters by Shaw et al. (Shawb [
      • Shaw R.L.
      • Booth A.
      • Sutton A.J.
      • Miller T.
      • Smith J.A.
      • Young B.
      • et al.
      Finding qualitative research: an evaluation of search strategies.
      ]) contains syntax inconsistencies (e.g., missing field notation, incorrect parentheses) and content-related discrepancies or redundant search lines. Rogers et al. [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ] based their filter Rogersd on Shawb, but did not revise the aforementioned weaknesses. The hybrid filter by DeJean et al. [
      • DeJean D.
      • Giacomini M.
      • Simeonov D.
      • Smith A.
      Finding qualitative research evidence for health technology assessment.
      ] connecting multiple filters contains this free-text term filter (Shawb) as well. The mentioned errors were not corrected, either.
      The “simple free-text terms” filter Rogersb [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ] seems to contain obvious documentation errors. Keywords like qualitative or fieldwork, which were used for searches in other databases, are missing in the reported search strategy for CINAHL. It is unclear how the authors intended to include these terms.
      For replicating the search filters by Shaw et al. [
      • Shaw R.L.
      • Booth A.
      • Sutton A.J.
      • Miller T.
      • Smith J.A.
      • Young B.
      • et al.
      Finding qualitative research: an evaluation of search strategies.
      ] and Wilczynski et al. [
      • Wilczynski N.L.
      • Marks S.
      • Haynes R.B.
      Search strategies for identifying qualitative studies in CINAHL.
      ], all searches were translated from Ovid to EBSCO (see Supplemental File).
      Altogether, the 15 filters Nesbit [
      • Nesbit K.
      Evidence-based filters for Ovid CINAHL. Date Last Updated 2012-09-26. University of Rochester.
      ], Rogersa,c [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ], Shawa,c [
      • Shaw R.L.
      • Booth A.
      • Sutton A.J.
      • Miller T.
      • Smith J.A.
      • Young B.
      • et al.
      Finding qualitative research: an evaluation of search strategies.
      ], Wilczynskia-h [
      • Wilczynski N.L.
      • Marks S.
      • Haynes R.B.
      Search strategies for identifying qualitative studies in CINAHL.
      ], as well as the heading searches (Headinga,b) were included for validation. Ten of these filters consist solely of controlled vocabulary (Headinga,b, Rogersa, Shawa, Wilczynskia-e,g). The other five filters combine text field searches and controlled vocabulary (Nesbit, Rogersc, Shawc, Wilczynskif,h). None of these filters incorporates exclusively one or multiple words in a specific text field.

      3.3 Generation of the gold standard

      The overview of systematic reviews by Dalton et al. [
      • Dalton J.
      • Booth A.
      • Noyes J.
      • Sowden A.J.
      Potential value of systematic reviews of qualitative evidence in informing user-centered health and social care: findings from a descriptive overview.
      ] identified 145 systematic reviews that included 3,012 primary studies. All publications were managed in EndNote. Studies whose study design is described by the review authors as quantitative and publications not indexed in bibliographic databases (e.g., gray literature) were excluded, and the duplicates were removed. This resulted in 2,715 primary studies.
      Nearly half of the primary studies were indexed in PsycINFO (n = 1,315, 48.4%). Included articles were published between 1969 and 2014. Whereas for CINAHL, 2,092 primary studies (77.1%) were indexed, published between 1981 and 2014.

      3.4 Validation of search filters

      3.4.1 PsycINFO

      Validating search filters with the GS showed that Rogersc [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ], which combines text field searches with controlled vocabulary, had the best sensitivity (95.44%, Table 3). It detected 1,255 of 1,315 references. The best precision ratio (4.73) was achieved with Rogersb, which searches the fields title and abstract with multiple words. However, sensitivity was comparatively low (72.70%). Among filters with a sensitivity >80%, McKibbonf demonstrated the best balance of sensitivity and precision. Thus, it was 2.38 times more precise than the very sensitive search filter Rogersc (Table 3). More details are presented in Supplemental File.
      Table 3Validation results for PsycINFO (1,315 qualitative studies)
      Search filterSensitivity (%)Precision ratio
      Index Terma0.611.28
      Index Termb2.363.00
      McKibbona [
      • McKibbon K.A.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
      ]
      15.132.04
      McKibbonb [
      • McKibbon K.A.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
      ]
      51.564.67
      McKibbonc [
      • McKibbon K.A.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
      ]
      89.891.48
      McKibbond [
      • McKibbon K.A.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
      ]
      59.324.60
      McKibbone [
      • McKibbon K.A.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
      ]
      72.174.16
      McKibbonf [
      • McKibbon K.A.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
      ]
      85.932.38
      Best balance of sensitivity and precision (filter with highest precision among filters with sensitivity > 80%).
      Rogersa [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ]
      4.030.64
      Rogersb [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ]
      72.704.73
      Rogersc [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ]
      95.44
      Best sensitivity (precision ratio set to 1.00).
      1.00
      Rogersd [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ]
      94.681.02
      Best sensitivity (precision ratio set to 1.00).
      # Best balance of sensitivity and precision (filter with highest precision among filters with sensitivity > 80%).
      When considering validation results for GS separately for systematic reviews and in the combination of primary studies and systematic reviews, a different trend was observable (see Supplemental File for details).

      3.4.2 CINAHL

      The filter Rogersa reached the best sensitivity (94.55%, Table 4). It located 1.978 of 2.092 articles. The best precision ratio (7.03) was achieved with Wilczynskib. It exclusively searches the term Audiorecording as the exact subject heading, resulting in a low sensitivity (35.66%). Among those filters achieving a sensitivity >80%, Wilczynskih showed the best balance of sensitivity and precision. Compared to Rogersa, the most sensitive filter, it was 3.59 times more precise (precision ratio, Table 4). More details are presented in Supplemental File.
      Table 4Validation results for CINAHL (2,092 qualitative studies)
      Search filterSensitivity (%)Precision ratio
      Headinga54.405.35
      Headingb66.634.78
      Nesbit [
      • Nesbit K.
      Evidence-based filters for Ovid CINAHL. Date Last Updated 2012-09-26. University of Rochester.
      ]
      88.863.21
      Rogersa [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ]
      94.55
      Best sensitivity (precision ratio set to 1.00).
      1.00
      Rogersc [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ]
      92.111.37
      Shawa [
      • Shaw R.L.
      • Booth A.
      • Sutton A.J.
      • Miller T.
      • Smith J.A.
      • Young B.
      • et al.
      Finding qualitative research: an evaluation of search strategies.
      ]
      90.581.02
      Shawc [
      • Shaw R.L.
      • Booth A.
      • Sutton A.J.
      • Miller T.
      • Smith J.A.
      • Young B.
      • et al.
      Finding qualitative research: an evaluation of search strategies.
      ]
      92.451.24
      Wilczynskia [
      • Wilczynski N.L.
      • Marks S.
      • Haynes R.B.
      Search strategies for identifying qualitative studies in CINAHL.
      ]
      71.703.06
      Wilczynskib [
      • Wilczynski N.L.
      • Marks S.
      • Haynes R.B.
      Search strategies for identifying qualitative studies in CINAHL.
      ]
      35.667.03
      Wilczynskic [
      • Wilczynski N.L.
      • Marks S.
      • Haynes R.B.
      Search strategies for identifying qualitative studies in CINAHL.
      ]
      72.660.56
      Wilczynskid [
      • Wilczynski N.L.
      • Marks S.
      • Haynes R.B.
      Search strategies for identifying qualitative studies in CINAHL.
      ]
      93.120.56
      Wilczynskie [
      • Wilczynski N.L.
      • Marks S.
      • Haynes R.B.
      Search strategies for identifying qualitative studies in CINAHL.
      ]
      91.061.45
      Wilczynskif [
      • Wilczynski N.L.
      • Marks S.
      • Haynes R.B.
      Search strategies for identifying qualitative studies in CINAHL.
      ]
      43.166.23
      Wilczynskig [
      • Wilczynski N.L.
      • Marks S.
      • Haynes R.B.
      Search strategies for identifying qualitative studies in CINAHL.
      ]
      51.825.97
      Wilczynskih [
      • Wilczynski N.L.
      • Marks S.
      • Haynes R.B.
      Search strategies for identifying qualitative studies in CINAHL.
      ]
      83.56
      Best balance of sensitivity and precision (filter with highest precision among filters with sensitivity > 80%).
      3.59
      Best balance of sensitivity and precision (filter with highest precision among filters with sensitivity > 80%).
      Best sensitivity (precision ratio set to 1.00).
      # Best balance of sensitivity and precision (filter with highest precision among filters with sensitivity > 80%).
      Performance measures for GS separated for systematic reviews and in the combination of primary studies and systematic reviews are presented in Supplemental File, too.

      4. Discussion

      4.1 Main findings

      This is a comprehensive overview of published search filters for locating qualitative research in the bibliographic databases PsycINFO and CINAHL. Validation results, which were obtained using a newly created GS irrespective of a medical or health-related topic, are presented. Based on our findings, the selection of a search filter depends on the specific research question and individual requirements.
      As already known and described elsewhere [
      • Booth A.
      Searching for qualitative research for inclusion in systematic reviews: a structured methodological review.
      ,
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ,
      • Evans D.
      Database searches for qualitative research.
      ], reporting on qualitative research is heterogeneous. It lacks a standardized vocabulary, which may be due to the numerous research designs, methods, and approaches utilized.
      Taking into account that it is difficult to identify all relevant articles, McKibbonf can be recommended for identifying qualitative primary studies in PsycINFO. This finding corresponds with McMaster University, suggesting it as the filter with the “Best balance of sensitivity and specificity” [
      HiRU [Health Information Research Unit]
      Search strategies for PsycINFO in Ovid syntax. Last modified February 9, 2016. Hamilton, CA-ON: HiRU.
      ]. This could be beneficial to get a comprehensive overview of qualitative research on a specific topic. In contrast, e.g., when searching qualitative research for conducting a systematic review, one would prefer the most sensitive filter Rogersc. This filter is also based on McKibbon’s three-term strategy. However, Rogers and colleagues [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ] extended it by using truncation and searching more fields (all searchable fields [af] instead of text word [tw], see Table 2).
      Wilczynskih demonstrated the best balance in terms of locating qualitative primary studies in CINAHL. This filter combines free-text searches in the title and abstract fields with two subject headings, including the heading Audiorecording used in Wilczynskib. Instead, when preparing a systematic review with the aim to find all relevant qualitative studies, one would choose the most sensitive filter Rogersa. Notably, this filter solely consists of controlled vocabulary.
      Although, employing controlled vocabulary yields (very) good precision in both databases, it is not advisable to use it as a sole search string due to very poor sensitivity, even if the NNR implies to be low.
      Besides strong variances in performance measures, we have demonstrated that qualitative research could be located with sufficiently good results in both databases. Regarding PsycINFO, the best-balanced filter consists of only three free-text terms, whereas for CINAHL, a somewhat more complex search strategy, including free-text terms, as well as controlled vocabulary, demonstrated to be best balanced. Both filters were already developed in 2006 and 2007, respectively. Although the age of the filters seems not to be relevant, it may be worth investigating further changes in terminology and controlled vocabulary.

      4.2 Limitations

      It needs to be stated that the presented precision calculations are rough estimates. Foremost, it cannot be negated that they are underestimated, i.e., the true precision is likely to be higher. References retrieved by search filters may contain qualitative research that was not part of our GS. As a result, using the relative recall approach prevents a proper calculation of the NNR, as this would require the exact number of irrelevant but found articles by the filters. Thus, the NNR may be underestimated, i.e., the presented NNR may be lower than it actually is. To compensate for this and to allow the comparison of the validated search filters, we calculated the precision ratio as a means of normalization. However, when interpreting the results, one needs to keep in mind that performance measures may differ if search filters will be added to a thematic search block (i.e., topic search terms).
      Likewise, the size of the GS for PsycINFO turned out to be small, because only half of the references included in the systematic reviews encompassed in Dalton et al. [
      • Dalton J.
      • Booth A.
      • Noyes J.
      • Sowden A.J.
      Potential value of systematic reviews of qualitative evidence in informing user-centered health and social care: findings from a descriptive overview.
      ] were indexed. Also, a selection bias may have been introduced. Few review authors used a published search filter for locating qualitative studies (see Supplemental File), which may have led to an overestimation of validation results. Additionally, some authors aimed to exclude studies of poor quality. Its potential impact could not be investigated due to scarce reporting, but it seems to be negligible. Moreover, we did not manually check the individual references of the GS in terms of their correct classification as qualitative research.
      Eleven search filters for CINAHL were originally designed for Ovid. We translated these strategies to run in EBSCO. While we cannot preclude that this may have an impact on the behavior of the filter originally intended by the developer, all adaptations were minor in complexity (see Supplemental File for more details).
      Due to feasibility, we did not contact authors to resolve discrepancies or ambiguities. Therefore respective search filters were excluded from validation, e.g., obviously incomplete search strategy (Rogersb).
      To decide on whether and when a search filter for qualitative research should be applied or not strongly depends on the project-specific requirements and circumstances. For example, in some cases, there is no need to use a filter when the number of records retrieved is low due to the inclusion of a thematic block. This increases the sensitivity and concurrently shifts the workload of filtering qualitative articles to the screening process.

      5. Conclusions

      This is the first publication that presents validation results for a broad range of published search filters for locating qualitative research in PsycINFO and CINAHL [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ,
      • McKibbon K.A.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
      ,
      • Wilczynski N.L.
      • Marks S.
      • Haynes R.B.
      Search strategies for identifying qualitative studies in CINAHL.
      ,
      • Shaw R.L.
      • Booth A.
      • Sutton A.J.
      • Miller T.
      • Smith J.A.
      • Young B.
      • et al.
      Finding qualitative research: an evaluation of search strategies.
      ,
      • Nesbit K.
      Evidence-based filters for Ovid CINAHL. Date Last Updated 2012-09-26. University of Rochester.
      ]. The GS for these two bibliographic databases were created using the relative recall approach irrespective of a medical or health-related topic.
      If a sensitivity above 80% is wanted, McKibbonf can be recommended as the best-balanced filter to be used in PsycINFO. Regarding CINAHL, Wilczynskih meets these requirements. However, if completeness is requested, Rogersc and Rogersa seem to be the appropriate choices for PsycINFO and CINAHL, respectively. But, of course, none of these filters can achieve completeness.

      CRediT authorship contribution statement

      Stefanie Rosumeck: Conceptualization, Methodology, Investigation, Formal analysis, Resources, Writing - review & editing. Mandy Wagner: Conceptualization, Methodology, Formal analysis, Resources, Writing - review & editing. Simon Wallraf: Investigation, Formal analysis, Writing - review & editing. Ulrike Euler: Writing - review & editing.

      Acknowledgments

      The authors kindly thank Kristina Döring and Christian Küffmeier for their assistance in the generation of the gold standards and gratefully acknowledge Dr. Laura Haas for proof-reading.
      This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

      Supplementary data

      References

        • Booth A.
        Searching for qualitative research for inclusion in systematic reviews: a structured methodological review.
        Syst Rev. 2016; 5
        • Noyes J.
        • Booth A.
        • Cargo M.
        • Flemming K.
        • Harden A.
        • Harris J.
        • et al.
        Qualitative evidence. Chapter 21.
        in: Higgins J.P.T. Thomas J. Chandler J. Cumpston M. Li T. Page M.J. Cochrane Handbook for Systematic Reviews of Interventions (Version 6). Wiley-Blackwell, Hoboken, US-NJ2019: 525-545
        • AGREE Next Steps Consortium
        Appraisal of guidelines for research & evaluation II. AGREE II Instrument. AGREE Research Trust.
        (Available at)
        • Rogers M.
        • Bethel A.
        • Abbott R.
        Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
        Res Synth Methods. 2018; 9: 579-586
        • Evans D.
        Database searches for qualitative research.
        J Med Libr Assoc. 2002; 90: 290-293
        • Lefebvre C.
        • Glanville J.
        • Beale S.
        • Boachie C.
        • Duffy S.
        • Fraser C.
        • et al.
        Assessing the performance of methodological search filters to improve the efficiency of evidence information retrieval: five literature reviews and a qualitative study.
        Health Technol Assess. 2017; 21
        • Wagner M.
        • Rosumeck S.
        • Küffmeier C.
        • Döring K.
        • Euler U.
        A validation study revealed differences in design and performance of MEDLINE search filters for qualitative research.
        J Clin Epidemiol. 2020; 120: 17-24
        • Higgins J.P.T.
        • Green S.
        Qualitative evidence: searching for qualitative evidence. Chapter 21, section 21.7.
        in: Higgins J.P.T. Thomas J. Chandler J. Cumpston M. Li T. Page M.J. Cochrane Handbook for Systematic Reviews of Interventions (Version 6). Wiley-Blackwell, Hoboken, US-NJ2019: 531-532
      1. Glanville J, Lefebvre C, Wright K, editors. Filters to identify qualitative research. Update date: 4 February 2020. York, GB: ISSG [InterTASC Information Specialists' Sub-Group]. Available at https://sites.google.com/a/york.ac.uk/issg-search-filters-resource/filters-to-identify-qualitative-research. Accessed April 9, 2020.

        • HiRU [Health Information Research Unit]
        Health information research unit. Our mission... Last modified February 9, 2016. Hamilton, CA-ON: HiRU.
        (Available at)
        https://hiru.mcmaster.ca/hiru/Default.aspx
        Date accessed: April 9, 2020
        • McKibbon K.A.
        • Wilczynski N.L.
        • Haynes R.B.
        Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
        Eval Health Prof. 2006; 29: 440-454
        • Wilczynski N.L.
        • Marks S.
        • Haynes R.B.
        Search strategies for identifying qualitative studies in CINAHL.
        Qual Health Res. 2007; 17: 705-710
        • Wong S.S.-L.
        • Wilczynski N.L.
        • Haynes R.B.
        Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.
        Stud Health Technol Inform. 2004; 107: 311-316
        • Durão S.
        • Kredo T.
        • Volmink J.
        Validation of a search strategy to identify nutrition trials in PubMed using the relative recall method.
        J Clin Epidemiol. 2015; 68: 610-616
        • Sampson M.
        • Zhang L.
        • Morrison A.
        • Barrowman N.J.
        • Clifford T.J.
        • Platt R.W.
        • et al.
        An alternative to the hand searching gold standard: validating methodological search filters using relative recall.
        BMC Med Res Methodol. 2006; 6: 33
        • Waffenschmidt S.
        • Hermanns T.
        • Gerber-Grote A.
        • Mostardt S.
        No suitable precise or optimized epidemiologic search filters were available for bibliographic databases.
        J Clin Epidemiol. 2017; 82: 112-118
        • Dalton J.
        • Booth A.
        • Noyes J.
        • Sowden A.J.
        Potential value of systematic reviews of qualitative evidence in informing user-centered health and social care: findings from a descriptive overview.
        J Clin Epidemiol. 2017; 88: 37-46
        • DeJean D.
        • Giacomini M.
        • Simeonov D.
        • Smith A.
        Finding qualitative research evidence for health technology assessment.
        Qual Health Res. 2016; 26: 1307-1317
        • Flemming K.
        • Briggs M.
        Electronic searching to locate qualitative research: evaluation of three strategies.
        J Adv Nurs. 2007; 57: 95-100
        • Gorecki C.A.
        • Brown J.M.
        • Briggs M.
        • Nixon J.
        Evaluation of five search strategies in retrieving qualitative patient-reported electronic data on the impact of pressure ulcers on quality of life.
        J Adv Nurs. 2010; 66: 645-652
        • Shaw R.L.
        • Booth A.
        • Sutton A.J.
        • Miller T.
        • Smith J.A.
        • Young B.
        • et al.
        Finding qualitative research: an evaluation of search strategies.
        BMC Med Res Methodol. 2004; 4: 5
        • HiRU [Health Information Research Unit]
        Search strategies for PsycINFO in Ovid syntax. Last modified February 9, 2016. Hamilton, CA-ON: HiRU.
        (Available at)
        • UTHealth [University of Texas Health Science Center at Houston]
        Search filters for various databases: Ovid PsycINFO. Last Updated: Sep 26, 2019. Houston, US-TX: UTHealth.
        (Available at)
        • Nesbit K.
        Evidence-based filters for Ovid CINAHL. Date Last Updated 2012-09-26. University of Rochester.
        (Available at)
        http://hdl.handle.net/1802/6440
        Date: 2002
        Date accessed: April 9, 2020
        • APA [American Psychological Association]
        Thesaurus of psychological index terms 2019 Update.
        APA, Washington, US-DC2019 (Available at)
        • Shaw R.L.
        • Booth A.
        • Sutton A.J.
        • Miller T.
        • Smith J.A.
        • Young B.
        • et al.
        Additional File 2 – Qualitative methodology search strategies. Finding qualitative research: an evaluation of search strategies.
        (Available at)
        • Rogers M.
        • Bethel A.
        • Abbott R.
        Data S1. Supporting Information.