Advertisement

A validation study revealed differences in design and performance of MEDLINE search filters for qualitative research

Open AccessPublished:December 17, 2019DOI:https://doi.org/10.1016/j.jclinepi.2019.12.008

      Abstract

      Objectives

      Several search filters exist to identify qualitative research, but so far none of them has been validated with an independent set of relevant references irrespective of a medical topic. The objective of this study was to provide a comparative overview of validation results for various MEDLINE search filters.

      Study Design and Setting

      Search filters were tested for plausibility. A relative recall approach was used to generate a gold standard based on an overview of systematic reviews of qualitative studies. For each review, the included qualitative studies were collected and checked for MEDLINE-indexing. The body of indexed articles yielded the gold standard. Validation tests were conducted to determine whether the references of the gold standard could be identified with the respective search filters.

      Results

      Thirteen search filters were validated in MEDLINE. One search filter by Wong et al. (2004) was found to be the most sensitive (93.63%). While medical subject heading “qualitative research” achieved the best precision (2.15%), sensitivity was the lowest (22.56%). University of Texas provided the best balanced search filter with a sensitivity of 81.96% and a precision of 0.80%.

      Conclusion

      Search filters to identify qualitative research in MEDLINE differ greatly in design and performance. The selection of the appropriate search filter depends on project-specific demands and resources.

      Keywords

      What is new?

        Key findings

      • MEDLINE search filters for identification of qualitative research differ greatly in design and performance.

        What this adds to what was known?

      • Initial validation of a search filter designed by University of Texas.
      • Comparative overview of validation performance of MEDLINE search filters using a uniform gold standard (independent from medical topic and with literature published in several years).

        What is the implication, what should change now?

      • It is not possible to suggest a single search filter to identify qualitative research in MEDLINE.
      • The choice for or against a search filter depends on project-specific demands and resources.

      1. Introduction

      Qualitative studies provide deep insights into patient preferences and experiences. In recent years, qualitative research has established itself as an important contribution to evidence-based health care. Therefore, studies of this kind should be considered for high-quality systematic reviews, as stated by the Cochrane Handbook for Systematic Reviews of Interventions [
      • Noyes J.
      • Popay J.
      • Pearson A.
      • Hannes K.
      • Booth A.
      Chapter 20: Qualitative research and Cochrane reviews. Version 5.1.0. Updated 20.03.2011. London: Cochrane Collaboration.
      ,
      • Booth A.
      Chapter 3: searching for studies.
      ]. Qualitative research comprises many research designs, methods, and approaches such as interviews, focus groups, ethnography, grounded theory, etc. Furthermore, published qualitative studies are often unstructured and contain no or only insufficient standardized vocabulary [
      • Evans D.
      Database searches for qualitative research.
      ]. In contrast to other study designs such as randomized controlled trials, it is therefore more difficult and complex to find qualitative studies in bibliographic databases [
      • Booth A.
      Searching for qualitative research for inclusion in systematic reviews: a structured methodological review.
      ,
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ]. To get a comprehensive picture of qualitative studies as well as systematic reviews of qualitative studies on a specific medical or health-related topic, a search filter with a balanced ratio of sensitivity (retrieval of all relevant articles) and precision (retrieval of relevant articles as a proportion of total number of records found) can be a helpful tool for systematic search in bibliographic databases [
      • Beale S.
      • Duffy S.
      • Glanville J.
      • Lefebvre C.
      • Wright D.
      • McCool R.
      • et al.
      Choosing and using methodological search filters: searchers’ views.
      ].
      Many search filters were not adequately validated by means of an independent set of relevant references (gold standard). Moreover, gold standards (GSs) are often restricted, lacking an optimal size and references from several years [
      • Jenkins M.
      Evaluation of methodological search filters – a review.
      ]. In addition, topic-specific GS limit generalizability.
      The aim of the present study was therefore to get an overview of already available search filters to identify qualitative research papers in the bibliographic database MEDLINE. To determine the filter with the highest sensitivity, the highest precision or the best balance between sensitivity and precision, all filters were validated with a newly generated GS without the constraint of a medical topic. With these validation results in mind, a search filter for MEDLINE can be selected with respect to individual requirements.

      2. Methods

      A four-stepped approach was used: 1. identification of search filters to locate qualitative research in the bibliographic database MEDLINE, 2. verification and plausibility checks of search filters, 3. generation of a GS irrespective of a medical or health-related topic, 4. validation of identified search filters against the newly created GS in MEDLINE using the Ovid platform.

      2.1 Identification of search filters

      A search was carried out to identify the most important search filters to locate qualitative research in the bibliographic database MEDLINE. Therefore, well-known sources and websites, for example, from the InterTASC Information Specialists' Subgroup [
      • Glanville J.
      • Lefebvre C.
      • Wright K.
      ISSG search filters resource. ISSG [InterTASC Information Specialists’ Sub-Group]. York (UK).
      ], the Cochrane Handbook for Systematic Reviews of Interventions [
      • Noyes J.
      • Popay J.
      • Pearson A.
      • Hannes K.
      • Booth A.
      Chapter 20: Qualitative research and Cochrane reviews. Version 5.1.0. Updated 20.03.2011. London: Cochrane Collaboration.
      ,
      • Booth A.
      Chapter 3: searching for studies.
      ], and the McMaster University's department of Health Information Research Unit [
      HiRU [Health Information Research Unit]
      Search Filters for Medline in Ovid Syntax and the PubMed translation. Last modified: February 9, 2016. Hamilton, CA: HiRU.
      ] were extensively searched.

      2.2 Verification and plausibility checks of search filters

      For validation, all search filters for qualitative studies were included that were generated for the bibliographic database MEDLINE using the Ovid platform, that were plausible and completely reproducible.
      For replicability of search filters, a plausibility check was performed on all search filters identified. If search filters contained considerable syntax errors or content inconsistencies, they were excluded from validation. Developers and authors were not contacted.
      The Medical Subject Heading (MeSH) “Qualitative Research” was introduced in 2003. However, some of the identified filters were developed before implementation of this controlled vocabulary and so far have not been updated [
      • McKibbon K.A.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
      ,
      • Wong S.S.-L.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.
      ]. Besides, the heading “Qualitative Research” comprises the sole narrower term “Hermeneutics”, with an entry date of 2014, resulting in two additional search filters (“Qualitative Research/” and “exp Qualitative Research/”) [
      NLM [U. S. National Library of Medicine]. Qualitative Research
      MeSH Descriptor Data 2019. Revision Date: 16.06.2014. Rockville Pike, US-MD: NLM.
      ].

      2.3 Generation of a gold standard

      A GS can be generated using various methods. On the one hand, journals were hand searched in which topic-relevant publications can be expected [
      • McKibbon K.A.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
      ,
      • Wong S.S.-L.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.
      ,
      • Wilczynski N.L.
      • Marks S.
      • Haynes R.B.
      Search strategies for identifying qualitative studies in CINAHL.
      ]. On the other hand, relevant articles can also be identified in a bibliographic database [
      • DeJean D.
      • Giacomini M.
      • Simeonov D.
      • Smith A.
      Finding qualitative research evidence for health technology assessment.
      ]. In contrast to creating a GS by manually compiling references (hand search), which is very time-consuming, Sampson et al. [
      • Sampson M.
      • Zhang L.
      • Morrison A.
      • Barrowman N.J.
      • Clifford T.J.
      • Platt R.W.
      • et al.
      An alternative to the hand searching gold standard: validating methodological search filters using relative recall.
      ] formed a GS by combining all studies that were included in various systematic reviews (relative recall approach). This allowed for the generation of a GS that includes relevant publications over a long period, regardless of a medical or health-related topic. The relative recall approach has already been used [
      • Durão S.
      • Kredo T.
      • Volmink J.
      Validation of a search strategy to identify nutrition trials in PubMed using the relative recall method.
      ,
      • Waffenschmidt S.
      • Hermanns T.
      • Gerber-Grote A.
      • Mostardt S.
      No suitable precise or optimized epidemiologic search filters were available for bibliographic databases.
      ] and is therefore used in the context of this validation.
      A review by Dalton et al. [
      • Dalton J.
      • Booth A.
      • Noyes J.
      • Sowden A.J.
      Potential value of systematic reviews of qualitative evidence in informing user-centered health and social care: findings from a descriptive overview.
      ] formed the basis to generate the GS. The aim of Dalton et al. was to provide a descriptive overview of systematic reviews of qualitative studies listed in the Database of Abstracts of Reviews of Effects (DARE). A search for the period from 2009 to 2014—using the internal tagging system—yielded 145 published systematic reviews. Until 2014, DARE itself includes hits from MEDLINE, Embase, PsycINFO, PubMed, and CINAHL and tagged relevant reviews as qualitative research since 2009 [
      • Dalton J.
      • Booth A.
      • Noyes J.
      • Sowden A.J.
      Potential value of systematic reviews of qualitative evidence in informing user-centered health and social care: findings from a descriptive overview.
      ,
      Centre for Reviews and Dissemination. York (UK): University of York.
      ]. Therefore, the source for the GS can be considered as comprehensive.
      In a first step, all 145 systematic reviews of qualitative studies included in Dalton et al. [
      • Dalton J.
      • Booth A.
      • Noyes J.
      • Sowden A.J.
      Potential value of systematic reviews of qualitative evidence in informing user-centered health and social care: findings from a descriptive overview.
      ] were obtained in full text and managed in EndNote. Thereafter, all primary studies included in these systematic reviews that also were available as journal articles or dissertations were loaded in an EndNote library. Gray literature and books were not considered because they are generally not indexed in bibliographic databases. There was no assessment as to whether the primary studies applied qualitative methods, as we assume that they are all qualitative studies and thus fulfill our inclusion criteria. However, if the included studies in the systematic reviews were clearly divided into qualitative and quantitative primary studies, only qualitative studies were considered to compile the GS (e.g., Torquato Lopes et al. [
      • Torquato Lopes A.P.A.
      • das Neves Decesaro M.
      The adjustments experienced by persons with an ostomy: an integrative review of the literature.
      ]). Finally, duplicates were removed and only publications indexed in MEDLINE were added to the entire GS.

      2.4 Validation of search filters

      During the validation process, it was determined whether the references of the GS could be identified with the respective filter in MEDLINE via Ovid. Based on the number of hits, the sensitivity, the precision, and the number needed to read (NNR) were calculated (Table 1). The relative recall approach does not allow for the estimation of the true precision and NNR [
      • Waffenschmidt S.
      • Hermanns T.
      • Gerber-Grote A.
      • Mostardt S.
      No suitable precise or optimized epidemiologic search filters were available for bibliographic databases.
      ]. However, to enhance the comparability of search filter performances, the precision ratio was calculated, while setting the precision ratio of 1 for the search filter with the highest sensitivity [
      • Waffenschmidt S.
      • Hermanns T.
      • Gerber-Grote A.
      • Mostardt S.
      No suitable precise or optimized epidemiologic search filters were available for bibliographic databases.
      ].
      Table 1Calculations
      Definitionsa—relevant articles of the gold standard retrieved with search filter
      b—irrelevant articles
      Could be articles relating to qualitative research and relevant but are not part of the gold standard.
      retrieved with search filter (hits in MEDLINE for each search filter minus references of the gold standard)
      c—relevant articles of the gold standard not retrieved with search filter
      Sensitivity (%)a/(a + c)*100
      Precision (%)a/(a + b)*100
      NNR(a + b)/a
      Precision ratePrecision/precision of the search filter with the highest sensitivity
      Abbreviation: NNR, number needed to read.
      # Could be articles relating to qualitative research and relevant but are not part of the gold standard.

      3. Results

      3.1 Identification of search filters

      Thirteen search filters for qualitative research could be identified [
      • McKibbon K.A.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
      ,
      • Wong S.S.-L.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.
      ,
      • DeJean D.
      • Giacomini M.
      • Simeonov D.
      • Smith A.
      Finding qualitative research evidence for health technology assessment.
      ,
      UTHealth [University of Texas Health Science Center at Houston]
      Search Filters for Various Databases: Ovid Medline. Last Updated: 28.11.2018. Houston, US-TX: UTHealth.
      ,
      • Shaw R.L.
      • Booth A.
      • Sutton A.J.
      • Miller T.
      • Smith J.A.
      • Young B.
      • et al.
      Finding qualitative research: an evaluation of search strategies.
      ] as well as the MeSH-term “Qualitative Research” (exploded as well as unexploded [
      NLM [U. S. National Library of Medicine]. Qualitative Research
      MeSH Descriptor Data 2019. Revision Date: 16.06.2014. Rockville Pike, US-MD: NLM.
      ]).
      Shaw et al. [
      • Shaw R.L.
      • Booth A.
      • Sutton A.J.
      • Miller T.
      • Smith J.A.
      • Young B.
      • et al.
      Finding qualitative research: an evaluation of search strategies.
      ] evaluated three different filters (“thesaurus terms”, “free-text terms”, “broad-based terms”) for several databases including MEDLINE via Ovid, where the GS consisted of references on support for breastfeeding. The McMaster University hedges team developed seven different search filters for qualitative studies that can be used for a variety of purposes (different “single term” and “two or three term” strategies) [
      • Wong S.S.-L.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.
      ]. The GS used were publications from 161 health care journals from the year 2000, which dealt with patient experiences and where data were gathered using qualitative methods [
      • Wong S.S.-L.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.
      ]. In a later publication, McKibbon et al. [
      • McKibbon K.A.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
      ], also from the McMaster University hedges team, list another filter which searched a single word in multiple fields. The aforementioned search filters were developed and published without taking into account the MeSH-term “Qualitative Research”, introduced in 2003 [
      NLM [U. S. National Library of Medicine]. Qualitative Research
      MeSH Descriptor Data 2019. Revision Date: 16.06.2014. Rockville Pike, US-MD: NLM.
      ]. The website of the University of Texas—School of Public Health (UTHealth) lists a further, nonvalidated search strategy for qualitative studies considering the MeSH-term mentioned previously [
      UTHealth [University of Texas Health Science Center at Houston]
      Search Filters for Various Databases: Ovid Medline. Last Updated: 28.11.2018. Houston, US-TX: UTHealth.
      ]. In a publication by DeJean et al. [
      • DeJean D.
      • Giacomini M.
      • Simeonov D.
      • Smith A.
      Finding qualitative research evidence for health technology assessment.
      ], an additional hybrid filter was validated.
      None of these search filters have been validated with a topic-independent GS covering published references from several years. In addition, no comparison was made in terms of sensitivity and precision.

      3.2 Verification and plausibility checks of search filters for qualitative research

      A total of 15 search filters were tested for plausibility (Table 2). All seven different search filters by Wong et al. (Wonga-g) and the search filter from UTHealth were included for validation [
      • Wong S.S.-L.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.
      ,
      UTHealth [University of Texas Health Science Center at Houston]
      Search Filters for Various Databases: Ovid Medline. Last Updated: 28.11.2018. Houston, US-TX: UTHealth.
      ]. The search filters Wongc (“maximizes sensitivity”), Wonge (“maximizes specificity”), and Wongg (“best balance of sensitivity and specificity”) can also be found on the McMaster University website [
      HiRU [Health Information Research Unit]
      Search Filters for Medline in Ovid Syntax and the PubMed translation. Last modified: February 9, 2016. Hamilton, CA: HiRU.
      ]. The single-term strategy “interviews.mp.”, identified in a paper by McKibbon et al. [
      • McKibbon K.A.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
      ], could be a transfer error while testing multiple hedges preliminary for the database PsycINFO. It is possible, that the single-term strategy “interviews.tw.” by Wong et al. (Wongb) [
      • Wong S.S.-L.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.
      ] with a different field notation could have been meant. As this uncertainty could not be ruled out, “interviews.mp.” (McKibbon) was kept as another search filter to identify qualitative articles [
      • McKibbon K.A.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
      ].
      Table 2Search strategies of all validated search filters
      Search filterLabeled by the filter author(s)Search strategy for MEDLINE (via Ovid)1
      Wonga [
      • Wong S.S.-L.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.
      ]
      Single term—best sensitivity and single term—best optimization of sensitivity and specificityinterview:.mp.
      Wongb [
      • Wong S.S.-L.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.
      ]
      Single term—best specificityinterviews.tw.
      Wongc [
      HiRU [Health Information Research Unit]
      Search Filters for Medline in Ovid Syntax and the PubMed translation. Last modified: February 9, 2016. Hamilton, CA: HiRU.
      ,
      • Wong S.S.-L.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.
      ]
      Two or three terms—best sensitivityinterview:.tw. or px.fs. or exp health services administration/
      Wongd [
      • Wong S.S.-L.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.
      ]
      Two or three terms—best sensitivity—small decrease in sensitivity with large increase in specificityinterview:.mp. or px.fs. or qualitative.tw.
      Wonge [
      HiRU [Health Information Research Unit]
      Search Filters for Medline in Ovid Syntax and the PubMed translation. Last modified: February 9, 2016. Hamilton, CA: HiRU.
      ,
      • Wong S.S.-L.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.
      ]
      Two or three terms – best specificityqualitative.tw. or themes.tw.
      Wongf [
      • Wong S.S.-L.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.
      ]
      Two or three terms—best specificity—small decrease in specificity with large increase in sensitivityinterviews.mp,pt. or qualitative.mp. or experiences.tw.
      Wongg [
      HiRU [Health Information Research Unit]
      Search Filters for Medline in Ovid Syntax and the PubMed translation. Last modified: February 9, 2016. Hamilton, CA: HiRU.
      ,
      • Wong S.S.-L.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.
      ]
      Two or three terms—best optimization of sensitivity and specificityinterview:.mp. or experience:.mp. or qualitative.tw.
      McKibbon [
      • McKibbon K.A.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
      ]
      High specificityinterviews.mp.
      Shawa [
      • Shaw R.L.
      • Booth A.
      • Sutton A.J.
      • Miller T.
      • Smith J.A.
      • Young B.
      • et al.
      Finding qualitative research: an evaluation of search strategies.
      ]
      Thesaurus termsQualitative Research/ or Nursing Methodology Research/ or Questionnaires/ or exp Attitude/ or Focus Groups/ or discourse analysis.mp. or content analysis.mp. or ethnographic research.mp. or ethnological research.mp. or ethnonursing research.mp. or constant comparative method.mp. or qualitative validity.mp. or purposive sample.mp. or observational method$.mp. or field stud$.mp. or theoretical sampl$.mp. or phenomenology/ or phenomenological research.mp. or life experience$.mp. or cluster sampl$.mp.
      Shawb [
      • Shaw R.L.
      • Booth A.
      • Sutton A.J.
      • Miller T.
      • Smith J.A.
      • Young B.
      • et al.
      Finding qualitative research: an evaluation of search strategies.
      ]
      Free-text termsexcluded from validation
      Shawc [
      • Shaw R.L.
      • Booth A.
      • Sutton A.J.
      • Miller T.
      • Smith J.A.
      • Young B.
      • et al.
      Finding qualitative research: an evaluation of search strategies.
      ]
      Broad-based termsfindings.af. or interview$.af. or interviews/ or qualitative.af.
      UTHealth [
      UTHealth [University of Texas Health Science Center at Houston]
      Search Filters for Various Databases: Ovid Medline. Last Updated: 28.11.2018. Houston, US-TX: UTHealth.
      ]
      -(((“semi-structured” or semistructured or unstructured or informal or “in-depth” or indepth or “face-to-face” or structured or guide) adj3 (interview* or discussion* or questionnaire*))).ti,ab. or (focus group* or qualitative or ethnograph* or fieldwork or “field work” or “key informant”).ti,ab. or interviews as topic/ or focus groups/ or narration/ or qualitative research/
      MeSHa [
      NLM [U. S. National Library of Medicine]. Qualitative Research
      MeSH Descriptor Data 2019. Revision Date: 16.06.2014. Rockville Pike, US-MD: NLM.
      ]
      -exp Qualitative Research/
      MeSHb [
      NLM [U. S. National Library of Medicine]. Qualitative Research
      MeSH Descriptor Data 2019. Revision Date: 16.06.2014. Rockville Pike, US-MD: NLM.
      ]
      -Qualitative Research/
      DeJean [
      • DeJean D.
      • Giacomini M.
      • Simeonov D.
      • Smith A.
      Finding qualitative research evidence for health technology assessment.
      ]
      Hybrid filter (compilation of different search filters connected with Boolean operator ‘OR’)excluded from validation
      1 notations like *,:, or $ are database specific syntax (please refer to database guide for more details).
      Of the three search filters developed by Shaw et al. [
      • Shaw R.L.
      • Booth A.
      • Sutton A.J.
      • Miller T.
      • Smith J.A.
      • Young B.
      • et al.
      Finding qualitative research: an evaluation of search strategies.
      ], the search filter with the “free-text terms” (Shawb) was excluded from the validation because it contains both syntax inconsistencies (e.g., missing field notations in some search lines, incorrect parentheses) and content-related discrepancies (e.g., “speigelberg$.tw.” instead of “spiegelberg$.tw.”, redundant search lines). The hybrid filter from DeJean et al. [
      • DeJean D.
      • Giacomini M.
      • Simeonov D.
      • Smith A.
      Finding qualitative research evidence for health technology assessment.
      ] relies among others on Shawb without correcting syntax errors or inconsistencies. It was therefore excluded as well. Both aforementioned filters were constructed for medical and health-related topics with a special focus on women's experiences. Shaw et al. [
      • Shaw R.L.
      • Booth A.
      • Sutton A.J.
      • Miller T.
      • Smith J.A.
      • Young B.
      • et al.
      Finding qualitative research: an evaluation of search strategies.
      ] searched for the topic breastfeeding and DeJean et al. [
      • DeJean D.
      • Giacomini M.
      • Simeonov D.
      • Smith A.
      Finding qualitative research evidence for health technology assessment.
      ] searched for qualitative articles in the field of chronic obstructive pulmonary disease as well as early breast cancer. Including terms such as “women's stor*” or “feminis$” seem appropriate for these topics (if male breast cancer is excluded) but is not suitable for an overall search filter to locate qualitative research in general. Then, terms such as “men's stor*” should have also been incorporated. In conclusion both search filters (Shawb [
      • Shaw R.L.
      • Booth A.
      • Sutton A.J.
      • Miller T.
      • Smith J.A.
      • Young B.
      • et al.
      Finding qualitative research: an evaluation of search strategies.
      ], DeJean [
      • DeJean D.
      • Giacomini M.
      • Simeonov D.
      • Smith A.
      Finding qualitative research evidence for health technology assessment.
      ]) were excluded from validation process.
      However, Shawa (“thesaurus terms”) and Shawc (“broad-based terms”) were included as well as the MeSH-term “Qualitative Research” (exploded and unexploded; MeSHa and MeSHb [
      NLM [U. S. National Library of Medicine]. Qualitative Research
      MeSH Descriptor Data 2019. Revision Date: 16.06.2014. Rockville Pike, US-MD: NLM.
      ]).
      Length of search filters ranged from a single word [
      • McKibbon K.A.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
      ,
      • Wong S.S.-L.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.
      ] or MeSH-term [
      NLM [U. S. National Library of Medicine]. Qualitative Research
      MeSH Descriptor Data 2019. Revision Date: 16.06.2014. Rockville Pike, US-MD: NLM.
      ], up to very long search strategies including free-text search terms and/or controlled vocabulary [
      UTHealth [University of Texas Health Science Center at Houston]
      Search Filters for Various Databases: Ovid Medline. Last Updated: 28.11.2018. Houston, US-TX: UTHealth.
      ,
      • Shaw R.L.
      • Booth A.
      • Sutton A.J.
      • Miller T.
      • Smith J.A.
      • Young B.
      • et al.
      Finding qualitative research: an evaluation of search strategies.
      ], respectively (Table 2).

      3.3 Generation of a gold standard

      Of the 145 systematic reviews included in the publication by Dalton et al. [
      • Dalton J.
      • Booth A.
      • Noyes J.
      • Sowden A.J.
      Potential value of systematic reviews of qualitative evidence in informing user-centered health and social care: findings from a descriptive overview.
      ], 131 were indexed in MEDLINE (90.3%, gold standard 1, Fig. 1). Characteristics of these reviews can be found in the initial publication by Dalton et al. [
      • Dalton J.
      • Booth A.
      • Noyes J.
      • Sowden A.J.
      Potential value of systematic reviews of qualitative evidence in informing user-centered health and social care: findings from a descriptive overview.
      ]. The reviews were published between 2009 and 2014 and focus on different medical or health-related topics, for example, cancer, mental health, or diabetes.
      Figure thumbnail gr1
      Fig. 1Generation of gold standard. (For interpretation of the references to color in this figure legend, the reader is referred to the Web version of this article.)
      For the systematic literature searches described in the systematic reviews, one to 26 bibliographic databases were used (mean: 7). MEDLINE (via PubMed and/or via Ovid) was searched the most, in 96.6% of all systematic reviews (140 of 145). CINAHL was accessed in 87.6%, PsycINFO in 82.8%, and Embase in 63.4% of the systematic reviews.
      In about 60% of included reviews, the search strategies included a search block to identify qualitative studies. This search block was very heterogeneous in the reviews, often only single words were used, such as “qualitative”, “grounded theory”, or “focus groups” without specifying a field notation. A clear reference to previously published search filters was very rare (see Supplementary Table). Owing to a lack of reporting standards regarding search strategies—and in particular the applied search filters—it is unclear how often published filters were used or adapted for the literature search. However, almost all reviews took additional sources (hand searches of specific websites and journals, review of reference lists, contact with authors and experts, forward and backward citation of references, etc.) into account to ensure a systematic search for qualitative studies. In the remaining reviews, the search was carried out without restriction to qualitative studies or no details about the entire literature search strategies were reported.
      In total, the systematic reviews included 3,012 primary studies, of which 2,898 were recorded in an EndNote library, excluding clearly defined quantitative studies, books, and gray literature. After deduplication, 2,715 articles remained, of which 2,192 references were indexed in MEDLINE (80.7%, gold standard 2, Fig. 1). A critical appraisal of the included primary qualitative studies was done using a wide variety of tools in 133 systematic reviews (91.7%). Authors of 46 reviews planned to exclude studies because of their poor quality.
      The entire GS consisting of qualitative studies and systematic reviews on qualitative studies included a total of 2,323 references. The publication period of the references included ranged from 1968 to 2014.

      3.4 Validation of search filters

      The validation with the entire GS showed that the search filter Wongc, which is also preferred by McMaster University as a very sensitive search filter [
      HiRU [Health Information Research Unit]
      Search Filters for Medline in Ovid Syntax and the PubMed translation. Last modified: February 9, 2016. Hamilton, CA: HiRU.
      ], had the best sensitivity in MEDLINE (93.63%) (Table 3). With this filter, 2,175 of 2,323 references could be retrieved. By contrast, there was a NNR of 1,418 articles, indicating a very high screening workload. During validation, the search filter Wongd [
      • Wong S.S.-L.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.
      ] showed a very good sensitivity (92.25%) with a considerably reduced screening effort (NNR: 491 articles), as described by Wong et al. [
      • Wong S.S.-L.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.
      ] with “best sensitivity—small decrease in sensitivity with large increase in specificity”.
      Table 3Validation results with the entire gold standard (2,323 articles)
      Search filterSensitivity (%)Precision (%)NNRPrecision ratio
      Wonga68.230.611658.6
      Wongb52.131.218317.2
      Wongc93.63
      Best sensitivity.
      0.071,4181.0
      Wongd92.250.204912.9
      Wonge63.840.9910214.0
      Wongf84.120.591718.3
      Wongg90.310.204902.9
      McKibbon57.041.069415.1
      Shawa75.380.244153.4
      Shawc88.030.128161.7
      UTHealth81.96
      Best balance between sensitivity and precision (filter with highest precision among filters with sensitivity > 80%).
      0.80
      Best balance between sensitivity and precision (filter with highest precision among filters with sensitivity > 80%).
      126
      Best balance between sensitivity and precision (filter with highest precision among filters with sensitivity > 80%).
      11.3
      Best balance between sensitivity and precision (filter with highest precision among filters with sensitivity > 80%).
      MeSHa22.562.15
      Best precision.
      4730.5
      MeSHb22.562.15
      Best precision.
      4730.5
      Abbreviation: NNR, number needed to read.
      Best sensitivity.
      Best precision.
      Best balance between sensitivity and precision (filter with highest precision among filters with sensitivity > 80%).
      The best precision of 2.15% could be reached with the MeSH-term “Qualitative Research”, but with the lowest sensitivity (22.56%) and a NNR of 47 articles (each, MeSHa and MeSHb) [
      NLM [U. S. National Library of Medicine]. Qualitative Research
      MeSH Descriptor Data 2019. Revision Date: 16.06.2014. Rockville Pike, US-MD: NLM.
      ]. It was irrelevant whether the MeSH-term was exploded or not.
      The search filter of the UTHealth [
      UTHealth [University of Texas Health Science Center at Houston]
      Search Filters for Various Databases: Ovid Medline. Last Updated: 28.11.2018. Houston, US-TX: UTHealth.
      ] demonstrated the best balance between sensitivity and precision (81.96% and 0.80%, respectively) for the entire GS. In this case, 126 articles need to be read to uncover one relevant article (Table 3). This search filter was thus 11.3 times more precise (precision ratio) than the very sensitive search filter Wongc [
      • Wong S.S.-L.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.
      ].
      If the GS is considered separately as systematic reviews of qualitative studies (gold standard 1) and primary studies (gold standard 2), the validation results are slightly different (Table 4). With GS 1, the best sensitivity could be reached with the Wongd search filter (97.71%). One hundred twenty-eight of the 131 references in GS 1 could be identified using this search filter. Furthermore, three search filters presented by Wong et al. [
      • Wong S.S.-L.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.
      ] reached a sensitivity of 96.18% (Wonge, Wongf, and Wongg), respectively. Wonge on the other hand, retrieved 126 of 131 references with a notable reduction in the expected screening workload (NNR: 518 articles). Moreover, the Wonge filter was 5.3 times more precise (precision ratio) than the search filter Wongd. Thus, this search filter achieved the best balance between sensitivity and precision. Wong et al. [
      • Wong S.S.-L.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.
      ] found that this search filter (Wongd) has the best sensitivity when compared with their other developed two- or three-term strategies, keeping specificity ≥ 50%. The search filter Wongc, listed as the most sensitive filter by McMaster University [
      HiRU [Health Information Research Unit]
      Search Filters for Medline in Ovid Syntax and the PubMed translation. Last modified: February 9, 2016. Hamilton, CA: HiRU.
      ,
      • Wong S.S.-L.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.
      ], performed poorer in validation with GS 1 (Table 4, sensitivity of 87.79%) than with the entire GS (Table 3, sensitivity of 93.63%). Again, the MeSH-term searches yielded the best precision (0.360%), but with a low sensitivity of 47.33%.
      Table 4Validation results with gold standard 1 (131 references, left) and gold standard 2 (2,192 references, right)
      Search filterGold standard 1 (systematic reviews)Gold standard 2 (qualitative studies)
      sens (%)pre (%)NNRpre/ratiosens (%)pre (%)NNRpre/ratio
      Wonga5.340.00714,2420.271.990.611669.1
      Wongb4.580.0137,8170.454.971.218318.1
      Wongc87.790.0118,8140.393.98
      Best sensitivity.
      0.071,4971.0
      Wongd97.71
      Best sensitivity.
      0.0362,7621.091.930.195222.9
      Wonge96.18
      Best balance between sensitivity and precision (filter with highest precision among filters with sensitivity > 80%).
      0.193
      Best balance between sensitivity and precision (filter with highest precision among filters with sensitivity > 80%).
      518
      Best balance between sensitivity and precision (filter with highest precision among filters with sensitivity > 80%).
      5.3
      Best balance between sensitivity and precision (filter with highest precision among filters with sensitivity > 80%).
      61.910.9011113.5
      Wongf96.180.0991,0082.783.390.551828.2
      Wongg96.180.0362,7531.089.960.195212.9
      McKibbon4.580.0119,4150.360.171.069515.9
      Shawa70.230.0352,8691.075.680.234383.4
      Shawc94.660.0214,8710.687.640.128691.7
      UTHealth93.890.1188493.381.25
      Best balance between sensitivity and precision (filter with highest precision among filters with sensitivity > 80%).
      0.75
      Best balance between sensitivity and precision (filter with highest precision among filters with sensitivity > 80%).
      135
      Best balance between sensitivity and precision (filter with highest precision among filters with sensitivity > 80%).
      11.1
      Best balance between sensitivity and precision (filter with highest precision among filters with sensitivity > 80%).
      MeSHa47.330.360
      Best precision.
      2789.921.081.90
      Best precision.
      5328.4
      MeSHb47.330.360
      Best precision.
      2789.921.081.90
      Best precision.
      5328.4
      Abbreviations: sens, sensitivity; pre, precision; NNR, number needed to read; pre/ratio, precision ratio.
      Best sensitivity.
      Best precision.
      Best balance between sensitivity and precision (filter with highest precision among filters with sensitivity > 80%).

      4. Discussion

      4.1 Main findings

      To our knowledge, these findings provide a good overview of already available search filters for qualitative research for the bibliographic database MEDLINE. Based on the validation results, which were generated using a GS according to the relative recall approach, a search filter for MEDLINE may be selected, with respect to the individual requirements.
      The search filter Wongc (two- or three-term strategy), which is also listed on the McMaster University website, can be recommended for a very sensitive search in MEDLINE [
      HiRU [Health Information Research Unit]
      Search Filters for Medline in Ovid Syntax and the PubMed translation. Last modified: February 9, 2016. Hamilton, CA: HiRU.
      ,
      • Wong S.S.-L.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.
      ]. Although this search filter was very sensitive, not all relevant qualitative studies from the GS were detected with this filter. This is probably due to the heterogeneous reporting in qualitative studies with no or only insufficient standardized vocabulary and many different research designs, methods, and approaches [
      • Evans D.
      Database searches for qualitative research.
      ,
      • Booth A.
      Searching for qualitative research for inclusion in systematic reviews: a structured methodological review.
      ,
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ].
      Although the MeSH-term searches (exploded and unexploded) have the best precision, they cannot be recommended as the sole search string because of the very poor sensitivity of about 23%, even if the NNR promises a low screening workload. With these MeSH-term searches, only a few qualitative studies can be found without any claim to completeness (about 500 out of 2,323 from the entire GS) but probably very precise hits. Here, a consequent tagging by indexers of the national library of medicine would have been desirable. In addition, it would be beneficial if study authors used a rather standardized wording to describe their qualitative studies.
      However, with most of the search filters tested, both systematic reviews and primary qualitative studies can be retrieved with sufficiently good results. Solely search filters that contain only the single word “interview” (with different field notations) showed very poor results in terms of sensitivity when only systematic reviews were considered (gold standard 1). Here, the use of longer search strategies or of MeSH-terms has proven to be much more advantageous.
      To our knowledge, the search filter developed by the University of Texas [
      UTHealth [University of Texas Health Science Center at Houston]
      Search Filters for Various Databases: Ovid Medline. Last Updated: 28.11.2018. Houston, US-TX: UTHealth.
      ] has never been validated and the date and method of development is unknown. It considers the MeSH-term “Qualitative Research” and might fill the gap between one- to three-term strategies by Wong et al. [
      • Wong S.S.-L.
      • Wilczynski N.L.
      • Haynes R.B.
      Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.
      ] and the opposite of a very long search strategy by Shaw et al. [
      • Shaw R.L.
      • Booth A.
      • Sutton A.J.
      • Miller T.
      • Smith J.A.
      • Young B.
      • et al.
      Finding qualitative research: an evaluation of search strategies.
      ]. Our findings show that this search filter obtained a good balance between sensitivity and precision. With its sufficiently good sensitivity combined with an appropriate workload for screening, it can be recommended for systematic searches on qualitative studies.
      Our results show strong differences for the available search filters for qualitative studies in terms of sensitivity, precision, and NNR. When searching qualitative studies in MEDLINE, the selection of a suitable search filter therefore depends on the research question, the claim for completeness, and the resources available for screening. Furthermore, a systematic search should include different bibliographic databases (not only MEDLINE). For example, the bibliographic database CINAHL may contain a good collection of qualitative studies as shown for studies on dementia [
      • Rogers M.
      • Bethel A.
      • Abbott R.
      Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
      ]. Furthermore, trials, registries, and gray literature should also be included to get a comprehensive overview of a topic.
      The relative recall approach by Sampson et al. [
      • Jenkins M.
      Evaluation of methodological search filters – a review.
      ] allows the creation of a GS regardless of a medical or health-related topic. The size of the GS and the long publication period is also of particular importance to improve the estimation of the validation measurements. In addition, the GS was checked for a possible selection bias. This would be conceivable, for example, if most of the 145 systematic reviews themselves have used only one particular published search filter to identify the qualitative studies. Because this assumption did not apply, with the GS in place, a good generalizability is given.
      However, a proper calculation of precision and thus the NNR was not possible with the relative recall approach. For this, the exact number of irrelevant but found references by the search filter would have been necessary. However, this number could only be approximately determined because of a lack of manual screening of the references and thus probably led to an underestimation of the true precision or NNR.
      The validation results also benefit from the fact that all search filters validated were developed exclusively for use in Ovid MEDLINE, so that no filters had to be translated (with attendant problems of translation inaccuracies).

      4.2 Limitations

      In all, it needs to be emphasized that the calculated performance measures are only rough estimations. It cannot be ruled out that precision calculations are underestimated because the retrieved references of the individual search filters could still contain relevant qualitative articles that were not part of the entire GS. Furthermore, it is probably possible that some references of the entire GS are not qualitative research because we did not manually verify this. However, to overcome this, we additionally presented the precision ratio as a means of normalization to compare the search filters with each other.
      Likewise, it could be possible that the GS 2 is biased in selection as some review authors aimed to exclude studies of poor quality. However, the magnitude of this impact could not be entirely examined because of scarce reporting, but seems to be acceptable.
      Because the GS were generated irrespective of a subject, there is a possibility that the performance measures may differ in combination with a specific medical topic. Nevertheless, by doing so, the generalizability is increased. It cannot be ruled out, that using a GS created from publications based on systematic reviews (including systematic searches using specific words referring to qualitative research, see Supplementary Table) influences the presented validation results.
      The true informative value of the measurements in everyday use depends on the combination of the search filter with the topic-specific search terms and other limitations. For example, the true numbers of performance measures can change considerably by adding a thematic search block. To ensure that all qualitative studies will be retrieved, it should also be examined whether the use of a restrictive search filter for qualitative studies should be completely omitted.

      5. Conclusions

      To the best of our knowledge, these are validation results for a broad range of published search filters in MEDLINE using a GS irrespective of a restrictive subject following the relative recall approach by Sampson et al. [
      • Sampson M.
      • Zhang L.
      • Morrison A.
      • Barrowman N.J.
      • Clifford T.J.
      • Platt R.W.
      • et al.
      An alternative to the hand searching gold standard: validating methodological search filters using relative recall.
      ].
      In this context, the filter Wongc can be recommended as a very sensitive search filter to identify primary studies and systematic reviews of qualitative research in MEDLINE. However, this comes along with a high number of articles to be read, which would be necessary to identify the relevant ones. The search filter of the University of Texas represents an appropriate choice if a sensitivity of above 80% is desired while minimizing the workload.

      CRediT authorship contribution statement

      Mandy Wagner: Conceptualization, Methodology, Investigation, Formal analysis, Resources, Writing - review & editing. Stefanie Rosumeck: Conceptualization, Methodology, Investigation, Formal analysis, Resources, Writing - review & editing. Christian Küffmeier: Resources. Kristina Döring: Resources. Ulrike Euler: Conceptualization, Methodology, Writing - review & editing.

      Acknowledgments

      The authors gratefully acknowledge Dr. Laura Haas for proof-reading and her valuable input.
      This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

      Supplementary data

      References

        • Noyes J.
        • Popay J.
        • Pearson A.
        • Hannes K.
        • Booth A.
        Chapter 20: Qualitative research and Cochrane reviews. Version 5.1.0. Updated 20.03.2011. London: Cochrane Collaboration.
        (Available at)
        http://handbook-5-1.cochrane.org/
        Date accessed: August 26, 2019
        • Booth A.
        Chapter 3: searching for studies.
        (Version 1 (updated August 2011))in: Noyes J. Booth A. Hannes K. Harden A. Harris J. Lewin S. Supplementary Guidance for Inclusion of Qualitative Research in Cochrane Systematic Reviews of Interventions. Cochrane Collaboration Qualitative Methods Group, 2011 (Available at)
        • Evans D.
        Database searches for qualitative research.
        J Med Libr Assoc. 2002; 90: 290-293
        • Booth A.
        Searching for qualitative research for inclusion in systematic reviews: a structured methodological review.
        Syst Rev. 2016; 5: 74
        • Rogers M.
        • Bethel A.
        • Abbott R.
        Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.
        Res Synth Methods. 2017; 9: 579-586
        • Beale S.
        • Duffy S.
        • Glanville J.
        • Lefebvre C.
        • Wright D.
        • McCool R.
        • et al.
        Choosing and using methodological search filters: searchers’ views.
        Health Info Libr J. 2014; 31: 133-147
        • Jenkins M.
        Evaluation of methodological search filters – a review.
        Health Info Libr J. 2004; 21: 148-163
        • Glanville J.
        • Lefebvre C.
        • Wright K.
        ISSG search filters resource. ISSG [InterTASC Information Specialists’ Sub-Group]. York (UK).
        (Available at)
        • HiRU [Health Information Research Unit]
        Search Filters for Medline in Ovid Syntax and the PubMed translation. Last modified: February 9, 2016. Hamilton, CA: HiRU.
        (Available at)
        • McKibbon K.A.
        • Wilczynski N.L.
        • Haynes R.B.
        Developing optimal search strategies for retrieving qualitative studies in PsycINFO.
        Eval Health Prof. 2006; 29: 440-454
        • Wong S.S.-L.
        • Wilczynski N.L.
        • Haynes R.B.
        Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.
        Stud Health Technol Inform. 2004; 107: 311-316
        • NLM [U. S. National Library of Medicine]. Qualitative Research
        MeSH Descriptor Data 2019. Revision Date: 16.06.2014. Rockville Pike, US-MD: NLM.
        (Available at)
        • Wilczynski N.L.
        • Marks S.
        • Haynes R.B.
        Search strategies for identifying qualitative studies in CINAHL.
        Qual Health Res. 2007; 17: 705-710
        • DeJean D.
        • Giacomini M.
        • Simeonov D.
        • Smith A.
        Finding qualitative research evidence for health technology assessment.
        Qual Health Res. 2016; 26: 1307-1317
        • Sampson M.
        • Zhang L.
        • Morrison A.
        • Barrowman N.J.
        • Clifford T.J.
        • Platt R.W.
        • et al.
        An alternative to the hand searching gold standard: validating methodological search filters using relative recall.
        BMC Med Res Methodol. 2006; 6: 33
        • Durão S.
        • Kredo T.
        • Volmink J.
        Validation of a search strategy to identify nutrition trials in PubMed using the relative recall method.
        J Clin Epidemiol. 2015; 68: 610-616
        • Waffenschmidt S.
        • Hermanns T.
        • Gerber-Grote A.
        • Mostardt S.
        No suitable precise or optimized epidemiologic search filters were available for bibliographic databases.
        J Clin Epidemiol. 2017; 82: 112-118
        • Dalton J.
        • Booth A.
        • Noyes J.
        • Sowden A.J.
        Potential value of systematic reviews of qualitative evidence in informing user-centered health and social care: findings from a descriptive overview.
        J Clin Epidemiol. 2017; 88: 37-46
      1. Centre for Reviews and Dissemination. York (UK): University of York.
        (Available at)
        • Torquato Lopes A.P.A.
        • das Neves Decesaro M.
        The adjustments experienced by persons with an ostomy: an integrative review of the literature.
        Ostomy Wound Manage. 2014; 60: 34-42
        • UTHealth [University of Texas Health Science Center at Houston]
        Search Filters for Various Databases: Ovid Medline. Last Updated: 28.11.2018. Houston, US-TX: UTHealth.
        (Available at)
        • Shaw R.L.
        • Booth A.
        • Sutton A.J.
        • Miller T.
        • Smith J.A.
        • Young B.
        • et al.
        Finding qualitative research: an evaluation of search strategies.
        BMC Med Res Methodol. 2004; 4: 5