Abstract
Objectives
Search filters can support qualitative evidence of information retrieval. Various search filters are available for the bibliographic databases PsycINFO and CINAHL. To date, no comparative overview of validation results of search filters verified with an independent gold standard exists.
Study Design and Setting
Identified search filters for PsycINFO and CINAHL were tested for plausibility. Gold standards were generated according to the relative recall approach using references included in an overview of systematic reviews of qualitative studies. All included references were collected and checked for indexing in PsycINFO and CINAHL. Validation tests for each search filter were conducted in both databases to determine whether the references of the gold standards could be retrieved or not.
Results
Twelve search filters for PsycINFO and fifteen for CINAHL were validated. The complexity and design of these search filters vary, as well as the validation results for the databases. When locating primary studies of qualitative research, the best sensitivity and precision ratio (among filters with a sensitivity of >80%) was achieved with a filter by McKibbon et al. for PsycINFO and a filter by Wilczynski et al. for CINAHL.
Conclusion
Project-specific requirements and resources influence the choice of a specific search filter for PsycINFO and CINAHL.
Keywords
What is new?
- •Search filters for qualitative research in PsycINFO and CINAHL differ greatly in design and performance.
- •It is not possible to suggest a single search filter to identify qualitative research in these databases.
Key findings
- •First comparative overview of validation measurements of search filters for PsycINFO and CINAHL using a gold standard irrespective of a medical or health-related topic and with literature published in several years.
What this adds to what is known?
- •Project-specific requirements and resources drive the decision for or against a search filter, and for which one.
- •To increase the likelihood of identifying all relevant articles, several bibliographic databases need to be searched, especially in the field of qualitative research.
What is the implication, what should change now?
1. Introduction
The implication of qualitative research in evidence-based health care has increased. Patients’ preferences and experiences need to be considered in high-quality systematic reviews and clinical practice guidelines [
1
, 2
, 3
].AGREE Next Steps Consortium
Appraisal of guidelines for research & evaluation II. AGREE II Instrument. AGREE Research Trust.
Appraisal of guidelines for research & evaluation II. AGREE II Instrument. AGREE Research Trust.
https://www.agreetrust.org/wp-content/uploads/2017/12/AGREE-II-Users-Manual-and-23-item-Instrument-2009-Update-2017.pdf
Date: 2017
Date accessed: April 9, 2020
Various research methods and approaches are used in qualitative research. However, most research papers are less structured and not standardized in their vocabulary. That makes relevant qualitative evidence information retrieval difficult and complex [
[1]
,[4]
,[5]
]. Lefebvre et al. [[6]
] asked information professionals to name “Areas where filters are needed/existing filters need to be improved” (p. 70). Interviewees mentioned, among other things, qualitative research filters.In recent years, search filters for identifying qualitative research in PsycINFO and CINAHL have been published but not adequately validated with an independent set of references (gold standard, GS).
The first aim of this paper is to provide a comprehensive overview of published filters for locating qualitative research in bibliographic databases PsycINFO and CINAHL. Second, this study compares the validation performances of search filters. Validations were conducted using a newly generated GS irrespective of a medical or health-related topic and with references published over a longer period of time. Given these results, it is possible to determine the search filter with the best sensitivity, best precision ratio, and/or best balance of sensitivity and precision.
The aims and methods of this paper are in accordance with a previous work by Wagner et al. [
[7]
] focusing on MEDLINE.2. Methods
Similar to the aforementioned publication [
[7]
], a four-step approach was used: 1. identification of search filters to locate qualitative research in the bibliographic databases PsycINFO and CINAHL, 2. verification and plausibility checks of identified search filters, 3. generation of a GS irrespective of a medical or health-related topic, 4. validation of identified search filters against the newly created GS in PsycINFO and CINAHL.2.1 Identification of search filters
For identifying published search filters for qualitative research a comprehensive literature search was carried out in PubMed. Additionally, other well-known sources like the Cochrane Handbook for Systematic Reviews of Interventions [
[8]
], and websites, such as those of the InterTASC Information Specialists’ Subgroup (ISSG) [[9]
] and the McMaster University’s Department of Health Information Research Unit [Glanville J, Lefebvre C, Wright K, editors. Filters to identify qualitative research. Update date: 4 February 2020. York, GB: ISSG [InterTASC Information Specialists' Sub-Group]. Available at https://sites.google.com/a/york.ac.uk/issg-search-filters-resource/filters-to-identify-qualitative-research. Accessed April 9, 2020.
[10]
], were checked. Published search filters with complete search strategy, irrespective of their publication date, were eligible.HiRU [Health Information Research Unit]
Health information research unit. Our mission... Last modified February 9, 2016. Hamilton, CA-ON: HiRU.
Health information research unit. Our mission... Last modified February 9, 2016. Hamilton, CA-ON: HiRU.
https://hiru.mcmaster.ca/hiru/Default.aspx
Date accessed: April 9, 2020
2.2 Verification and plausibility checks of search filters
All filters created to run in the bibliographic databases PsycINFO or CINAHL, irrespective of the host platform, were taken into account. Newly developed filters, as well as adaptations of already published search filters, were included and considered for validation.
Search strategies must be plausible and reproducible in terms of database syntax, e.g., to just list a number of keywords and state that these were searched with truncation is insufficient. Hence, all identified search filters were checked. In case of considerable syntax errors or content inconsistencies (e.g., including search terms that restrict the search to a thematic topic), the respective filter was excluded from the validation process. For reasons of feasibility, developers of search filters and authors were not contacted.
2.3 Generation of the gold standard
Different methods to generate a GS were reported, e.g., topic-related hand searching of specific journals [
11
, 12
, 13
] or combining all studies included in various systematic reviews of a topic (relative recall approach) [14
, 15
, 16
]. In comparison to generating a GS with relevant articles published over a long period of time, the relative recall approach is more time saving and was already successfully applied in a previous project [[7]
]. Thus, this approach was used to generate the GS for PsycINFO and CINAHL.Dalton et al. [
[17]
] aimed at providing an overview of systematic reviews of qualitative research indexed in the Database of Abstracts of Reviews of Effects (DARE) from 2009 to 2014 regardless of the medical or health-related topic. Included systematic reviews were used as a starting point for the new GS. This can be considered as comprehensive, as DARE itself incorporates hits from MEDLINE, Embase, PsycINFO, PubMed and CINAHL. For more details on systematic literature searches used in the systematic reviews, see Supplemental File.First, all 145 systematic reviews included by Dalton et al. [
[17]
] were obtained in full-text. Second, all primary studies (journal articles or dissertations) included in these systematic reviews were obtained and managed in EndNote. We assumed that all included articles addressed qualitative research, so we did not investigate whether each study applied qualitative methods. In a final step, duplicates were removed, and only articles indexed in PsycINFO or CINAHL were added to the respective database-specific GS.2.4 Validation of search filters
The validation of search filters followed a method described elsewhere [
[7]
]. For each reference of the GS, it was determined whether it could be located with the particular search filter in PsycINFO or CINAHL. Performance measurements, such as sensitivity, precision, and number needed to read (NNR), were determined based on the number of hits revealed (Table 1). For overcoming the disadvantage that the relative recall approach does not allow for an estimation of the true precision, a precision ratio was calculated based on Waffenschmidt et al. [[16]
]. Accordingly, the precision ratio was set to 1 for the search filter with the highest sensitivity. Precision ratios for the other filters were then calculated by the precision of each filter divided by the precision of the most sensitive filter. This allows a better interpretation of validation results and enhances the comparability of the performance of the investigated filters. For example, irrespective of specificity, a precision ratio of 2 indicates that a filter is twice as precise and would require half the number of records to be screened to find one qualitative article, compared to the search filter with the highest sensitivity.Table 1Calculations
| Definitions | a - relevant articles of the gold standard retrieved with search filter |
|---|---|
| b - irrelevant articles retrieved with search filter (hits in the database for each search filter minus references of the gold standard) | |
| c - relevant articles of the gold standard not retrieved with the search filter | |
| Sensitivity (%) | a/(a+c)∗100 |
| Precision (%) | a/(a+b)∗100 |
| NNR | (a+b)/a |
| Precision rate | Precision/precision of the search filter with the highest sensitivity |
Abbreviation: NNR, number needed to read.
# Could be articles relating to qualitative research and relevant but are not part of the gold standard.
3. Results
3.1 Identification of search filters
A systematic search in PubMed for literature on qualitative research search filters (Box 1) revealed 377 hits. One person screened all titles and abstracts. Then, the full-texts of eight articles were obtained and screened by one person [
[4]
,[6]
,[11]
,[12]
,18
, 19
, 20
, 21
]. Five articles presented search strategies for locating qualitative research in PsycINFO [[4]
,[11]
] and CINAHL [[4]
,[12]
,[18]
,[21]
].Box 1
Search strategy (PubMed), search date: December, 10th 2019
Tabled
1
| (Information Storage and Retrieval[Mesh:NoExp] OR Databases as Topic[Mesh:NoExp] OR Databases, Bibliographic[Mesh:NoExp] OR PsycINFO[tiab] OR PsychINFO[tiab] OR PsyINFO[tiab] OR CINAHL[tiab] OR CINHAL[tiab]) AND (Qualitative Research[Mesh] OR qualitative[tiab]) AND (“search filter”[tiab] OR “search filters”[tiab] OR “search strategy”[tiab] OR “search strategies”[tiab] OR hedge[tiab] OR hedges[tiab]) |
Scanning the chapter on searching for qualitative evidence of the Cochrane Handbook of Systematic Reviews of Intervention [
[8]
], as well as the websites of the ISSG [[9]
] and of the McMaster University [Glanville J, Lefebvre C, Wright K, editors. Filters to identify qualitative research. Update date: 4 February 2020. York, GB: ISSG [InterTASC Information Specialists' Sub-Group]. Available at https://sites.google.com/a/york.ac.uk/issg-search-filters-resource/filters-to-identify-qualitative-research. Accessed April 9, 2020.
[22]
], yielded one additional search strategy for locating qualitative research in PsycINFO [HiRU [Health Information Research Unit]
Search strategies for PsycINFO in Ovid syntax. Last modified February 9, 2016. Hamilton, CA-ON: HiRU.
Search strategies for PsycINFO in Ovid syntax. Last modified February 9, 2016. Hamilton, CA-ON: HiRU.
https://hiru.mcmaster.ca/hiru/HIRU_Hedges_PsycINFO_Strategies.aspx#Qualitative
Date accessed: April 9, 2020
[23]
] and CINAHL [UTHealth [University of Texas Health Science Center at Houston]
Search filters for various databases: Ovid PsycINFO. Last Updated: Sep 26, 2019. Houston, US-TX: UTHealth.
Search filters for various databases: Ovid PsycINFO. Last Updated: Sep 26, 2019. Houston, US-TX: UTHealth.
http://libguides.sph.uth.tmc.edu/search_filters/ovid_psycinfo_filters
Date accessed: April 9, 2020
[24]
], respectively.3.1.1 PsycINFO
In total, 14 different strategies for locating qualitative research in PsycINFO were identified [
[4]
,[11]
,[23]
], including two index term searches.UTHealth [University of Texas Health Science Center at Houston]
Search filters for various databases: Ovid PsycINFO. Last Updated: Sep 26, 2019. Houston, US-TX: UTHealth.
Search filters for various databases: Ovid PsycINFO. Last Updated: Sep 26, 2019. Houston, US-TX: UTHealth.
http://libguides.sph.uth.tmc.edu/search_filters/ovid_psycinfo_filters
Date accessed: April 9, 2020
McKibbon et al. [
[11]
] (McMaster University Hedges Team) developed and evaluated six different filters, including single-term and multiple-term strategies. Different variations of the keywords interview, qualitative, themes, and experiences were tested. Rogers et al. [[4]
] evaluated five filters consisting of subject headings, simple free-text terms, complex free-text terms, and two broad-based strategies. All these strategies are adaptations of search filters published before. Another nonvalidated search filter, which incorporates the index term “Qualitative Research”, was identified on the website of the University of Texas – School of Public Health (UTHealth) [[23]
]. The index term was introduced in 2003 and changed to “Qualitative Methods” in 2019 [UTHealth [University of Texas Health Science Center at Houston]
Search filters for various databases: Ovid PsycINFO. Last Updated: Sep 26, 2019. Houston, US-TX: UTHealth.
Search filters for various databases: Ovid PsycINFO. Last Updated: Sep 26, 2019. Houston, US-TX: UTHealth.
http://libguides.sph.uth.tmc.edu/search_filters/ovid_psycinfo_filters
Date accessed: April 9, 2020
[25]
]. Exploded and unexploded searches of this term were considered.APA [American Psychological Association]
Thesaurus of psychological index terms 2019 Update.
Thesaurus of psychological index terms 2019 Update.
APA,
Washington, US-DC2019
https://www.apa.org/pubs/databases/training/2019-thesaurus-update.pdf
Date accessed: April 9, 2020
3.1.2 CINAHL
In total, 19 different strategies for locating qualitative research in CINAHL were identified [
[4]
,[12]
,[18]
,[21]
,[24]
], including two subject heading searches.Shaw et al. [
[21]
] evaluated a thesaurus term, a free-text term, and a broad-based terms strategy. Rogers et al. [[4]
] compared five different strategies, and four of them were eligible for validation: subject headings, simple free-text terms, broad-based terms, and a complex free-text strategy. The Hedges Team of McMaster University published eight search filters for CINAHL via Ovid (different search terms and combinations) [[12]
]. Several years ago, the Edward G. Miner Library at the University of Rochester Medical Center published various search filters for CINAHL via Ovid. One of them focused on locating qualitative research [[24]
]. Although it is marked as “to be used for historical purposes only”, it is cited on the ISSG website. Therefore, we decided to consider this comprehensive search filter for validation. In addition, a hybrid filter by DeJean et al. [[18]
] was identified, which just combined existing search filters. Of the controlled vocabulary in CINAHL, the subject heading “Qualitative Studies” was considered as an exploded and unexploded search string.3.2 Verification and plausibility checks of search filters for qualitative research
3.2.1 PsycINFO
Two (UTHealth [
[23]
], Rogerse [UTHealth [University of Texas Health Science Center at Houston]
Search filters for various databases: Ovid PsycINFO. Last Updated: Sep 26, 2019. Houston, US-TX: UTHealth.
Search filters for various databases: Ovid PsycINFO. Last Updated: Sep 26, 2019. Houston, US-TX: UTHealth.
http://libguides.sph.uth.tmc.edu/search_filters/ovid_psycinfo_filters
Date accessed: April 9, 2020
[4]
]) of the 14 identified search filters failed the test for plausibility and replicability and were excluded from validation (Table 2).Table 2Search strategies of identified search filters for PsycINFO via Ovid
| Search filter | Labeled by the filter author(s) | Search strategy for PsycINFO (via Ovid) |
|---|---|---|
| Index Terma | - | Qualitative Methods/ |
| Index Termb | - | exp Qualitative Methods/ |
| McKibbona [ [11] ] | Single term – best sensitivity and single term - best optimization of sensitivity & specificity | interview.tw. |
| McKibbonb [ [11] ] | Single term – best specificity | interviews.tw. |
| McKibbonc [ [11] ] | Combination of terms – best sensitivity | experience∗.mp. OR interview∗.tw. OR qualitative∗.tw. |
| McKibbond [ [11] ] | Combination of terms – best specificity | qualitative∗.tw. OR themes.tw. |
| McKibbone [ [11] ] | Combination of terms – small drop in specificity with a substantive gain in sensitivity | interviews.tw. OR qualitative∗.tw. |
| McKibbonf [ [11] ] | Combination of terms – best optimization of sensitivity & specificity | experiences.tw. OR interview∗.tw. OR qualitative.tw. |
| Rogersa [ [4] ] | Subject headings terms | Qualitative Research/ OR exp Questionnaires/ OR Interviewing/ OR Attitudes/ OR Ethnology/ OR Phenomenology/ OR Observation Methods/ OR Discourse Analysis/ OR Content Analysis/ |
| Rogersb [ [4] ] | Simple free-text terms | (((“semi-structured” OR semistructured OR unstructured OR informal OR “in-depth” OR indepth OR “face-to-face” OR structured OR guide) ADJ3 (interview∗ OR discussion∗ OR questionnaire∗)) OR (focus group∗ OR qualitative OR ethnograph∗ OR fieldwork OR “field work” OR “key informant”)).ti,ab. |
| Rogersc [ [4] ] | Broad terms 1 (based on Wong [ [13] ]) | (interview∗ OR experience∗).af. OR qualitative.tw. |
| Rogersd [ [4] ] | Broad terms 2 (based on Shaw [ [26] ])
Additional File 2 – Qualitative methodology search strategies. Finding qualitative research: an evaluation of search strategies. https://static-content.springer.com/esm/art%3A10.1186%2F1471-2288-4-5/MediaObjects/12874_2003_63_MOESM2_ESM.doc Date accessed: April 9, 2020 | (findings OR interview∗ OR qualitative).af. |
| Rogerse [ [27] ] | Complex free-text terms | excluded from validation |
| UTHealth [ [23] ]UTHealth [University of Texas Health Science Center at Houston] Search filters for various databases: Ovid PsycINFO. Last Updated: Sep 26, 2019. Houston, US-TX: UTHealth. http://libguides.sph.uth.tmc.edu/search_filters/ovid_psycinfo_filters Date accessed: April 9, 2020 | Qualitative studies | excluded from validation |
# Please refer to database guide for more details on search syntax.
The filter UTHealth consisting of free-text terms and controlled vocabulary is not executable. Due to syntax errors regarding the placement of parenthesis, this search filter could not be compiled and validated. Rogerse was adapted from Shaw’s “free-text terms’ filter, which was probably developed for MEDLINE via Ovid (according to displayed syntax), to run in PsycINFO via Ovid. Shaw et al. [
[21]
] developed it primarily to locate qualitative research on breastfeeding support. Therefore, it includes search terms like “women’s stor$” or “feminis$”. Regarding Rogerse, some inconsistencies, such as missing field notations, were corrected by the authors, but some weaknesses remained, so it was excluded, too.For replicating the “subject heading terms” search by Rogers et al. [
[4]
] (Rogersa), all reported terms were combined with the Boolean operator “OR” and the obviously missing slash for the last subject heading was added. All terms of the broad terms search (Rogersd, based on Shaw et al. [[21]
]) were connected with Boolean “OR” to be executable.Altogether, twelve filters by McKibbon et al. [
[11]
] (McKibbona-f) and Rogers et al. [[4]
] (Rogersa-d), as well as the index term searches (Index Terma,b) were included for validation. Three of these filters comprise solely controlled vocabulary (Index Terma,b, Rogersa). Six filters consist of free-text term searches with one or multiple words in a specific text field (McKibbona,b,d-f, Rogersb). The other three filters combine both text field searches and controlled vocabulary (McKibbonc, Rogersc,d).3.2.2 CINAHL
Four (DeJean [
[18]
], Rogersb [[4]
], Rogersd [[4]
], Shawb [[21]
]) out of nineteen identified search filters failed the test for plausibility and replicability. Those filters were excluded from validation (see Supplemental File).One of the search filters by Shaw et al. (Shawb [
[21]
]) contains syntax inconsistencies (e.g., missing field notation, incorrect parentheses) and content-related discrepancies or redundant search lines. Rogers et al. [[4]
] based their filter Rogersd on Shawb, but did not revise the aforementioned weaknesses. The hybrid filter by DeJean et al. [[18]
] connecting multiple filters contains this free-text term filter (Shawb) as well. The mentioned errors were not corrected, either.The “simple free-text terms” filter Rogersb [
[4]
] seems to contain obvious documentation errors. Keywords like qualitative or fieldwork, which were used for searches in other databases, are missing in the reported search strategy for CINAHL. It is unclear how the authors intended to include these terms.For replicating the search filters by Shaw et al. [
[21]
] and Wilczynski et al. [[12]
], all searches were translated from Ovid to EBSCO (see Supplemental File).Altogether, the 15 filters Nesbit [
[24]
], Rogersa,c [[4]
], Shawa,c [[21]
], Wilczynskia-h [[12]
], as well as the heading searches (Headinga,b) were included for validation. Ten of these filters consist solely of controlled vocabulary (Headinga,b, Rogersa, Shawa, Wilczynskia-e,g). The other five filters combine text field searches and controlled vocabulary (Nesbit, Rogersc, Shawc, Wilczynskif,h). None of these filters incorporates exclusively one or multiple words in a specific text field.3.3 Generation of the gold standard
The overview of systematic reviews by Dalton et al. [
[17]
] identified 145 systematic reviews that included 3,012 primary studies. All publications were managed in EndNote. Studies whose study design is described by the review authors as quantitative and publications not indexed in bibliographic databases (e.g., gray literature) were excluded, and the duplicates were removed. This resulted in 2,715 primary studies.Nearly half of the primary studies were indexed in PsycINFO (n = 1,315, 48.4%). Included articles were published between 1969 and 2014. Whereas for CINAHL, 2,092 primary studies (77.1%) were indexed, published between 1981 and 2014.
3.4 Validation of search filters
3.4.1 PsycINFO
Validating search filters with the GS showed that Rogersc [
[4]
], which combines text field searches with controlled vocabulary, had the best sensitivity (95.44%, Table 3). It detected 1,255 of 1,315 references. The best precision ratio (4.73) was achieved with Rogersb, which searches the fields title and abstract with multiple words. However, sensitivity was comparatively low (72.70%). Among filters with a sensitivity >80%, McKibbonf demonstrated the best balance of sensitivity and precision. Thus, it was 2.38 times more precise than the very sensitive search filter Rogersc (Table 3). More details are presented in Supplemental File.Table 3Validation results for PsycINFO (1,315 qualitative studies)
| Search filter | Sensitivity (%) | Precision ratio |
|---|---|---|
| Index Terma | 0.61 | 1.28 |
| Index Termb | 2.36 | 3.00 |
| McKibbona [ [11] ] | 15.13 | 2.04 |
| McKibbonb [ [11] ] | 51.56 | 4.67 |
| McKibbonc [ [11] ] | 89.89 | 1.48 |
| McKibbond [ [11] ] | 59.32 | 4.60 |
| McKibbone [ [11] ] | 72.17 | 4.16 |
| McKibbonf [ [11] ] | 85.93 | 2.38 |
| Rogersa [ [4] ] | 4.03 | 0.64 |
| Rogersb [ [4] ] | 72.70 | 4.73 |
| Rogersc [ [4] ] | 95.44 | 1.00 |
| Rogersd [ [4] ] | 94.68 | 1.02 |
∗ Best sensitivity (precision ratio set to 1.00).
# Best balance of sensitivity and precision (filter with highest precision among filters with sensitivity > 80%).
When considering validation results for GS separately for systematic reviews and in the combination of primary studies and systematic reviews, a different trend was observable (see Supplemental File for details).
3.4.2 CINAHL
The filter Rogersa reached the best sensitivity (94.55%, Table 4). It located 1.978 of 2.092 articles. The best precision ratio (7.03) was achieved with Wilczynskib. It exclusively searches the term Audiorecording as the exact subject heading, resulting in a low sensitivity (35.66%). Among those filters achieving a sensitivity >80%, Wilczynskih showed the best balance of sensitivity and precision. Compared to Rogersa, the most sensitive filter, it was 3.59 times more precise (precision ratio, Table 4). More details are presented in Supplemental File.
Table 4Validation results for CINAHL (2,092 qualitative studies)
| Search filter | Sensitivity (%) | Precision ratio |
|---|---|---|
| Headinga | 54.40 | 5.35 |
| Headingb | 66.63 | 4.78 |
| Nesbit [ [24] ] | 88.86 | 3.21 |
| Rogersa [ [4] ] | 94.55 | 1.00 |
| Rogersc [ [4] ] | 92.11 | 1.37 |
| Shawa [ [21] ] | 90.58 | 1.02 |
| Shawc [ [21] ] | 92.45 | 1.24 |
| Wilczynskia [ [12] ] | 71.70 | 3.06 |
| Wilczynskib [ [12] ] | 35.66 | 7.03 |
| Wilczynskic [ [12] ] | 72.66 | 0.56 |
| Wilczynskid [ [12] ] | 93.12 | 0.56 |
| Wilczynskie [ [12] ] | 91.06 | 1.45 |
| Wilczynskif [ [12] ] | 43.16 | 6.23 |
| Wilczynskig [ [12] ] | 51.82 | 5.97 |
| Wilczynskih [ [12] ] | 83.56 | 3.59 |
∗ Best sensitivity (precision ratio set to 1.00).
# Best balance of sensitivity and precision (filter with highest precision among filters with sensitivity > 80%).
Performance measures for GS separated for systematic reviews and in the combination of primary studies and systematic reviews are presented in Supplemental File, too.
4. Discussion
4.1 Main findings
This is a comprehensive overview of published search filters for locating qualitative research in the bibliographic databases PsycINFO and CINAHL. Validation results, which were obtained using a newly created GS irrespective of a medical or health-related topic, are presented. Based on our findings, the selection of a search filter depends on the specific research question and individual requirements.
As already known and described elsewhere [
[1]
,[4]
,[5]
], reporting on qualitative research is heterogeneous. It lacks a standardized vocabulary, which may be due to the numerous research designs, methods, and approaches utilized.Taking into account that it is difficult to identify all relevant articles, McKibbonf can be recommended for identifying qualitative primary studies in PsycINFO. This finding corresponds with McMaster University, suggesting it as the filter with the “Best balance of sensitivity and specificity” [
[22]
]. This could be beneficial to get a comprehensive overview of qualitative research on a specific topic. In contrast, e.g., when searching qualitative research for conducting a systematic review, one would prefer the most sensitive filter Rogersc. This filter is also based on McKibbon’s three-term strategy. However, Rogers and colleagues [HiRU [Health Information Research Unit]
Search strategies for PsycINFO in Ovid syntax. Last modified February 9, 2016. Hamilton, CA-ON: HiRU.
Search strategies for PsycINFO in Ovid syntax. Last modified February 9, 2016. Hamilton, CA-ON: HiRU.
https://hiru.mcmaster.ca/hiru/HIRU_Hedges_PsycINFO_Strategies.aspx#Qualitative
Date accessed: April 9, 2020
[4]
] extended it by using truncation and searching more fields (all searchable fields [af] instead of text word [tw], see Table 2).Wilczynskih demonstrated the best balance in terms of locating qualitative primary studies in CINAHL. This filter combines free-text searches in the title and abstract fields with two subject headings, including the heading Audiorecording used in Wilczynskib. Instead, when preparing a systematic review with the aim to find all relevant qualitative studies, one would choose the most sensitive filter Rogersa. Notably, this filter solely consists of controlled vocabulary.
Although, employing controlled vocabulary yields (very) good precision in both databases, it is not advisable to use it as a sole search string due to very poor sensitivity, even if the NNR implies to be low.
Besides strong variances in performance measures, we have demonstrated that qualitative research could be located with sufficiently good results in both databases. Regarding PsycINFO, the best-balanced filter consists of only three free-text terms, whereas for CINAHL, a somewhat more complex search strategy, including free-text terms, as well as controlled vocabulary, demonstrated to be best balanced. Both filters were already developed in 2006 and 2007, respectively. Although the age of the filters seems not to be relevant, it may be worth investigating further changes in terminology and controlled vocabulary.
4.2 Limitations
It needs to be stated that the presented precision calculations are rough estimates. Foremost, it cannot be negated that they are underestimated, i.e., the true precision is likely to be higher. References retrieved by search filters may contain qualitative research that was not part of our GS. As a result, using the relative recall approach prevents a proper calculation of the NNR, as this would require the exact number of irrelevant but found articles by the filters. Thus, the NNR may be underestimated, i.e., the presented NNR may be lower than it actually is. To compensate for this and to allow the comparison of the validated search filters, we calculated the precision ratio as a means of normalization. However, when interpreting the results, one needs to keep in mind that performance measures may differ if search filters will be added to a thematic search block (i.e., topic search terms).
Likewise, the size of the GS for PsycINFO turned out to be small, because only half of the references included in the systematic reviews encompassed in Dalton et al. [
[17]
] were indexed. Also, a selection bias may have been introduced. Few review authors used a published search filter for locating qualitative studies (see Supplemental File), which may have led to an overestimation of validation results. Additionally, some authors aimed to exclude studies of poor quality. Its potential impact could not be investigated due to scarce reporting, but it seems to be negligible. Moreover, we did not manually check the individual references of the GS in terms of their correct classification as qualitative research.Eleven search filters for CINAHL were originally designed for Ovid. We translated these strategies to run in EBSCO. While we cannot preclude that this may have an impact on the behavior of the filter originally intended by the developer, all adaptations were minor in complexity (see Supplemental File for more details).
Due to feasibility, we did not contact authors to resolve discrepancies or ambiguities. Therefore respective search filters were excluded from validation, e.g., obviously incomplete search strategy (Rogersb).
To decide on whether and when a search filter for qualitative research should be applied or not strongly depends on the project-specific requirements and circumstances. For example, in some cases, there is no need to use a filter when the number of records retrieved is low due to the inclusion of a thematic block. This increases the sensitivity and concurrently shifts the workload of filtering qualitative articles to the screening process.
5. Conclusions
This is the first publication that presents validation results for a broad range of published search filters for locating qualitative research in PsycINFO and CINAHL [
[4]
,[11]
,[12]
,[21]
,[24]
]. The GS for these two bibliographic databases were created using the relative recall approach irrespective of a medical or health-related topic.If a sensitivity above 80% is wanted, McKibbonf can be recommended as the best-balanced filter to be used in PsycINFO. Regarding CINAHL, Wilczynskih meets these requirements. However, if completeness is requested, Rogersc and Rogersa seem to be the appropriate choices for PsycINFO and CINAHL, respectively. But, of course, none of these filters can achieve completeness.
CRediT authorship contribution statement
Stefanie Rosumeck: Conceptualization, Methodology, Investigation, Formal analysis, Resources, Writing - review & editing. Mandy Wagner: Conceptualization, Methodology, Formal analysis, Resources, Writing - review & editing. Simon Wallraf: Investigation, Formal analysis, Writing - review & editing. Ulrike Euler: Writing - review & editing.
Acknowledgments
The authors kindly thank Kristina Döring and Christian Küffmeier for their assistance in the generation of the gold standards and gratefully acknowledge Dr. Laura Haas for proof-reading.
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Supplementary data
- Supplemental File
References
- Searching for qualitative research for inclusion in systematic reviews: a structured methodological review.Syst Rev. 2016; 5
- Qualitative evidence. Chapter 21.in: Higgins J.P.T. Thomas J. Chandler J. Cumpston M. Li T. Page M.J. Cochrane Handbook for Systematic Reviews of Interventions (Version 6). Wiley-Blackwell, Hoboken, US-NJ2019: 525-545
- Appraisal of guidelines for research & evaluation II. AGREE II Instrument. AGREE Research Trust.(Available at)https://www.agreetrust.org/wp-content/uploads/2017/12/AGREE-II-Users-Manual-and-23-item-Instrument-2009-Update-2017.pdfDate: 2017Date accessed: April 9, 2020
- Locating qualitative studies in dementia on MEDLINE, EMBASE, CINAHL, and PsycINFO: a comparison of search strategies.Res Synth Methods. 2018; 9: 579-586
- Database searches for qualitative research.J Med Libr Assoc. 2002; 90: 290-293
- Assessing the performance of methodological search filters to improve the efficiency of evidence information retrieval: five literature reviews and a qualitative study.Health Technol Assess. 2017; 21
- A validation study revealed differences in design and performance of MEDLINE search filters for qualitative research.J Clin Epidemiol. 2020; 120: 17-24
- Qualitative evidence: searching for qualitative evidence. Chapter 21, section 21.7.in: Higgins J.P.T. Thomas J. Chandler J. Cumpston M. Li T. Page M.J. Cochrane Handbook for Systematic Reviews of Interventions (Version 6). Wiley-Blackwell, Hoboken, US-NJ2019: 531-532
Glanville J, Lefebvre C, Wright K, editors. Filters to identify qualitative research. Update date: 4 February 2020. York, GB: ISSG [InterTASC Information Specialists' Sub-Group]. Available at https://sites.google.com/a/york.ac.uk/issg-search-filters-resource/filters-to-identify-qualitative-research. Accessed April 9, 2020.
- Health information research unit. Our mission... Last modified February 9, 2016. Hamilton, CA-ON: HiRU.(Available at)https://hiru.mcmaster.ca/hiru/Default.aspxDate accessed: April 9, 2020
- Developing optimal search strategies for retrieving qualitative studies in PsycINFO.Eval Health Prof. 2006; 29: 440-454
- Search strategies for identifying qualitative studies in CINAHL.Qual Health Res. 2007; 17: 705-710
- Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.Stud Health Technol Inform. 2004; 107: 311-316
- Validation of a search strategy to identify nutrition trials in PubMed using the relative recall method.J Clin Epidemiol. 2015; 68: 610-616
- An alternative to the hand searching gold standard: validating methodological search filters using relative recall.BMC Med Res Methodol. 2006; 6: 33
- No suitable precise or optimized epidemiologic search filters were available for bibliographic databases.J Clin Epidemiol. 2017; 82: 112-118
- Potential value of systematic reviews of qualitative evidence in informing user-centered health and social care: findings from a descriptive overview.J Clin Epidemiol. 2017; 88: 37-46
- Finding qualitative research evidence for health technology assessment.Qual Health Res. 2016; 26: 1307-1317
- Electronic searching to locate qualitative research: evaluation of three strategies.J Adv Nurs. 2007; 57: 95-100
- Evaluation of five search strategies in retrieving qualitative patient-reported electronic data on the impact of pressure ulcers on quality of life.J Adv Nurs. 2010; 66: 645-652
- Finding qualitative research: an evaluation of search strategies.BMC Med Res Methodol. 2004; 4: 5
- Search strategies for PsycINFO in Ovid syntax. Last modified February 9, 2016. Hamilton, CA-ON: HiRU.(Available at)https://hiru.mcmaster.ca/hiru/HIRU_Hedges_PsycINFO_Strategies.aspx#QualitativeDate accessed: April 9, 2020
- Search filters for various databases: Ovid PsycINFO. Last Updated: Sep 26, 2019. Houston, US-TX: UTHealth.(Available at)http://libguides.sph.uth.tmc.edu/search_filters/ovid_psycinfo_filtersDate accessed: April 9, 2020
- Evidence-based filters for Ovid CINAHL. Date Last Updated 2012-09-26. University of Rochester.(Available at)
- Thesaurus of psychological index terms 2019 Update.APA, Washington, US-DC2019 (Available at)https://www.apa.org/pubs/databases/training/2019-thesaurus-update.pdfDate accessed: April 9, 2020
- Additional File 2 – Qualitative methodology search strategies. Finding qualitative research: an evaluation of search strategies.(Available at)https://static-content.springer.com/esm/art%3A10.1186%2F1471-2288-4-5/MediaObjects/12874_2003_63_MOESM2_ESM.docDate accessed: April 9, 2020
- Data S1. Supporting Information.
Article info
Publication history
Published online: September 25, 2020
Accepted:
September 3,
2020
Footnotes
Declarations of interest: None.
Funding: This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Author statement: 1) conceived and designed the experiments: MW, SR. 2) performed the experiments: SR, SW. 3) analyzed and interpreted the data: SR, MW, SW. 4) contributed reagents, materials, analysis tools, or data: MW, SR. 5) wrote the paper: SR, MW, SW, UE.
Identification
Copyright
© 2020 The Authors. Published by Elsevier Inc.
User license
Creative Commons Attribution – NonCommercial – NoDerivs (CC BY-NC-ND 4.0) | How you can reuse 
Elsevier's open access license policy
Creative Commons Attribution – NonCommercial – NoDerivs (CC BY-NC-ND 4.0)
Permitted
For non-commercial purposes:
- Read, print & download
- Redistribute or republish the final article
- Text & data mine
- Translate the article (private use only, not for distribution)
- Reuse portions or extracts from the article in other works
Not Permitted
- Sell or re-use for commercial purposes
- Distribute translations or adaptations of the article
Elsevier's open access license policy
