If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Corresponding author: Institute for Quality Assurance and Transparency in Healthcare (IQTIG), Katharina-Heinroth-Ufer 1, 10787 Berlin, Germany. Tel.: +49-30-585826-0; fax: +49-30-5858526-437.
Several search filters exist to identify qualitative research, but so far none of them has been validated with an independent set of relevant references irrespective of a medical topic. The objective of this study was to provide a comparative overview of validation results for various MEDLINE search filters.
Study Design and Setting
Search filters were tested for plausibility. A relative recall approach was used to generate a gold standard based on an overview of systematic reviews of qualitative studies. For each review, the included qualitative studies were collected and checked for MEDLINE-indexing. The body of indexed articles yielded the gold standard. Validation tests were conducted to determine whether the references of the gold standard could be identified with the respective search filters.
Results
Thirteen search filters were validated in MEDLINE. One search filter by Wong et al. (2004) was found to be the most sensitive (93.63%). While medical subject heading “qualitative research” achieved the best precision (2.15%), sensitivity was the lowest (22.56%). University of Texas provided the best balanced search filter with a sensitivity of 81.96% and a precision of 0.80%.
Conclusion
Search filters to identify qualitative research in MEDLINE differ greatly in design and performance. The selection of the appropriate search filter depends on project-specific demands and resources.
MEDLINE search filters for identification of qualitative research differ greatly in design and performance.
What this adds to what was known?
•
Initial validation of a search filter designed by University of Texas.
•
Comparative overview of validation performance of MEDLINE search filters using a uniform gold standard (independent from medical topic and with literature published in several years).
What is the implication, what should change now?
•
It is not possible to suggest a single search filter to identify qualitative research in MEDLINE.
•
The choice for or against a search filter depends on project-specific demands and resources.
1. Introduction
Qualitative studies provide deep insights into patient preferences and experiences. In recent years, qualitative research has established itself as an important contribution to evidence-based health care. Therefore, studies of this kind should be considered for high-quality systematic reviews, as stated by the Cochrane Handbook for Systematic Reviews of Interventions [
in: Noyes J. Booth A. Hannes K. Harden A. Harris J. Lewin S. Supplementary Guidance for Inclusion of Qualitative Research in Cochrane Systematic Reviews of Interventions. Cochrane Collaboration Qualitative Methods Group,
2011
]. Qualitative research comprises many research designs, methods, and approaches such as interviews, focus groups, ethnography, grounded theory, etc. Furthermore, published qualitative studies are often unstructured and contain no or only insufficient standardized vocabulary [
]. In contrast to other study designs such as randomized controlled trials, it is therefore more difficult and complex to find qualitative studies in bibliographic databases [
]. To get a comprehensive picture of qualitative studies as well as systematic reviews of qualitative studies on a specific medical or health-related topic, a search filter with a balanced ratio of sensitivity (retrieval of all relevant articles) and precision (retrieval of relevant articles as a proportion of total number of records found) can be a helpful tool for systematic search in bibliographic databases [
Many search filters were not adequately validated by means of an independent set of relevant references (gold standard). Moreover, gold standards (GSs) are often restricted, lacking an optimal size and references from several years [
]. In addition, topic-specific GS limit generalizability.
The aim of the present study was therefore to get an overview of already available search filters to identify qualitative research papers in the bibliographic database MEDLINE. To determine the filter with the highest sensitivity, the highest precision or the best balance between sensitivity and precision, all filters were validated with a newly generated GS without the constraint of a medical topic. With these validation results in mind, a search filter for MEDLINE can be selected with respect to individual requirements.
2. Methods
A four-stepped approach was used: 1. identification of search filters to locate qualitative research in the bibliographic database MEDLINE, 2. verification and plausibility checks of search filters, 3. generation of a GS irrespective of a medical or health-related topic, 4. validation of identified search filters against the newly created GS in MEDLINE using the Ovid platform.
2.1 Identification of search filters
A search was carried out to identify the most important search filters to locate qualitative research in the bibliographic database MEDLINE. Therefore, well-known sources and websites, for example, from the InterTASC Information Specialists' Subgroup [
in: Noyes J. Booth A. Hannes K. Harden A. Harris J. Lewin S. Supplementary Guidance for Inclusion of Qualitative Research in Cochrane Systematic Reviews of Interventions. Cochrane Collaboration Qualitative Methods Group,
2011
HiRU [Health Information Research Unit] Search Filters for Medline in Ovid Syntax and the PubMed translation. Last modified: February 9, 2016. Hamilton, CA: HiRU.
2.2 Verification and plausibility checks of search filters
For validation, all search filters for qualitative studies were included that were generated for the bibliographic database MEDLINE using the Ovid platform, that were plausible and completely reproducible.
For replicability of search filters, a plausibility check was performed on all search filters identified. If search filters contained considerable syntax errors or content inconsistencies, they were excluded from validation. Developers and authors were not contacted.
The Medical Subject Heading (MeSH) “Qualitative Research” was introduced in 2003. However, some of the identified filters were developed before implementation of this controlled vocabulary and so far have not been updated [
]. Besides, the heading “Qualitative Research” comprises the sole narrower term “Hermeneutics”, with an entry date of 2014, resulting in two additional search filters (“Qualitative Research/” and “exp Qualitative Research/”) [
] formed a GS by combining all studies that were included in various systematic reviews (relative recall approach). This allowed for the generation of a GS that includes relevant publications over a long period, regardless of a medical or health-related topic. The relative recall approach has already been used [
] formed the basis to generate the GS. The aim of Dalton et al. was to provide a descriptive overview of systematic reviews of qualitative studies listed in the Database of Abstracts of Reviews of Effects (DARE). A search for the period from 2009 to 2014—using the internal tagging system—yielded 145 published systematic reviews. Until 2014, DARE itself includes hits from MEDLINE, Embase, PsycINFO, PubMed, and CINAHL and tagged relevant reviews as qualitative research since 2009 [
] were obtained in full text and managed in EndNote. Thereafter, all primary studies included in these systematic reviews that also were available as journal articles or dissertations were loaded in an EndNote library. Gray literature and books were not considered because they are generally not indexed in bibliographic databases. There was no assessment as to whether the primary studies applied qualitative methods, as we assume that they are all qualitative studies and thus fulfill our inclusion criteria. However, if the included studies in the systematic reviews were clearly divided into qualitative and quantitative primary studies, only qualitative studies were considered to compile the GS (e.g., Torquato Lopes et al. [
]). Finally, duplicates were removed and only publications indexed in MEDLINE were added to the entire GS.
2.4 Validation of search filters
During the validation process, it was determined whether the references of the GS could be identified with the respective filter in MEDLINE via Ovid. Based on the number of hits, the sensitivity, the precision, and the number needed to read (NNR) were calculated (Table 1). The relative recall approach does not allow for the estimation of the true precision and NNR [
]. However, to enhance the comparability of search filter performances, the precision ratio was calculated, while setting the precision ratio of 1 for the search filter with the highest sensitivity [
UTHealth [University of Texas Health Science Center at Houston] Search Filters for Various Databases: Ovid Medline. Last Updated: 28.11.2018. Houston, US-TX: UTHealth.
] evaluated three different filters (“thesaurus terms”, “free-text terms”, “broad-based terms”) for several databases including MEDLINE via Ovid, where the GS consisted of references on support for breastfeeding. The McMaster University hedges team developed seven different search filters for qualitative studies that can be used for a variety of purposes (different “single term” and “two or three term” strategies) [
]. The GS used were publications from 161 health care journals from the year 2000, which dealt with patient experiences and where data were gathered using qualitative methods [
], also from the McMaster University hedges team, list another filter which searched a single word in multiple fields. The aforementioned search filters were developed and published without taking into account the MeSH-term “Qualitative Research”, introduced in 2003 [
]. The website of the University of Texas—School of Public Health (UTHealth) lists a further, nonvalidated search strategy for qualitative studies considering the MeSH-term mentioned previously [
UTHealth [University of Texas Health Science Center at Houston] Search Filters for Various Databases: Ovid Medline. Last Updated: 28.11.2018. Houston, US-TX: UTHealth.
None of these search filters have been validated with a topic-independent GS covering published references from several years. In addition, no comparison was made in terms of sensitivity and precision.
3.2 Verification and plausibility checks of search filters for qualitative research
A total of 15 search filters were tested for plausibility (Table 2). All seven different search filters by Wong et al. (Wonga-g) and the search filter from UTHealth were included for validation [
UTHealth [University of Texas Health Science Center at Houston] Search Filters for Various Databases: Ovid Medline. Last Updated: 28.11.2018. Houston, US-TX: UTHealth.
]. The search filters Wongc (“maximizes sensitivity”), Wonge (“maximizes specificity”), and Wongg (“best balance of sensitivity and specificity”) can also be found on the McMaster University website [
HiRU [Health Information Research Unit] Search Filters for Medline in Ovid Syntax and the PubMed translation. Last modified: February 9, 2016. Hamilton, CA: HiRU.
], could be a transfer error while testing multiple hedges preliminary for the database PsycINFO. It is possible, that the single-term strategy “interviews.tw.” by Wong et al. (Wongb) [
] with a different field notation could have been meant. As this uncertainty could not be ruled out, “interviews.mp.” (McKibbon) was kept as another search filter to identify qualitative articles [
HiRU [Health Information Research Unit] Search Filters for Medline in Ovid Syntax and the PubMed translation. Last modified: February 9, 2016. Hamilton, CA: HiRU.
HiRU [Health Information Research Unit] Search Filters for Medline in Ovid Syntax and the PubMed translation. Last modified: February 9, 2016. Hamilton, CA: HiRU.
HiRU [Health Information Research Unit] Search Filters for Medline in Ovid Syntax and the PubMed translation. Last modified: February 9, 2016. Hamilton, CA: HiRU.
Qualitative Research/ or Nursing Methodology Research/ or Questionnaires/ or exp Attitude/ or Focus Groups/ or discourse analysis.mp. or content analysis.mp. or ethnographic research.mp. or ethnological research.mp. or ethnonursing research.mp. or constant comparative method.mp. or qualitative validity.mp. or purposive sample.mp. or observational method$.mp. or field stud$.mp. or theoretical sampl$.mp. or phenomenology/ or phenomenological research.mp. or life experience$.mp. or cluster sampl$.mp.
UTHealth [University of Texas Health Science Center at Houston] Search Filters for Various Databases: Ovid Medline. Last Updated: 28.11.2018. Houston, US-TX: UTHealth.
(((“semi-structured” or semistructured or unstructured or informal or “in-depth” or indepth or “face-to-face” or structured or guide) adj3 (interview* or discussion* or questionnaire*))).ti,ab. or (focus group* or qualitative or ethnograph* or fieldwork or “field work” or “key informant”).ti,ab. or interviews as topic/ or focus groups/ or narration/ or qualitative research/
], the search filter with the “free-text terms” (Shawb) was excluded from the validation because it contains both syntax inconsistencies (e.g., missing field notations in some search lines, incorrect parentheses) and content-related discrepancies (e.g., “speigelberg$.tw.” instead of “spiegelberg$.tw.”, redundant search lines). The hybrid filter from DeJean et al. [
] relies among others on Shawb without correcting syntax errors or inconsistencies. It was therefore excluded as well. Both aforementioned filters were constructed for medical and health-related topics with a special focus on women's experiences. Shaw et al. [
] searched for qualitative articles in the field of chronic obstructive pulmonary disease as well as early breast cancer. Including terms such as “women's stor*” or “feminis$” seem appropriate for these topics (if male breast cancer is excluded) but is not suitable for an overall search filter to locate qualitative research in general. Then, terms such as “men's stor*” should have also been incorporated. In conclusion both search filters (Shawb [
However, Shawa (“thesaurus terms”) and Shawc (“broad-based terms”) were included as well as the MeSH-term “Qualitative Research” (exploded and unexploded; MeSHa and MeSHb [
UTHealth [University of Texas Health Science Center at Houston] Search Filters for Various Databases: Ovid Medline. Last Updated: 28.11.2018. Houston, US-TX: UTHealth.
], 131 were indexed in MEDLINE (90.3%, gold standard 1, Fig. 1). Characteristics of these reviews can be found in the initial publication by Dalton et al. [
]. The reviews were published between 2009 and 2014 and focus on different medical or health-related topics, for example, cancer, mental health, or diabetes.
Fig. 1Generation of gold standard. (For interpretation of the references to color in this figure legend, the reader is referred to the Web version of this article.)
For the systematic literature searches described in the systematic reviews, one to 26 bibliographic databases were used (mean: 7). MEDLINE (via PubMed and/or via Ovid) was searched the most, in 96.6% of all systematic reviews (140 of 145). CINAHL was accessed in 87.6%, PsycINFO in 82.8%, and Embase in 63.4% of the systematic reviews.
In about 60% of included reviews, the search strategies included a search block to identify qualitative studies. This search block was very heterogeneous in the reviews, often only single words were used, such as “qualitative”, “grounded theory”, or “focus groups” without specifying a field notation. A clear reference to previously published search filters was very rare (see Supplementary Table). Owing to a lack of reporting standards regarding search strategies—and in particular the applied search filters—it is unclear how often published filters were used or adapted for the literature search. However, almost all reviews took additional sources (hand searches of specific websites and journals, review of reference lists, contact with authors and experts, forward and backward citation of references, etc.) into account to ensure a systematic search for qualitative studies. In the remaining reviews, the search was carried out without restriction to qualitative studies or no details about the entire literature search strategies were reported.
In total, the systematic reviews included 3,012 primary studies, of which 2,898 were recorded in an EndNote library, excluding clearly defined quantitative studies, books, and gray literature. After deduplication, 2,715 articles remained, of which 2,192 references were indexed in MEDLINE (80.7%, gold standard 2, Fig. 1). A critical appraisal of the included primary qualitative studies was done using a wide variety of tools in 133 systematic reviews (91.7%). Authors of 46 reviews planned to exclude studies because of their poor quality.
The entire GS consisting of qualitative studies and systematic reviews on qualitative studies included a total of 2,323 references. The publication period of the references included ranged from 1968 to 2014.
3.4 Validation of search filters
The validation with the entire GS showed that the search filter Wongc, which is also preferred by McMaster University as a very sensitive search filter [
HiRU [Health Information Research Unit] Search Filters for Medline in Ovid Syntax and the PubMed translation. Last modified: February 9, 2016. Hamilton, CA: HiRU.
], had the best sensitivity in MEDLINE (93.63%) (Table 3). With this filter, 2,175 of 2,323 references could be retrieved. By contrast, there was a NNR of 1,418 articles, indicating a very high screening workload. During validation, the search filter Wongd [
The best precision of 2.15% could be reached with the MeSH-term “Qualitative Research”, but with the lowest sensitivity (22.56%) and a NNR of 47 articles (each, MeSHa and MeSHb) [
UTHealth [University of Texas Health Science Center at Houston] Search Filters for Various Databases: Ovid Medline. Last Updated: 28.11.2018. Houston, US-TX: UTHealth.
] demonstrated the best balance between sensitivity and precision (81.96% and 0.80%, respectively) for the entire GS. In this case, 126 articles need to be read to uncover one relevant article (Table 3). This search filter was thus 11.3 times more precise (precision ratio) than the very sensitive search filter Wongc [
If the GS is considered separately as systematic reviews of qualitative studies (gold standard 1) and primary studies (gold standard 2), the validation results are slightly different (Table 4). With GS 1, the best sensitivity could be reached with the Wongd search filter (97.71%). One hundred twenty-eight of the 131 references in GS 1 could be identified using this search filter. Furthermore, three search filters presented by Wong et al. [
] reached a sensitivity of 96.18% (Wonge, Wongf, and Wongg), respectively. Wonge on the other hand, retrieved 126 of 131 references with a notable reduction in the expected screening workload (NNR: 518 articles). Moreover, the Wonge filter was 5.3 times more precise (precision ratio) than the search filter Wongd. Thus, this search filter achieved the best balance between sensitivity and precision. Wong et al. [
] found that this search filter (Wongd) has the best sensitivity when compared with their other developed two- or three-term strategies, keeping specificity ≥ 50%. The search filter Wongc, listed as the most sensitive filter by McMaster University [
HiRU [Health Information Research Unit] Search Filters for Medline in Ovid Syntax and the PubMed translation. Last modified: February 9, 2016. Hamilton, CA: HiRU.
], performed poorer in validation with GS 1 (Table 4, sensitivity of 87.79%) than with the entire GS (Table 3, sensitivity of 93.63%). Again, the MeSH-term searches yielded the best precision (0.360%), but with a low sensitivity of 47.33%.
Table 4Validation results with gold standard 1 (131 references, left) and gold standard 2 (2,192 references, right)
To our knowledge, these findings provide a good overview of already available search filters for qualitative research for the bibliographic database MEDLINE. Based on the validation results, which were generated using a GS according to the relative recall approach, a search filter for MEDLINE may be selected, with respect to the individual requirements.
The search filter Wongc (two- or three-term strategy), which is also listed on the McMaster University website, can be recommended for a very sensitive search in MEDLINE [
HiRU [Health Information Research Unit] Search Filters for Medline in Ovid Syntax and the PubMed translation. Last modified: February 9, 2016. Hamilton, CA: HiRU.
]. Although this search filter was very sensitive, not all relevant qualitative studies from the GS were detected with this filter. This is probably due to the heterogeneous reporting in qualitative studies with no or only insufficient standardized vocabulary and many different research designs, methods, and approaches [
Although the MeSH-term searches (exploded and unexploded) have the best precision, they cannot be recommended as the sole search string because of the very poor sensitivity of about 23%, even if the NNR promises a low screening workload. With these MeSH-term searches, only a few qualitative studies can be found without any claim to completeness (about 500 out of 2,323 from the entire GS) but probably very precise hits. Here, a consequent tagging by indexers of the national library of medicine would have been desirable. In addition, it would be beneficial if study authors used a rather standardized wording to describe their qualitative studies.
However, with most of the search filters tested, both systematic reviews and primary qualitative studies can be retrieved with sufficiently good results. Solely search filters that contain only the single word “interview” (with different field notations) showed very poor results in terms of sensitivity when only systematic reviews were considered (gold standard 1). Here, the use of longer search strategies or of MeSH-terms has proven to be much more advantageous.
To our knowledge, the search filter developed by the University of Texas [
UTHealth [University of Texas Health Science Center at Houston] Search Filters for Various Databases: Ovid Medline. Last Updated: 28.11.2018. Houston, US-TX: UTHealth.
] has never been validated and the date and method of development is unknown. It considers the MeSH-term “Qualitative Research” and might fill the gap between one- to three-term strategies by Wong et al. [
]. Our findings show that this search filter obtained a good balance between sensitivity and precision. With its sufficiently good sensitivity combined with an appropriate workload for screening, it can be recommended for systematic searches on qualitative studies.
Our results show strong differences for the available search filters for qualitative studies in terms of sensitivity, precision, and NNR. When searching qualitative studies in MEDLINE, the selection of a suitable search filter therefore depends on the research question, the claim for completeness, and the resources available for screening. Furthermore, a systematic search should include different bibliographic databases (not only MEDLINE). For example, the bibliographic database CINAHL may contain a good collection of qualitative studies as shown for studies on dementia [
] allows the creation of a GS regardless of a medical or health-related topic. The size of the GS and the long publication period is also of particular importance to improve the estimation of the validation measurements. In addition, the GS was checked for a possible selection bias. This would be conceivable, for example, if most of the 145 systematic reviews themselves have used only one particular published search filter to identify the qualitative studies. Because this assumption did not apply, with the GS in place, a good generalizability is given.
However, a proper calculation of precision and thus the NNR was not possible with the relative recall approach. For this, the exact number of irrelevant but found references by the search filter would have been necessary. However, this number could only be approximately determined because of a lack of manual screening of the references and thus probably led to an underestimation of the true precision or NNR.
The validation results also benefit from the fact that all search filters validated were developed exclusively for use in Ovid MEDLINE, so that no filters had to be translated (with attendant problems of translation inaccuracies).
4.2 Limitations
In all, it needs to be emphasized that the calculated performance measures are only rough estimations. It cannot be ruled out that precision calculations are underestimated because the retrieved references of the individual search filters could still contain relevant qualitative articles that were not part of the entire GS. Furthermore, it is probably possible that some references of the entire GS are not qualitative research because we did not manually verify this. However, to overcome this, we additionally presented the precision ratio as a means of normalization to compare the search filters with each other.
Likewise, it could be possible that the GS 2 is biased in selection as some review authors aimed to exclude studies of poor quality. However, the magnitude of this impact could not be entirely examined because of scarce reporting, but seems to be acceptable.
Because the GS were generated irrespective of a subject, there is a possibility that the performance measures may differ in combination with a specific medical topic. Nevertheless, by doing so, the generalizability is increased. It cannot be ruled out, that using a GS created from publications based on systematic reviews (including systematic searches using specific words referring to qualitative research, see Supplementary Table) influences the presented validation results.
The true informative value of the measurements in everyday use depends on the combination of the search filter with the topic-specific search terms and other limitations. For example, the true numbers of performance measures can change considerably by adding a thematic search block. To ensure that all qualitative studies will be retrieved, it should also be examined whether the use of a restrictive search filter for qualitative studies should be completely omitted.
5. Conclusions
To the best of our knowledge, these are validation results for a broad range of published search filters in MEDLINE using a GS irrespective of a restrictive subject following the relative recall approach by Sampson et al. [
In this context, the filter Wongc can be recommended as a very sensitive search filter to identify primary studies and systematic reviews of qualitative research in MEDLINE. However, this comes along with a high number of articles to be read, which would be necessary to identify the relevant ones. The search filter of the University of Texas represents an appropriate choice if a sensitivity of above 80% is desired while minimizing the workload.
(Version 1 (updated August 2011))in: Noyes J. Booth A. Hannes K. Harden A. Harris J. Lewin S. Supplementary Guidance for Inclusion of Qualitative Research in Cochrane Systematic Reviews of Interventions. Cochrane Collaboration Qualitative Methods Group,
2011 (Available at)
30817-0&id=gr1.jpg){kind=link}