If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Université de Paris, UMR 1153 CRESS Inserm, 75004, Paris, FranceClinical Epidemiology Unit, Hôtel-Dieu Hospital, Assistance Publique-Hôpitaux de Paris, (AP-HP), 75004, Paris, FranceCochrane France, Hôtel-Dieu Hospital, 75004, Paris, France
Université de Paris, UMR 1153 CRESS Inserm, 75004, Paris, FranceClinical Epidemiology Unit, Hôtel-Dieu Hospital, Assistance Publique-Hôpitaux de Paris, (AP-HP), 75004, Paris, FranceCochrane France, Hôtel-Dieu Hospital, 75004, Paris, France
Université de Paris, UMR 1153 CRESS Inserm, 75004, Paris, FranceClinical Epidemiology Unit, Hôtel-Dieu Hospital, Assistance Publique-Hôpitaux de Paris, (AP-HP), 75004, Paris, FranceCochrane France, Hôtel-Dieu Hospital, 75004, Paris, France
Université de Paris, UMR 1153 CRESS Inserm, 75004, Paris, FranceClinical Epidemiology Unit, Hôtel-Dieu Hospital, Assistance Publique-Hôpitaux de Paris, (AP-HP), 75004, Paris, FranceCochrane France, Hôtel-Dieu Hospital, 75004, Paris, France
To assess the sensitivity of two secondary electronic sources of COVID-19 studies: 1) the Cochrane COVID-19 Study Register (https://covid-19.cochrane.org/); and, 2) the Living Overview of the Evidence (L•OVE) COVID-19 platform (https://iloveevidence.com/).
Study design and setting
We identified reports of randomized controlled trials (RCTs) and observational studies (OS) assessing preventive interventions or treatment for COVID-19.
The reference standard comprised all reports included in the COVID-NMA platform (covid-nma.com), in two major living systematic reviews of RCTs assessing pharmacologic treatment of COVID-19, or identified in either of the two secondary sources evaluated. The search for all sources was conducted through September 7, 2020.
Our primary outcome was the proportion of the reports included in the reference standard that were identified by each secondary source.
Results
We identified 680 reports, 91 RCT reports, 97 RCT protocols, and 492 OS reports. The Cochrane COVID-19 Study Register identified 88% [95% confidence interval, 79–94] of the RCT reports, 90% [82–95] of the RCT protocols, and 82% [78–85] of the OS reports. The L•OVE platform identified 100% [97–100] of the RCT reports and RCT protocols and 100% [99–100] of the OS reports.
Conclusion
These platforms proved to be a viable screening alternative to searching every individual source.
Two secondary sources (the Cochrane COVID-19 study register and the LOVE COVID-19 platform) demonstrated very good sensitivity for the identification of studies assessing interventions to prevent or treat COVID-19.
•
The workload associated with using these secondary sources was reduced.
What this adds to what is known
•
Ongoing living systematic reviews on COVID-19 could rely on these secondary sources.
What is the implication, what should change now
•
Validated secondary sources on other topics should be developed to improve the ease of implementation of living evidence syntheses.
1. Introduction
In December 2019, the first reports documented a novel coronavirus outbreak in Wuhan, Hubei Province, China. This novel coronavirus, known as SARS-CoV-2, spread from China to several countries and then around the world; the declared COVID-19, the illness caused by the virus, to be a pandemic on March 11, 2020 [
]. Since then, there has been an explosion of research globally seeking to combat this major public health threat by evaluating the effectiveness and safety of various interventions aimed at preventing or treating COVID-19.
Under these circumstances, the rapid production of high-quality, comprehensive, up-to-date living systematic reviews is indispensable to facilitate evidence-based decision making. One example includes the COVID-NMA initiative, which identifies, extracts, and synthesizes all randomized controlled trials (RCTs) evaluating the prevention and/or treatment of COVID-19 [
]. All results are made available and updated weekly on the COVID-NMA platform (covid-nma.com).
However, performing and promptly updating a high-quality living systematic review is very labor intensive. A complex strategy that screens a large number of records is usually needed to accurately and comprehensively identify all eligible reports. To ensure the sustainability of living systematic reviews, it is essential to explore new approaches to study identification [
Searching ClinicalTrials.gov and the international clinical trials registry platform to inform systematic reviews: what are the optimal search approaches?.
In the context of the COVID-19, some organizations have developed secondary electronic sources that systematic reviewers can use to identify eligible studies for their living reviews. In particular, the Cochrane COVID-19 study register [
] developed by the Epistemonikos Foundation (https://app.iloveevidence.com/covid19), offer free access to their up-to-date databases of COVID-19 studies, screened and annotated by PICO and study design. Searches on these platforms are facilitated through filters and data can be easily downloaded. Consequently, we decided to explore the sensitivity of these sources to determine if they could serve as a principal or even unique source for the search strategy of living systematic reviews of interventional studies for COVID-19.
The objective of this study was to evaluate the sensitivity of two secondary sources (the Cochrane COVID-19 study register and the L•OVE COVID-19 platform) for the identification of publications in preprint or peer-reviewed journal of 1) RCTs, 2) RCT protocols, and 3) OS assessing interventions to prevent or treat COVID-19.
2. Methods
2.1 Study design
We conducted a cross-sectional study, embedded within the COVID-NMA initiative [
We included RCT reports and protocols (i.e., preprint or publication in a journal of the protocol) and OS that assessed:
1)
Interventions to prevent the spread of SARS-CoV-2 infection (e.g., vaccine, prophylactic pharmacologic treatments, and nonpharmacologic interventions for preventing SARS-CoV-2 transmission, such as personal protective equipment)
2)
Interventions for treating COVID-19 (e.g., anti-infective agents, specific and nonspecific immunomodulators, supportive treatments for hospitalized patients, and general treatments for viral infection)
3)
Post-acute care interventions for COVID-19 patients (e.g., rehabilitation, treatment for long-COVID)
4)
Models of practice and organization of care aimed at improving care of COVID-19 patients.
Prognostic studies, systematic reviews, meta-analyses, post-hoc analyses, and diagnostic test sensitivity studies were excluded, as were reports of studies evaluating traditional Chinese medicine or other herbal therapies.
2.3 Secondary sources evaluated
2.3.1 The Cochrane COVID-19 study register
The Cochrane COVID-19 Study Register was funded by Cochrane to support rapid evidence synthesis. This register is study-based and it searches the following primary data sources: PubMed, Embase, [
]. Retraction Watch is also used to identify retraction notices and expressions of concern. Searches are updated daily in PubMed and ClinicalTrials.gov, weekly in Embase, ICTRP, and medRxiv, and monthly in CENTRAL. The Cochrane COVID-19 Study Register does not search and is not linked to the L•OVE COVID-19 Platform.
During our study, the Cochrane registry was negotiating a licensing agreement with Elsevier to publish Embase records in the Cochrane register (i.e., not available for our search). It began searching medRxiv in May 2020 and CENTRAL in November 2020. Given these changes, we conducted a supplemental search on December 29, 2020, to determine whether the missing studies had been added and were available in the registry, by systematically entering in the search field for each missing study: 1) the title, 2) the title and first author's name, 3) the first author's name. We decided to update our database with the studies identified during this process and to present results before and after this update. More details on this process are available in Appendix A and at https://community.cochrane.org/about-covid-19-study-register.
2.3.2 The living overview of the evidence (L•OVE) COVID-19 platform
The Living Overview of the Evidence (L•OVE) COVID-19 is an open access platform funded by the Epistemonikos Foundation, a nonprofit organization. The primary data sources used to set up this platform include several bibliographic sources, preprint platforms and trial registries, including PubMed, EMBASE, CINAHL (the Cumulative Index to Nursing and Allied Health Literature), PsycINFO, LILACS (Latin American & Caribbean Health Sciences Literature), Wanfang Database, CBM (Chinese Biomedical Literature Database), CNKI (Chinese National Knowledge Infrastructure), VIP (Chinese Scientific Journal Database), IRIS (WHO Institutional Repository for Information Sharing), IRIS PAHO (PAHO Institutional Repository for Information Sharing), IBECS (Spanish Bibliographic Index on Health Sciences), Microsoft Academic, ICTRP Search Portal, Clinicaltrials.gov, ISRCTN registry, Chinese Clinical Trial Registry, IRCT (Iranian Registry of Clinical Trials), EU Clinical Trials Register (Clinical trials for COVID-19), Japan NIPH Clinical Trials Search, JPRN (Japan Primary Registries Network) (JPRN - includes JapicCTI, JMACCT CTR, jRCT, UMIN CTR), MedRxiv, BioRxiv, SSRN Preprints, Research Square, ChinaXiv and SciELO Preprints. The (L•OVE) COVID-19 Platform does not search or link to the Cochrane COVID-19 Study Register. Screening and classification for the (L•OVE) COVID-19 platform rely both on artificial intelligence and manual screening.
On September 23, 2020, the L•OVE platform was updated with the addition of observational studies that were available on the L•OVE platform when we exported the data and consequently not screened at the time of our study. We decided to update our database with the observational studies identified during this process and to present results before and after this update. More details on the process are available in Appendix B and at https://app.iloveevidence.com/covid19/methods.
2.4 Reference standard
The reference standard comprised all reports of studies that met the eligibility criteria that were 1) identified during the process of the COVID-NMA initiative, which systematically searched for reports of RCTs, protocols of RCTs and reports of OS from March, 2020, through September 7, 2020 (inclusive) or 2) included in two living systematic reviews of RCTs assessing pharmacologic treatment for COVID-19, [
] or 3) identified by any of the two secondary sources evaluated.
The search strategy and eligibility criteria of the COVID-NMA initiative and the two living systematic reviews of pharmacologic treatments are detailed here [
The searches of the two secondary sources are detailed in Table 1. They were conducted weekly (each Wednesday at same time) from July 15, 2020 to September 7, 2020 (inclusive).
Table 1Search in secondary sources evaluated
Cochrane's COVID-19 Study Register
The Living Overview of the Evidence (L•OVE) COVID-19 Platform
Search 1Field “Study characteristics”/“Study type”: selection of “Interventional”Field “Study characteristics”/“Study aim”: selection of “Treatment And Management”Click on “Select All” and “Export” as a RIS document.Search 2
•
Field “Study characteristics”/“Study type”: selection of “Interventional”
•
Field “Study characteristics”/“Study aim”: selection of “Prevention”
•
Click on “Select All” and “Export” as a RIS document.
Search 3
•
Field “Study characteristics”/“Study type”: selection of “Observational”
•
Field “Study characteristics”/ “Study aim”: selection of “Treatment And Management”
•
Click on “Select All” and “Export” as a RIS document.
Search 4
•
Field “Study characteristics”/“Study type”: selection of “Observational”
•
Field “Study characteristics”/“Study aim”: selection of “Prevention”
•
Click on “Select All” and “Export” as a RIS document.
Weekly update of the search:
•
we conducted the same searches with a selection of the field “updated”/ “last week”.
Search 1 Field 1: “Select type of question”/“Prevention or treatment“ Field 2: “Select population”/“COVID-19” Field 3: “Primary studies”/ used the filter “Reporting data”, “RCT” click on “Export” in RIS format. Search 2 Field 1: “Select type of question”/ “Prevention or treatment” Field 2: “Select population”/“COVID-19” Field 3: “Primary studies”/used the filter “Reporting data”, “Not RCT” click on “Export” in RIS format.
] for screening. Updates were screened on MS Excel for the L•OVE platform and on Rayyan for the Cochrane COVID-19 Study Register.
All records identified in each secondary source were screened independently by the same two trained reviewers (OP, SC). Disagreements were resolved by consensus and, if necessary, consultation with a senior reviewer. Quality control included systematically checking whether the studies identified in one secondary source were not wrongly excluded during the screening process of the other secondary source.
References to all reports meeting our eligibility criteria and identified by the COVID-NMA initiative and in the two living reviews were extracted by two reviewers independently. Any disagreements were resolved by consensus.
2.6 Outcome criteria
Our primary outcomes were the proportion of reports included in the reference standard identified by each secondary source. We considered 1) RCT reports, 2) RCT protocols, and 3) OS reports — all assessing interventions for preventing and treating COVID-19.
Our secondary outcome was the workload, which we assessed by the total number of records requiring screening when using these two secondary sources.
2.7 Statistical analysis
Categorical data were described using percentages, and 95% confidence intervals. The proportion of studies identified was reported for each type of report (RCTs, RCT protocol, OS) for each platform.
3. Change to the protocol
We initially planned to evaluate the COVID-Evidence database [
]. However, we only searched the database to August 12, 2020, because the platform was not updated for several weeks and the search strategy had been modified. Consequently, we decided not to report the results.
As the L•OVE platform and the Cochrane COVID-19 Study Register were still in process of improvement at the time of the study, we decided to conduct a new search based on the updated platforms. We present both results in the text and the most updated results in the abstract.
We informed the responsible parties of both platforms of our results and reported any information that could explain the results.
4. Results
4.1 Reference standard
Overall, we identified 680 reports that met our eligibility criteria: 91 RCT reports, 97 RCT protocols, and 492 OS reports (Fig. 1).
Table 2 reports the sensitivity of both platforms. Before the update, the L•OVE platform identified 91/91 (100% [97–100]) of the RCT reports, 97/97 (100% [97–100]) of the RCT protocols, and 348/492 (71% [66–75]) of the OS reports. After the database was updated, the L•OVE platform also identified 100% [99-100] of OS reports.
Table 2Number (percentages and [95% confidence interval]) of studies identified in the different secondary sources
Before the update, the Cochrane COVID-19 Study Register identified 29/91 (32% [22–42]) RCT reports; 48/97 (49% [39-60]) protocols of RCTs; 252/492 (51% [47–56]) reports of OS. After updating the database, the register identified 80/91 (88% [79–94]) RCT reports, 87/97 (90% [82–95]) protocols of RCTs and 403/492 (82% [78–85]) reports of OS.
After discussing our results with the Cochrane register (R Featherstone), an issue related to the data filter was identified. The filter was working at the study level as opposed to the reference level; the filter retrieved new study records updated in the last week, but it did not retrieve new references added to studies already contained in the register. Consequently, the studies identified after the supplemental search on December 29, 2020 may have been in the registered but not retrieved by the filter. This inconsistency has now been modified.
4.3 Workload
The workload was determined only before the update. We screened 2,645 records from the L•OVE platform and 6,120 from the Cochrane COVID-19 Study Register 5 (Fig. 1).
In contrast, the workload for each living review was more intensive: 45,812 records were screened for the COVID-NMA initiative (search date: 7 September 2020; inclusion of RCTs, RCT protocols, and OS — all for preventive interventions and treatment of COVID-19); 8,864 for the review by Siemieniuk et al. (search date: 10 August 2020; inclusion of RCTs of pharmacologic treatments of COVID-19) and 6,693 for the review by Juul et al. (search date: August 07, 2020; inclusion of RCTs of treatment of COVID-19).
The two systematic reviewers also assessed the difficulties of using both secondary sources. The Cochrane COVID-19 Study Register was considered to be the most user-friendly as it allowed filtering by date to identify new studies, a functionality that we did not find on the L•OVE platform at the time of the study.
5. Discussion
This study assessed the sensitivity of two secondary sources. Our results showed very good sensitivity in the L•OVE COVID-19 platform for all types of reports after the update. It identified all RCTs, RCT protocols, and OS while considerably reducing the workload for this result. Results were also very good for the Cochrane COVID-19 Study Register, which identified 88% of the RCTs, 90% of the RCT protocols, and 82% of the OS after the update. An issue related to the filters implemented in the Cochrane COVID-19 Study Register has been identified and is now corrected. Furthermore, the workload was considerably lower compared to the usual process, particularly on the L•OVE platform.
Our results are consistent with other studies evaluating the Cochrane COVID-19 Study Register [
Shemilt I, Noel-Storr A, Thomas J, Featherstone R, Mavergames C. Machine learning reduced workload for the Cochrane COVID-19 study register: development and evaluation of the Cochrane COVID-19 study classifier. Published online August 29, 2021. doi:10.21203/rs.3.rs-689189/v1
]. To our knowledge, the sensitivity of the L•OVE platform has never been evaluated.
Our study has important strengths. The reference standard was based on several sources with high-quality screening processes, and we are confident that we did not miss any reports. Two trained researchers independently performed the search and screening in the two secondary sources, with consensus when they initially disagreed. The screening of the databases was performed independently of the record identified in other sources to avoid verification bias.
Our study has important implications. Considering the reduced workload and the high sensitivity of these sources, searching primary sources may be not necessary for systematic reviews assessing COVID-19 interventions, particularly if they are also searching registries. Still, it might be useful to search these two secondary sources as they could serve as a quality control for each other.
Our study also has some limitations. We did not include studies evaluating traditional Chinese medicine or other herbal therapies as these interventions were not considered in the COVID-NMA initiative. The platforms evaluated are being continuously improved, and the results may evolve over time. Particularly, the L•OVE COVID-19 platform completed the screening after the end of our study. Similarly, Cochrane was in the process of negotiating their Embase agreement between July and September. Our results showed a considerable improvement of both sources’ sensitivity after their updated searches. There may also be an issue related to the time between the moment the study is published and the moment it appears on the different platforms, which might create some delay and decrease our speed in identifying new studies and updating our living systematic review. However, both secondary sources are updated regularly, reducing the likelihood of significant delays. Finally, while we relied on several sources in our reference standard to identify RCTs, the identification of RCT protocols and OS relied only on the COVID-NMA platform and the results identified by the two sources.
In conclusion, these secondary sources are essential tools that make living systematic reviews sustainable. The Cochrane COVID-19 Study Register provided very good results, an issue related to the filter has been identified and corrected; a second evaluation will be needed. The L•OVE platform allowed the identification of all studies included in our reference standard and proved to be a viable screening alternative with an acceptable workload. It also implemented some changes (addition of filters) to improve its use. These platforms could be used as a sole source for living evidence synthesis of COVID-19 studies.
Author contributions
Isabelle Boutron: Conceptualization, Methodology, Writing - Review & Editing, Supervision. Olivier Pierre: Methodology, Data extraction, Analysis, Writing- Original draft preparation. Carolina Riveros: screening and data extraction. Sarah. Charpy: screening and data extraction.
Funding
This study is part of the COVID-NMA project and was funded by the Agence Nationale de la Recherche (ANR).
Conflict of interest
Isabelle Boutron is director of Cochrane France, Member of Cochrane editorial board.
Acknowledgment
We are very grateful to Robin Featherstone Information Specialist, Cochrane Editorial & Methods Department and Camila Ávila, Juan Vásquez, Camilo Vergara and Gabriel Rada from Epistemonikos Foundation who provided feedback on our manuscript and facilitated this study.
Appendix A: Search strategy implemented by the Cochrane COVID-19 Study Register (see https://community.cochrane.org/about-covid-19-study-register)
Tabled
1
Source
Current Strategy (last updated November 09, 2020)
PubMed
(2019 nCoV[tiab] OR 2019nCoV[tiab] OR corona virus[tiab] OR corona viruses[tiab] OR coronavirus[tiab] OR coronaviruses[tiab] OR COVID[tiab] OR COVID19[tiab] OR nCov 2019[tiab] OR SARS-CoV2[tiab] OR SARS CoV-2[tiab] OR SARSCoV2[tiab] OR SARSCoV-2[tiab] OR "Coronavirus"[Mesh:NoExp] OR "COVID-19"[nm] OR "COVID-19 drug treatment"[nm] OR "COVID-19 diagnostic testing"[nm] OR "COVID-19 serotherapy"[nm] OR "COVID-19 vaccine"[nm] OR "LAMP assay"[nm] OR "severe acute respiratory syndrome coronavirus 2"[nm] OR "spike protein, SARS-CoV-2"[nm]) NOT ("animals"[mh] NOT "humans"[mh]) NOT (editorial[pt] OR newspaper article[pt])
Embase.com
((('coronaviridae'/de OR 'coronavirinae'/de OR 'coronaviridae infection'/de OR 'coronavirus disease 2019′/exp OR 'coronavirus infection'/de OR 'SARS-related coronavirus'/de OR 'Severe acute respiratory syndrome coronavirus 2′/exp OR '2019 nCoV':ti,ab,kw OR 2019nCoV:ti,ab,kw OR ((corona* OR corono*) NEAR/1 (virus* OR viral* OR virinae*)):ti,ab,kw OR coronavir*:ti,ab,kw OR coronovir*:ti,ab,kw OR COVID:ti,ab,kw OR COVID19:ti,ab,kw OR HCoV*:ti,ab,kw OR 'nCov 2019′:ti,ab,kw OR 'SARS CoV2′:ti,ab,kw OR 'SARS CoV 2′:ti,ab,kw OR SARSCoV2:ti,ab,kw OR 'SARSCoV 2′:ti,ab,kw) NOT (('animal experiment'/de OR 'animal'/exp) NOT ('human'/exp OR 'human experiment'/de))) NOT 'editorial'/it) NOT ([medline]/lim OR [pubmed-not-medline]/lim) AND [1-12-2019]/sd
CENTRAL
1 ("2019 nCoV" OR 2019nCoV OR "corona virus*" OR coronavirus* OR COVID OR COVID19 OR "nCov 2019" OR "SARS-CoV2" OR "SARS CoV-2" OR SARSCoV2 OR "SARSCoV-2"):TI,AB AND CENTRAL:TARGET 2 Coronavirus:MH AND CENTRAL:TARGET 3 Coronavirus:EH AND CENTRAL:TARGET 4 #1 OR #2 OR #3 5 2019 TO 2020:YR AND CENTRAL:TARGET 6 #5 AND #4 7 INSEGMENT 8 #6 NOT #7
ClinicalTrials.gov
COVID-19 OR 2019-nCoV OR SARS-CoV-2 OR coronavirus
WHO ICTRP
We screen the entire COVID-19.csv file available from https://www.who.int/emergencies/diseases/novel-coronavirus-2019
medRxiv
We screen the entire COVID-19 results identified by the Stephen B. Thacker CDC Library
SLCTR - Sri Lanka Clinical Trials Registry (updated daily, via ICTRP search portal)
36.
medRxiv (updated several times a day)
37.
bioRxiv (updated several times a day)
38.
SSRN Preprints (updated several times a day)
39.
ChinaXiv (updated every 2 weeks)
40.
SciELO Preprints (updated weekly)
41.
Research Square (updated daily)
b.
The following search strategy is used:
coronavir* OR coronovirus* OR betacoronavir* OR "beta-coronavirus" OR "beta-coronaviruses" OR "corona virus" OR "virus corona" OR "corono virus" OR "virus corono" OR hcov* OR "covid-19" OR covid19* OR "covid 19" OR "2019-ncov" OR cv19* OR "cv-19" OR "cv 19" OR "n-cov" OR ncov* OR (wuhan* and (virus OR viruses OR viral)) OR sars* OR sari OR (covid* and (virus OR viruses OR viral)) OR "severe acute respiratory syndrome" OR mers* OR "middle east respiratory syndrome" OR "middle-east respiratory syndrome" OR "covid-19-related" OR "2019-ncov-related" OR "cv-19-related" OR "n-cov-related"
c.
Inclusion criteria for the L•OVE database:
The database includes all articles related to COVID-19 or other coronaviruses with the following study designs: systematic reviews, primary studies, broad syntheses (e.g., guidelines, overviews of reviews, policy briefs), other articles with designs not included in the above-mentioned categories. Articles are classified as the various research questions (e.g., diagnostic, prevention or treatment) by artificial intelligence on entry into the Epistemonikos database.
Appendix C: Episthemonikos search strategy
The following search strategy was used coronavir* OR coronovirus* OR betacoronavir* OR "beta-coronavirus" OR "beta-coronaviruses" OR "corona virus" OR "virus corona" OR "corono virus" OR "virus corono" OR hcov* OR covid* OR "2019-ncov" OR cv19* OR "cv-19" OR "cv 19" OR "n-cov" OR ncov* OR (wuhan* and (virus OR viruses OR viral)) OR "2019-ncov-related" OR "cv-19-related" OR "n-cov-related" OR sars* OR sari OR "severe acute respiratory syndrome" OR antisars* OR "anti-sars-cov-2" OR "anti-sars-cov2" OR "anti-sarscov-2" OR "anti-sarscov-2"
References
Coronavirus disease (COVID-19) – World Health Organization.
Searching ClinicalTrials.gov and the international clinical trials registry platform to inform systematic reviews: what are the optimal search approaches?.
Shemilt I, Noel-Storr A, Thomas J, Featherstone R, Mavergames C. Machine learning reduced workload for the Cochrane COVID-19 study register: development and evaluation of the Cochrane COVID-19 study classifier. Published online August 29, 2021. doi:10.21203/rs.3.rs-689189/v1
00309-7&id=gr1.jpg){kind=link}