If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Epistemonikos Foundation, Santiago, ChileUnidad de investigación en medicina estomatológica preventiva y social (UIMEPS), Universidad del Magdalena, Santa Marta, Colombia
Epistemonikos Foundation, Santiago, ChileUC Evidence Center, Cochrane Chile Associated Center, Pontificia Universidad Católica de Chile, Santiago, ChileInternal Medicine Department, Faculty of Medicine, Pontificia Universidad Católica de Chile, Santiago, Chile
The COVID-19 L·OVE is a free access repository and classification platform.
•
The comprehensiveness and currency of the repository were assessed.
•
We compared it against all the studies included in the SRs published during April 2021.
•
The COVID-19 L·OVE repository is highly comprehensive and current.
•
This evaluation addressed only primary studies and not the other types of articles.
Abstract
Background and Objective
The coronavirus disease 2019 Living OVerview of Evidence (COVID-19 L·OVE) is a public repository and classification platform for COVID-19 articles. The repository contains more than 430,000 articles as of September 20, 2021 and intends to provide a one-stop shop for COVID-19 evidence. Considering that systematic reviews conduct high-quality searches, this study assesses the comprehensiveness and currency of the repository against the total number of studies in a representative sample of COVID-19 systematic reviews.
Methods
Our sample was generated from all the studies included in the systematic reviews of COVID-19 published during April 2021. We estimated the comprehensiveness of COVID-19 L·OVE repository by determining how many of the individual studies in the sample were included in the COVID-19 L·OVE repository. We estimated the currency as the percentage of studies that was available in the COVID-19 L·OVE repository at the time the systematic reviews conducted their own search.
Results
We identified 83 eligible systematic reviews that included 2,132 studies. COVID-19 L·OVE had an overall comprehensiveness of 99.67% (2,125/2,132). The overall currency of the repository, that is, the proportion of articles that would have been obtained if the search of the reviews was conducted in COVID-19 L·OVE instead of searching the original sources, was 96.48% (2,057/2,132). Both the comprehensiveness and the currency were 100% for randomized trials (82/82).
Conclusion
The COVID-19 L·OVE repository is highly comprehensive and current. Using this repository instead of traditional manual searches in multiple databases can save a great amount of work to people conducting systematic reviews and would improve the comprehensiveness and timeliness of evidence syntheses. This tool is particularly important for supporting living evidence synthesis processes.
The COVID-19 L OVE repository is highly comprehensive and current. Using as a reference the total number of studies included in a representative sample of systematic reviews, the overall comprehensiveness and currency of the repository were 99.62% and 96.48%, respectively. Both the comprehensiveness and the currency were 100% for randomised trials.
What this adds to what is known?
•
The COVID-19 L·OVE repository can be safely used as the sole source for studies in any COVID-19 topic.
What is the implication, what should change now?
•
The COVID-19 L·OVE can speed up access to evidence without sacrificing quality and therefore encourage timely evidence-informed decisions.
1. Introduction
Researchers from all over the world have been rapidly working to respond to the pandemic. As a result, an overabundance of scientific articles is making it difficult for healthcare professionals, policy makers, journalists, and the general public to keep pace with the body of knowledge about severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) and coronavirus disease 2019 (COVID-19) [
Identifying all the articles relevant to a specific purpose requires sifting through multiple electronic databases. Owing to the urgent need to share new findings fast, articles are frequently shared on preprint servers, so these must also be consulted [
]. However, these resources seem to have a very low level of use. Indeed, it is not clear if they have actually facilitated access to the information or just added to the number of sources that researchers and others must sift through [
One of the reasons why these COVID-19 resources might be underused is because users distrust that they contain all the relevant information. One way of testing this is by comparing exhaustive searches, like those conducted in systematic reviews, against searches using these resources [
In June 2020 Epistemonikos Foundation launched COVID-19 Living OVerview of Evidence (L·OVE), a free access repository and classification platform for COVID-19 evidence available at https://app.iloveevidence.com/covid19 (the repository is available through the advanced search interface) (Fig. 1).
Fig. 1COVID-19 L·OVE website. (1) The COVID-19 L·OVE repository (https://app.iloveevidence.com/covid19/repository) is accessible through the ‘advanced search’ interface. An empty search retrieves all the available records. (2) The classification platform allocates the records to the different terms or a combination of terms relevant to COVID-19. The performance of this component is not addressed in this article. (3) The latest version of the methods and the updated report of the results are provided on the website.
The systematic methods used to build and maintain this resource, the high level of automation used to retrieve the articles and the large number of sources that are harvested makes it one of the largest COVID-19 repositories available. As of September 20, 2021, it contains 434,659 articles (Fig. 2).
Fig. 2Cumulative number of articles identified in the COVID-19 L·OVE repository. The COVID-19 L·OVE repository was built through multiple iterations that included the addition of relevant sources, the refinement of the search strategies used for each source, and the application of technological developments to improve the harvesting process from each source. Modifications that translated into substantial upload of articles are shown in the figure.
One of the main purposes of the COVID-19 L·OVE repository is to replace the need to search multiple sources of COVID-19 evidence. Considering the high quality of searches conducted in systematic reviews, we designed this study to assess how comprehensive and current the repository is against a reference standard composed of the totality of studies included in a representative sample of COVID-19 systematic reviews.
2. Methods
2.1 Methods used to build and maintain the repository
The COVID-19 L·OVE is based on two interrelated components: a repository and a classification platform. The latest version of the methods used in each of these components is available in the methods section of the COVID-19 L·OVE website (https://app.iloveevidence.com/covid19/methods).
Given that this article only covers the methods and performance of the repository component of the COVID-19 L·OVE, we briefly summarize the said methods, as of September 2021, here:
The COVID-19 L·OVE repository was built, and is maintained, by systematic searches of 42 databases, trial registries, and preprint servers. Searches are not restricted by study design, language, or publication status. A full list of sources and the frequency of the searches is available in Appendix 1.
We adapted our main COVID-19 Boolean strategy (Appendix 1) to the syntax of each source. The information is obtained from the sources using different technology solutions, such as querying publicly available application programming interfaces, subscribing to RDF site summary feeds, parsing comma-separated values files posted on websites, and running traditional manual searches.
To identify articles that an electronic search could potentially miss, we manually check all the systematic reviews and other types of evidence syntheses (e.g., overviews of systematic reviews, scoping reviews, guidelines) and add all articles included in those. In addition, we evaluate potentially eligible articles that users send by e-mail and other means (e.g., Twitter).
As randomized trials are particularly relevant for decision-making, we also run a weekly search for randomized trials on Twitter using the terms #COVID19 OR #COVID-19 OR #COVID_19 OR #COVID randomized OR randomized, and scan relevant scientific conferences, press release websites, and the websites of the main trials or companies relevant to COVID-19. The complete list of sources is available in Appendix 1.
All the articles retrieved by the electronic searches are assessed by two automated classifiers specifically developed for this project. The first classifier is a binary exact-match classifier based on a continuously updated list of terms obtained by applying the Word2vec technology with proprietary software developed by Epistemonikos to the corpus of documents available in the repository [
]. The terms with more similar vectors are analyzed by a team of content and methods experts and are selected based on their incremental recall (i.e., their capacity to identify new ‘positives’ in the unclassified records). The second classifier combines a highly specific COVID-19 Boolean strategy with the publication date of the articles (year 2020 or more recent).
The articles excluded by the classifier are not checked. However, any time an article is identified by another means (e.g., a study included in a systematic review), the methods team checks for the presence of any term that can be added to the search strategy or the list of terms used by the exact-match classifier.
The articles included by the classifiers are screened by the COVID-19 L·OVE users, collaborators, or methods team (e.g., during collective screening of the classification platform). Articles are only included if they directly address an issue concerning COVID-19 or the indirect consequences of COVID-19 (e.g., the consequences of lockdown). We do not include COVID-19 articles that might be relevant but were conducted in different contexts (e.g., telemedicine before the COVID-19 pandemic and facemasks for influenza).
2.2 Methods to assess the performance of the repository
2.2.1 Sample
The sampling method is based on the relative recall method, where a sample of primary studies from published systematic reviews is used as a reference standard to evaluate the performance [
]. Our sample was composed of all the primary studies, available as journal article or preprint, included in the systematic reviews with a publication date during April 2021 that were identified in the COVID-19 L·OVE platform. As we identified the same articles from multiple sources, we used the oldest date available as ‘publication date’, typically, the date at the journal's website.
We considered eligible any review that:
•
Fulfilled the definition of systematic review used in the Epistemonikos database [
Addressed a question directly relevant to COVID-19. We excluded reviews addressing an issue broader than COVID-19. That is, reviews including studies of COVID-19 and other conditions (e.g., other coronaviruses) or using indirect evidence (e.g., evidence from previous pandemics).
•
Clearly reported the search date.
•
Provided the list of included studies.
•
Only included studies published before the search date reported in the review.
We extracted the following data from the systematic reviews: authors, title, type of article, date of publication, date of the search, and list of included studies (as per the definition of primary study used in Epistemonikos database [
]). If a review listed an article that was not a primary study under ‘included studies’, we did not add it to the sample. From the primary studies we extracted authors, title, date of publication, type of article (e.g., preprint, journal), and study design.
2.2.2 Comprehensiveness
To evaluate the comprehensiveness (sensitivity or recall) of COVID-19 L·OVE, we determined if the primary studies in the sample were contained within the repository at the time that COVID-19 L·OVE detected the systematic review. All the studies included in a specific review that were available in the repository before the detection date were defined as being contained in the COVID-19 L·OVE repository. The studies that entered the repository after this date were defined as not being contained because the missing studies might have been added after the list of studies included in the systematic review was checked manually, which is part of the search strategy of COVID-19 L·OVE.
We calculated comprehensiveness as:
2.2.3 Currency
We defined currency as the percentage of references that was available in the COVID-19 L·OVE repository at the time of the review search in comparison to the total number of references in the sample.
In other words, our definition of currency is the proportion of references that would have been obtained if the search of the review was conducted in COVID-19 L·OVE instead of searching the original sources.
We calculated currency as:
2.2.4 Audit
To understand the reasons for the failure or delay in the identification of articles, we conducted an audit of all references that were not contained in the repository or were not contained at the search date of the reviews. That is, the studies that were not added to the numerator of the comprehensiveness and currency calculation, respectively.
3. Results
3.1 Description of the sample
We identified 405 potentially eligible systematic reviews published during April 2021. We excluded 322 reviews because they were protocols (n = 14), addressed a population broader than COVID-19 (n = 269), or did not clearly report the last search date (n = 39). Our final sample was generated from 83 eligible systematic reviews. The details about the reviews used to build the sample are available in Appendix 2. The 83 reviews included 2,683 studies overall. After removing 533 duplicates and 18 studies based on unpublished data only, the final sample resulted in 2,132 studies. The selection process is summarized in Figure 3.
Fig. 3Adapted PRISMA flow diagram summarizing the sample selection process.
From the 83 systematic reviews in our sample, 66 (79.52%) were journal articles, seven (8.43%) were preprints, and one (1.20%) was an Health Technology Assessment report. The search date of the reviews ranged from April 8, 2020 to April 21, 2021 (median = November 21, 2020) and the number of studies included in each review ranged from two to 350 (median = 21). The 2,132 primary studies in the sample corresponded to 82 (3.85%) randomized trials and 2,050 (96.15%) studies of other designs. There were 2,016 (94.56%) journal articles and 116 (5.44%) preprints.
3.2 Comprehensiveness
Based on our sample, the overall comprehensiveness of the COVID-19 L·OVE repository was 99.67% (2,125/2,132).
Only seven of 2,132 studies in the sample were not contained in the repository. The missing seven studies were from two systematic reviews. Six of them were observational studies in Chinese from a review of COVID-19–related pressure injuries [
An audit of these seven studies showed that all of them were available in Chinese databases that were in the list of sources searched by COVID-19 L·OVE. However, retrieval from Chinese databases is one of the few processes not fully automated in the repository, so the most likely explanation for their omission was human error.
The COVID-19 L·OVE repository had perfect comprehensiveness for randomized trials (100%, 82/82) and very high comprehensiveness for other types of studies (99.66%, 2,043/2,050). The coverage was very high for journal articles (99.65%, 2,009/2,016) and perfect for preprints (100%, 116/116). The details are presented in Table 1.
Table 1Comprehensiveness of the COVID-19 L·OVE repository
Based on our sample, the overall currency of the COVID-19 L·OVE repository was 96.48% (2,057/2,132).
All the randomized trials in the sample (82/82) and 96.34% (1,975/2,050) of the other types of study were available in the repository when the reviews were searched. The currency was 97.17% (1,959/2,016) and 84.48% (98/116) for journal articles and preprints, respectively. The details are presented in Table 1.
An audit of the 75 articles that entered COVID-19 L·OVE with a delay showed that 27 of 75 (36%) were not available in the sources of COVID-19 L·OVE at the time of the review search. These were all articles that were available on the journal's website but were not yet indexed in the electronic databases harvested by COVID-19 L·OVE. Most reviews capturing these studies reported a search in Google or Google Scholar or used strategies to complement the electronic searches, which is the most likely manner in which they captured these studies.
Thirteen of 75 (17.33%) references were not available in COVID-19 L·OVE at the time of the search but were entered in the following 2 days. This delay is due to the frequency of the searches of the sources that feed the COVID-19 L·OVE repository, which varies from daily to weekly.
Five references of 75 (6.67%) were added to the repository, although they were unavailable in any electronic sources used to maintain the repository. They entered because they were referred to in reviews that COVID-19 L OVE captured.
The manual addition of studies referenced by reviews is one of the strategies used to feed the repository, but this process is prone to delays because it depends on manual work. Sixteen of 75 (21.33%) references corresponded to preprints from searches conducted before July 2020, when automated searches in preprint servers were not fully deployed in the COVID-19 L·OVE repository.
Finally, 14 of 75 (18.67%) references were entered into the repository with a substantial delay because of different technical issues in the retrieval system. All these issues have been identified and solved at the time of writing this article.
4. Discussion
This formal evaluation of the COVID-19 L·OVE repository was based on a large and representative sample of more than 2,000 studies from all the eligible systematic reviews published during a whole month.
Our main conclusion is that the comprehensiveness and currency of the repository range from very high to perfect for all types of primary studies released as journal articles or preprints. It is particularly remarkable that the coverage and currency for randomized trials was 100%. In practical terms, our results show that the COVID-19 L·OVE repository can be safely used as the sole source for studies in a broad range of COVID-19 topics.
Our results are in agreement with previous evaluations of the performance of COVID-19 L·OVE. The comprehensiveness of the COVID-19 L·OVE repository was assessed by researchers from the COVID-network meta-analysis initiative. This evaluation demonstrated that COVID-19 L·OVE identified 100% of the randomized trials and observational studies that were identified through the initial extensive search strategy, which included electronic databases, preprint servers, and several other COVID-19 resources [
]. A comparative analysis of the studies included in multiple systematic reviews found that among 25 systematic reviews addressing one specific question, they included 17 primary studies overall. All of them were contained in the COVID-19 L·OVE platform and there were 11 additional primary studies that were not identified by any of the reviews [
The available information on the performance of other COVID-19 resources is very limited. For most resources we only know the number of articles they include. We know, for instance, that as of September 20, 2021 there were 355,746 records in the World Health Organization COVID-19 database [
]. But using the number of articles as a proxy for the comprehensiveness is unreliable because the differences might be explained by the use of different inclusion criteria, which could lead, for example, to the inclusion of nonscientific records (e.g., news articles) and the retrieval of non-COVID literature.
As far as we know, only two studies have assessed another secondary COVID-19 resource. One evaluation showed that the overall comprehensiveness of the Cochrane COVID-19 Study Register was only 77.2% and there were substantial issues with currency, especially in relation to preprints [
]. Another study reported that the same register identified 88% of the randomized trials reports, 90% of the randomized trials protocols, and 82% of the observational studies reports [
One limitation of our study is that a sample obtained by the relative recall method might not be representative of the total number of existing studies [
]. Considering the high level of standardization of systematic reviews it is possible they all cover much of the same territory. An evaluation against a sample derived from a manual review of journals and other sources might provide a more reliable estimate [
]. However, this approach may not be suitable in the context of the deluge of scientific information about COVID-19.
Another limitation of our evaluation is the scope of the assessment. We addressed only primary studies and not the other types of scientific articles that are contained in the COVID-19 L·OVE repository, which includes any type of scientific article. Considering the inclusive nature of the methods used to maintain the COVID-19 L·OVE repository, we can expect similar results for the other types of articles, but a formal evaluation would provide a definitive answer. Also, it is important to point out that for some research questions, particularly within complex topics, it might be appropriate to search in sources that are not regularly searched for to build and maintain the COVID-19 L·OVE repository.
Our study is not designed to assess the specificity of the repository and the performance and usability of COVID-19 L·OVE search interfaces nor any of the components of the COVID-19 classification platform. We expect to further develop and formally evaluate all these aspects, which are a key to increase the reliability of the search processes and to promote the adoption of this resource.
6. Implications
Accessing all the available studies for a particular topic is a key to avoiding being misled by research [
]. Unfortunately, substantial time and resources are needed to comprehensively identify all the evidence, as required by rigorous systematic review methods [
]. More importantly, it can speed up access to evidence without sacrificing quality and therefore encourage timely evidence-informed decisions.
The community of researchers producing systematic reviews and other types of evidence synthesis has been overwhelmed by the outpouring of new research [
]. A resource like COVID-19 L·OVE can facilitate the production and update of systematic reviews and make it possible to sustain living systematic reviews of COVID-19 [
Finally, the implications of this project extend beyond COVID-19. The replication of this approach in other areas would significantly improve access to scientific evidence while reducing research waste [
G.R., F.V., C.V., J.V., and C.A. lead the project. G.R., C.V., and F.V. devised and improved the search strategy of the repository. C.V., J.V., and I.J. developed the different software needed for the project. J.C., J.C., V.J., M.L., M.M., A.R., P.R., K.S., I.S., P.Z., and G.Z. screened and selected the systematics reviews and identified their included primary studies. P.R. and C.A. provided methodological support. G.R., F.V.P., and C.V. analysed the data. F.V.P. and G.R. wrote the first draft of the article. All authors commented on and approved the final manuscript.
Patient and Public Involvement: There was no patient and public involvement in the whole process of conducting this research.
Data sharing statement: The datasets used and analysed during the present study or datasets needed to reproduce the results of this study (e.g., a list of included/excluded records) are available from the corresponding author on reasonable request.
Funding: This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
Declaration of interests: The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: COVID-19 L·OVE was developed and is maintained by Epistemonikos Foundation, a nonprofit organization with a strict policy to avoid financial conflicts of interest. Details about this policy and financial support can be found on the website (http://www.epistemonikos.cl/). All authors, as founders, board members, developers, or contributors, have some degree of academic conflicts of interest with this article.
00117-2&id=gr1.jpg){kind=link}
00117-2&id=gr2.jpg){kind=link}
00117-2&id=gr3.jpg){kind=link}