Abstract
Aim
Methods
Results
Conclusion
Keywords
- •There are several limitations with the current evidence synthesis methodologies.
Key findings
- •Recommendations to improve the limitations of the methodologies are fragmented.
- •The development of robust methodologies to cope with the high research output published in response to COVID-19 is imminent.
- •A framework of Artificial Intelligence, international collaboration and funding is necessary.
- •The review provides insight for methodologists to plan for similar situations.
- •Further work is urgently needed in this area
1. Background
- Carollo A.
- Balagtas J.P.M.
- Neoh M.J.-Y.
- Esposito G.
- •Identify the challenges to conducting evidence synthesis during the COVID-19 pandemic
- •Propose recommendations on how to address the identified challenges
2. Methods
2.1 Inclusion criteria
2.2 Search strategy
2.3 Information sources
Haddaway, N.R., Grainger, M.J., Gray, C.T. (2021). citationchaser: an R package for forward and backward citations chasing in academic searching (Version 0.0.3). Retrieved from https://github.com/nealhaddaway/citationchaser
2.4 Data extraction
2.5 Mapping the results
3. Results

3.1 Publication characteristics
Study name | Article description | Methodology | Challenges | Recommendations |
---|---|---|---|---|
[4] | Mapping COVID-19 research | Literature review | Low methodological quality and poor reporting of trials | The need to improve methodology and reporting of studies by having a robust peer review process |
(Alper, Richardson, Lehmann, & Subbian) 2020 | Covid 19-Knowledge accelerator initiative | Commentary | Inefficiencies across multiple steps in generating evidence | The need to have computable evidence |
[6]
A Bibliometric Analysis of COVID-19 across. Sci Social Sci Res Landscape. 2020; 12 (9132-NA)https://doi.org/10.3390/su12219132 | Analysis of COVID-19 research across science and Social Science Research Landscape | Research study | High output of research data addressing COVID 19 pandemic and lack of collaboration between researchers from different disciplines | the need for a complete and in-depth approach that considers various scientific disciplines in COVID-19 research to benefit not only the scientific community but evidence-based policymaking. |
(Bell) 2021 | Evidence synthesis and COVID | Editorial | Outdated reviews Review process is laborious and slow to complete as new evidence is being added. | The need to have weekly reviews conducted to ensure the currency of the evidence. The need to prioritise topics for updating of the evidence |
[9] | Investigation of the presence of publication bias in COVID-19 studies | Research study | Reporting of only positive studies | Pre-registration of studies and public sharing of data for all study types Meta analysis of observational studies should also be undertaken |
(Biesty et al.) 2020 | Investigation of qualitative synthesis methodologies to respond to COVID-19 Pandemic | Discussion paper | Potential criticism about rapid review Concerns about the generalisation of the qualitative evidence Time limitation | The need to produce a Rapid qualitative approach for the Cochrane collaboration to guide decisions based on worked examples and case studies |
3.2 Challenges

Study name | Primary studies | Databases | Team capacity | Process | Resources | Contextual factors |
---|---|---|---|---|---|---|
[4] | X | |||||
(Alper, Richardson, Lehmann, & Subbian) 2020 | X | |||||
[6]
A Bibliometric Analysis of COVID-19 across. Sci Social Sci Res Landscape. 2020; 12 (9132-NA)https://doi.org/10.3390/su12219132 | X | X | ||||
(Bell) 2021 | X | |||||
[9] | X | X | ||||
(Biesty et al.) 2020 | X | |||||
(Chen, Allot, & Lu) 2020 | X | X | ||||
[18]
Using Preprints in Evidence Synthesis: Commentary on experience during the COVID-19 pandemic. J Clin Epidemiol. 2021; https://doi.org/10.1016/j.jclinepi.2021.05.010 | X | X | ||||
[20] | X | |||||
[21] | X | X | ||||
[27] | X | |||||
[30] | X | |||||
[34] | X | X | X | |||
(Hanney, Kanya, Pokhrel, Jones, & Boaz) 2020 | X | X | X | |||
(Murad et al.) 2020 | X | X | ||||
(Nakagawa et al., 2020) | X | X | X | |||
(Negrini, Mg, Côté, & Arienti, 2021) | X | |||||
(Nouri et al., 2020) | X | X | ||||
(Nussbaumer-Streit, Klerings, & Gartlehner, 2020) | X | X | ||||
(Oikonomidi et al., 2020) | X | X | X | |||
(Page et al., 2020) | X | X | ||||
(Palayew et al., 2020) | X | X | X | X | ||
(Ruano, Gómez-García, Pieper, & Puljak, 2020) | X | |||||
(Schünemann et al., 2020) | X | |||||
(Shokraneh & Russell-Rose) 2020 | X | X | ||||
(Stewart et al., 2020) | X | X | X | |||
(Tricco et al., 2020) | X | X | X | |||
(van Schalkwyk et al., 2020) | X |
3.3 Upstream challenges: Evidence production
- Aristovnik A.
- Ravšelj D.
- Umek L.
3.4 Upstream challenges: Evidence management
3.5 ES level challenges: inefficiency of the process
3.6 ES level challenges: Duplication of work
- Aristovnik A.
- Ravšelj D.
- Umek L.
3.7 ES level challenges: Funding
3.8 Downstream challenges: Evidence dissemination/publication
- Clyne B.
- Walsh K.A.
- O'Murchu E.
- Sharp M.K.
- Comber L.
- KK O.B.
- et al.
3.9 Context challenges
4. Discussion
WHO. (2021). Coronavirus disease (COVID-19) pandemic. Retrieved from https://www.who.int/emergencies/diseases/novel-coronavirus-2019
Advances on evidence synthesis | Examples of projects incorporating these advances |
---|---|
Living searches | Epistemonikos, REcmap |
Automation tools | COVID-END, Epistemonikos, COVID-NMA, PAHO Ongoing Living Review |
Prioritization of topics | COVID-END, REcmap |
Crowdsourcing | Cochrane, COVID-NMA, |
Shared platforms | LIT-COVID, REcmap, COVID-NMA, |
Collaboration and Partnership | LIT-COVID, Epistemonikos, REcmap, COVID-NMA, PAHO Ongoing Living Review |
New methodologies for evidence synthesis | Rapid and living reviews |
EPPI-reviewer: EPPI-reviewer: . Retrieved from https://eppi.ioe.ac.uk/cms/Default.aspx?alias=eppi.ioe.ac.uk/cms/er4.
5. Conclusion
Author statement
Acknowledgement
Funding
Appendix. Supplementary materials
References
- Coronavirus disease-19: an interim evidence synthesis of the world association for infectious diseases and immunological disorders (Waidid).Front Med. 2020; 7 (no pagination)
- Developing trustworthy recommendations as part of an urgent response (1–2 weeks): a GRADE concept paper.J Clin Epidemiol. 2021; 129: 1-11https://doi.org/10.1016/j.jclinepi.2020.09.037
- Evidence synthesis 2.0: when systematic, scoping, rapid, living, and overviews of reviews come together.NA. 2020; 123: 162-165https://doi.org/10.1016/j.jclinepi.2020.01.025
- COVID-19 coronavirus research has overall low methodological quality thus far: case in point for chloroquine/hydroxychloroquine.J Clin Epidemiol. 2020; 123: 120-126https://doi.org/10.1016/j.jclinepi.2020.04.016
- It is time for computable evidence synthesis: The COVID-19 Knowledge Accelerator initiative.J Am Med Inform Assoc. 2020; 27: 1338-1339https://doi.org/10.1093/jamia/ocaa114
- A Bibliometric Analysis of COVID-19 across.Sci Social Sci Res Landscape. 2020; 12 (9132-NA)https://doi.org/10.3390/su12219132
Aronson, J.K., Heneghan, C., Mahtani, K.R., Plüddemann, A. (2018). A word about evidence: ‘rapid reviews’ or ‘restricted reviews’?, 23(6), 204-205. doi:10.1136/bmjebm-2018-111025
- Evidence synthesis in the time of COVID-19.Climacteric. 2021; 24: 211-213https://doi.org/10.1080/13697137.2021.1904676
- A metaresearch study revealed susceptibility of Covid-19 treatment research to white hat bias: first, do no harm.J Clin Epidemiol. 2021; 136: 55-63https://doi.org/10.1016/j.jclinepi.2021.03.020
- A QuESt for speed: rapid qualitative evidence syntheses as a response to the COVID-19 pandemic.Syst Rev. 2020; 9: 256https://doi.org/10.1186/s13643-020-01512-5
Boland, L., Kothari, A., McCutcheon, C., Graham, I.D. (2020). Building an integrated knowledge translation (IKT) evidence base: colloquium proceedings and research direction. 18(1), 1-7. doi:10.1186/s12961-019-0521-3
- The COVID-NMA Project: Building an Evidence Ecosystem for the COVID-19 Pandemic.American College of Physicians, 2020
- Future of evidence ecosystem series: 1. Introduction Evidence synthesis ecosystem needs dramatic change.J Clin Epidemiol. 2020; 123: 135-142
- A scientometric approach to review the role of the medial preoptic area (MPOA).Parental Behavior. 2021; 11 (393-NA)https://doi.org/10.3390/brainsci11030393
Carroll, C., Booth, A., Leaviss, J., Rick, J. (2013). “Best fit” framework synthesis: refining the method. 13(1), 37-37. doi:10.1186/1471-2288-13-37
Chen, Q., Allot, A., Lu, Z. (2020). Keep up with the latest coronavirus research. 579(7798), 193-193. doi:10.1038/d41586-020-00694-1
- LitCovid: an open database of COVID-19 literature.Nucleic Acids Res. 2021; 49: D1534-D1540https://doi.org/10.1093/nar/gkaa952
- Using Preprints in Evidence Synthesis: Commentary on experience during the COVID-19 pandemic.J Clin Epidemiol. 2021; https://doi.org/10.1016/j.jclinepi.2021.05.010
- Lessons for the global primary care response to COVID-19: a rapid review of evidence from past epidemics.Family practice, 2021: 15
- Achieving effective informed oversight by DMCs in COVID clinical trials.J Clin Epidemiol. 2020; 126: 167-171https://doi.org/10.1016/j.jclinepi.2020.07.001
- Evidence-based medicine in times of crisis.J Clin Epidemiol. 2020; 126: 164-166https://doi.org/10.1016/j.jclinepi.2020.07.002
- Supporting clinicians during the COVID-19 pandemic.American College of Physicians, 2020
- Use of hydroxychloroquine in hospitalised COVID-19 patients is associated with reduced mortality: Findings from the observational multicentre Italian CORIST study.NA. 2020; 82: 38-47https://doi.org/10.1016/j.ejim.2020.08.019
- Evidence-based medicine in times of crisis.J Clin Epidemiol. 2020; 126: 164-166https://doi.org/10.1016/j.jclinepi.2020.07.002
Dunn, A.G., Bourgeois, F.T. (2020). Is it time for computable evidence synthesis. 27(6), 972-975. doi:10.1093/jamia/ocaa035
EPPI-reviewer: EPPI-reviewer: . Retrieved from https://eppi.ioe.ac.uk/cms/Default.aspx?alias=eppi.ioe.ac.uk/cms/er4.
- Access to data from clinical trials in the COVID-19 crisis: open, flexible, and time-sensitive.J Clin Epidemiol. 2021; 130: 143-146https://doi.org/10.1016/j.jclinepi.2020.10.008
- Bibliometric analysis on COVID-19: a comparison of research between English and Chinese studies.Front Public Health. 2020; 8: 477
Fell, M.J. (2019). The economic impacts of open science: a rapid evidence assessment. 7(3), 46-NA. doi:10.3390/publications7030046
- Rapid reviews for rapid decision-making during the coronavirus disease (COVID-19) pandemic, Norway, 2020.Euro Surveill. 2020; 25: 05https://doi.org/10.2807/1560-7917.ES.2020.25.19.2000687
Fu, D.Y., Hughey, J.J. (2019). Meta-research: releasing a preprint is associated with more attention and citations for the peer-reviewed article. 8(NA), NA-NA. doi:10.7554/elife.52646
Grimshaw, J.M., Tovey, D.I., Lavis, J.N. (2020). COVID-END: an international network to better co-ordinate and maximize the impact of the global evidence synthesis and guidance response to COVID-19.
- Meta-analysis and the science of research synthesis.Nature. 2018; 555: 175-182
Haddaway, N.R., Akl, E.A., Page, M.J., Welch, V., Keenan, C., Lotfi, T. (2020). Open synthesis and the coronavirus pandemic in 2020. 126(NA), 184-191. doi:10.1016/j.jclinepi.2020.06.032
- Open synthesis and the coronavirus pandemic in 2020.J Clin Epidemiol. 2020; 126: 184-191https://doi.org/10.1016/j.jclinepi.2020.06.032
Haddaway, N.R., Grainger, M.J., Gray, C.T. (2021). citationchaser: an R package for forward and backward citations chasing in academic searching (Version 0.0.3). Retrieved from https://github.com/nealhaddaway/citationchaser
Hanney, S., Kanya, L., Pokhrel, S., Jones, T.H., Boaz, A. (2020). What is the evidence on policies, interventions and tools for establishing and/or strengthening national health research systems and their effectiveness? [internet]. NA(NA), NA-NA. doi:NA
- How to strengthen a health research system: WHO's review, whose literature and who is providing leadership?.Health Res Policy Syst. 2020; 18: 72https://doi.org/10.1186/s12961-020-00581-1
- Low-cost dexamethasone reduces death by up to one third in hospitalised patients with severe respiratory complications of COVID-19. 5. RECOVERY Trial Press Release, 2020: 2020 (Available at: https://www.ox.ac.uk/news/2020-06-16-low-cost-dexamethasone-reduces-death-one-thirdhospitalised-patients-severe. Accessed July)
- An evidence-based approach to scoping reviews.Worldviews Evidence-Based Nurs. 2016; 13: 118-123
WHO. (2021). Coronavirus disease (COVID-19) pandemic. Retrieved from https://www.who.int/emergencies/diseases/novel-coronavirus-2019
Article info
Publication history
Footnotes
Declaration of Competing Interests: None to declare. The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.