Advertisement

Substantial delays in clinical data published by the European Medicines Agency – a cross sectional study

  • Asger Sand Paludan-Müller
    Correspondence
    Corresponding author. Tel.: +45 6550 4809.
    Affiliations
    Cochrane Denmark and Centre for Evidence-Based Medicine Odense (CEBMO), Department of Clinical Research, University of Southern Denmark, Odense, Denmark

    Open Patient data Exploratory Network (OPEN), Odense University Hospital, Odense, Denmark
    Search for articles by this author
  • Ingrid Rose Maclean-Nyegaard
    Affiliations
    Cochrane Denmark and Centre for Evidence-Based Medicine Odense (CEBMO), Department of Clinical Research, University of Southern Denmark, Odense, Denmark

    Open Patient data Exploratory Network (OPEN), Odense University Hospital, Odense, Denmark
    Search for articles by this author
  • Klaus Munkholm
    Affiliations
    Cochrane Denmark and Centre for Evidence-Based Medicine Odense (CEBMO), Department of Clinical Research, University of Southern Denmark, Odense, Denmark

    Open Patient data Exploratory Network (OPEN), Odense University Hospital, Odense, Denmark
    Search for articles by this author
Open AccessPublished:March 12, 2022DOI:https://doi.org/10.1016/j.jclinepi.2022.02.004

      Highlights

      • Clinical data packages released by the EMA contain Clinical Study Reports (CSRs).
      • CSRs can increase the transparency and validity of interpretations of clinical trials.
      • The time to publication of packages was overall much longer than the EMAs own deadlines.
      • No package was published within EMAs deadlines following their standard procedure.
      • The time to publication was much shorter for Covid-19 related packages than for others.

      Abstract

      Background

      Reporting bias poses a fundamental threat to the transparency and validity of interpretations of clinical trials, which may, in part, be mitigated through access Clinical Study Reports (CSRs). The European Medicines Agency (EMA), under their Policy 0070, prospectively publishes clinical data, including CSRs, submitted as part of marketing authorization applications or post-authorization procedures, although this practice is currently suspended for non-COVID-19 medicines, and have set out planned timelines for publication.

      Methods

      We conducted a cross-sectional study assessing the content and characteristics of all clinical data packages released by the EMA under Policy 0070 and the time to their publication. We extracted the number and characteristics of trials included in the clinical packages, assessed the delay to publication relative to the EMAs planned timeline and whether it differed between the EMAs various transparency measures and types of application procedures.

      Results

      We identified 148 clinical data packages that contained data on a total of 1,005 clinical trials, of which 261 (26%) were labelled as phase 3 trials. Full CSRs were available for 913 (90•8%) of the trials. The median time to publication was 511 (IQR 411 to 574) days. Only 2 (1•4%) of the clinical data packages were published within the EMA's planned timeline. The delay was shorter for clinical data packages released under the EMAs transparency measures for COVID.19 medicines compared with their standard transparency measure.

      Conclusion

      The clinical data packages released by the EMA under Policy 0070 contained CSRs on many trials but were published with considerable delays relative to the timeline set forth by the EMA, reducing their potential impact on reporting bias.

      Keywords

      Abbreviations:

      CSR (Clinical study report), EC (European Commission), EMA (European Medicines Agency), IQR (Interquartile range)
      What is new?
      • Clinical Study Reports (CSRs) and other regulatory data can potentially reduce reporting bias but can be difficult to obtain; however, the clinical data packages published by the EMA contain large amounts of data of relevance for public health and are an important source of information in research synthesis.
      • Publication times were generally much longer than the EMA's own timeline, however, delays were shorter for data packages published under the EMAs transparency measures for COVID-19 medicines compared with those published under their standard transparency measure.
      • Resuming publication of all clinical data and in a timely manner, should be a priority, as this will help researcher reach valid conclusions about benefits and harms of treatments.

      1. Introduction

      There is convincing evidence that reporting bias, the selective publication of entire studies [
      • Schmucker C
      • Schell LK
      • Portalupi S
      • Oeller P
      • Cabrera L
      • Bassler D
      • et al.
      Extent of non-publication in cohorts of studies approved by research ethics committees or included in trial registries.
      ] as well as the selective reporting of outcomes [
      • Chan A-W
      • Hróbjartsson A
      • Haahr MT
      • Gøtzsche PC
      • Altman DG
      Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles.
      ] poses a fundamental threat to the transparency, validity and reproducibility of published results of clinical trials. By distorting the evidence base for drugs and other interventions, reporting bias prevents fully informed decisions about patient care and is a major contributor to research waste [
      • Glasziou P
      • Altman DG
      • Bossuyt P
      • Boutron I
      • Clarke M
      • Julious S
      • et al.
      Reducing waste from incomplete or unusable reports of biomedical research.
      ].
      Trial registries and public availability of trial protocols may reduce the risk of reporting bias by providing information on planned trials. Additionally, trial registries may contain outcome data from completed trials. However, the information about the results available in registries is insufficient just as is the case for published reports when evaluating clinical trials [
      • Wieseler B
      • Kerekes MF
      • Vervoelgyi V
      • McGauran N
      • Kaiser T
      Impact of document type on reporting quality of clinical drug trials: a comparison of registry reports, clinical study reports, and journal publications.
      ,
      • Paludan-Müller AS
      • Créquit P
      • Boutron I.
      Reporting of harms in oncological clinical study reports submitted to the European medicines agency compared to trial registries and publications—a methodological review.
      ], and study protocols are often not publicly available [
      • Lucey M
      • Clark J
      • Glasziou P.
      Public availability of trial protocols.
      ]. The most complete reports of the planning, conduct and results of randomized trials of drugs are the full, unabridged Clinical Study Reports (CSRs) compiled by pharmaceutical companies as part of their application for marketing approval with drug regulatory agencies [
      • Chan A-W
      • Song F
      • Vickers A
      • Jefferson T
      • Dickersin K
      • Gøtzsche PC
      • et al.
      Increasing value and reducing waste: addressing inaccessible research.
      ,
      • Doshi P
      • Jefferson T.
      Clinical study reports of randomised controlled trials: an exploratory review of previously confidential industry reports.
      ]. Containing the full study protocol as well as the most comprehensive account of the results of trials [
      • Wieseler B
      • Kerekes MF
      • Vervoelgyi V
      • McGauran N
      • Kaiser T
      Impact of document type on reporting quality of clinical drug trials: a comparison of registry reports, clinical study reports, and journal publications.
      ,
      • Maund E
      • Tendal B
      • Hróbjartsson A
      • Jørgensen KJ
      • Lundh A
      • Schroll J
      • et al.
      Benefits and harms in clinical trials of duloxetine for treatment of major depressive disorder: comparison of clinical study reports, trial registries, and publications.
      ,
      • Schroll JB
      • Penninga EI
      • Gøtzsche PC.
      Assessment of adverse events in protocols, clinical study reports, and published papers of trials of orlistat: a document analysis.
      ], CSRs may thus, at least in part, mitigate some of the concerns related to reporting bias [
      • Chan A-W
      • Song F
      • Vickers A
      • Jefferson T
      • Dickersin K
      • Gøtzsche PC
      • et al.
      Increasing value and reducing waste: addressing inaccessible research.
      ,
      • Jefferson T
      • Doshi P
      • Boutron I
      • Golder S
      • Heneghan C
      • Hodkinson A
      • et al.
      When to include clinical study reports and regulatory documents in systematic reviews.
      ].
      As of October 2016, acknowledging the need for transparency and that access to clinical data will benefit public health, the European Medicines Agency (EMA) has published clinical data, including CSRs, based on their flagship policy on publication of clinical data for medicinal products for human use (Policy 0070), which was the first of its kind worldwide and came into effect on January 1, 2015 [
      • Ferran J-M
      • Nevitt SJ.
      European medicines agency Policy 0070: an exploratory review of data utility in clinical study reports for academic research.
      ]. The policy outlines the EMA's intention to prospectively release clinical data, defined as clinical overviews, clinical summaries, clinical study reports, protocols, protocol amendments, sample case report forms, and documentation of statistical methods and, additionally, individual patient data submitted to the EMA under the centralized marketing authorization procedure [

      European Medicines Agency. European medicines agency policy on publication of clinical data for medicinal products for human use - Policy 0070. 2019. https://www.ema.europa.eu/en/documents/other/european-medicines-agency-policy-publication-clinical-data-medicinal-products-human-use_en.pdf.

      ]. All clinical data submitted as part of a marketing authorization application or as part of a post-authorization procedure for an existing centrally authorized medicinal product falls within the scope of Policy 0070 [

      European Medicines Agency. European medicines agency policy on publication of clinical data for medicinal products for human use - Policy 0070. 2019. https://www.ema.europa.eu/en/documents/other/european-medicines-agency-policy-publication-clinical-data-medicinal-products-human-use_en.pdf.

      ].
      The EMA suspended the release of clinical data on December 5, 2018, due to the relocation of the Agency from London to Amsterdam, meaning that only applications submitted before August 2018 were processed [

      European Medicines Agency. Clinical data publication. European Medicines Agency. 2018. https://www.ema.europa.eu/en/human-regulatory/marketing-authorisation/clinical-data-publication (accessed 29 Jun 2020).

      ]. In late 2020, applying measures to increase the transparency of its regulatory activities on treatments and vaccines for COVID-19, the EMA began releasing clinical data for submissions concerning pharmaceutical treatments and vaccines for COVID-19; however, the publication of clinical data pertaining to other medicines remains suspended [].
      The EMA aims to publish clinical data within 60 days of the European commission (EC) decision for procedures that concern marketing authorization-, line extension- or extension applications, and within 150 days for procedures concerning withdrawn applications and so-called Article 58 applications [

      European Medicines Agency. Clinical data publication. 2018. https://www.ema.europa.eu/en/human-regulatory/marketing-authorisation/clinical-data-publication (accessed 19 Mar 2021).

      ]. However, a recent study that accessed clinical data released under Policy 0070 for targeted therapy and immunotherapy for cancer found considerable delays in publication [
      • Paludan-Müller AS
      • Créquit P
      • Boutron I.
      Reporting of harms in oncological clinical study reports submitted to the European medicines agency compared to trial registries and publications—a methodological review.
      ]. For the 37 accessed clinical study reports, the median delay between the EC's decision and publication was 1.21 years (range 0.91-1.78) [
      • Paludan-Müller AS
      • Créquit P
      • Boutron I.
      Reporting of harms in oncological clinical study reports submitted to the European medicines agency compared to trial registries and publications—a methodological review.
      ]. The time to publication of all clinical data released by the EMA under Policy 0070, however, is unknown. We therefore wished to assess the time to publication for all released clinical data packages under Policy 0070 and to assess the characteristics of the packages and the clinical trials they contain data on.

      2. Methods

      We registered a protocol at the Open Science Framework prior to commencing the research project, which is available at https://osf.io/7rd46, and reported the study according to the STROBE guideline.

      2.1 Identification of packages and additional data

      Clinical data packages and corresponding information were obtained by combining information from three related sources.
      We initially searched the EMA clinical data website (https://clinicaldata.ema.europa.eu/web/cdp/home) (last search August 10, 2021) to identify and download all clinical data packages released by the EMA under policy 0070.
      We then searched the ECs Union Register of Medicinal Products (https://ec.europa.eu/health/documents/community-register/html/index_en.htm) for the product names concerned in the identified clinical data packages, whereafter information on the EC's decision date pertaining to the specific clinical data package was accessed.
      We lastly searched the EMA register of medicines (https://www.ema.europa.eu/en/medicines) to identify the specific procedure to which each individual clinical data package was related. We then looked at European public assessment reports and other documents to determine which indication the procedure concerned; where multiple indications were listed; we assigned the indication we judged to be most relevant to the procedure.

      2.2 Data extraction

      We extracted data into various pre-piloted electronic spreadsheets using a three-tiered process.
      First, one researcher (ASP-M) extracted the following data from all the identified clinical data packages: the name of the active substance and the corresponding ATC code, the marketing authorization holder, the number of documents released, the type of approval procedure, the EMA procedure number designated each clinical data package, the indication concerned, the clinical specialty concerned based on the indication for the drug relevant to the package, the EMA publication date, and the date of the EC's decision on marketing authorization relevant to each clinical data package.
      Second, one researcher (ASP-M) used a Microsoft Windows™ command prompt script to extract the names of all folders and files contained in the clinical data packages. We determined the number of different types of documents in the clinical data packages by counting the number of files in the folders “clinical overview”, “clinical summary”, and “clinical study reports” for each procedure. We then examined the files in the ‘clinical study reports’ folder of all clinical data packages and noted all individual clinical trials for which files were available.
      Third, for each individual trial, one researcher (ASP-M, IRM-N, or KM) used the available documents to extract the following information for all clinical trials contained in the data packages: trial ID; trial development phase; whether the trial was randomized; whether the trial was placebo-controlled; whether any available clinical study report was an interim report of an ongoing trial, abridged, or full; clinicaltrials.gov identifier, EudraCT identifier, Universal trial number; and number of pages of the clinical study report. We did not extract information from in-vitro studies, animal studies, observational studies, or pooled analyses of multiple trials.

      2.3 Analysis

      We described the characteristics of the included clinical data packages and the clinical trials they pertained to. For each clinical data package, we calculated both the raw time from the date of the EC's decision to publication of the package and the delay time representing the time elapsed from the EMA planned publication time to publication. When calculating delay time, we took the various timelines for different approval procedures into consideration. For clinical data packages published after the EMA planned timeline, we analyzed whether the delay time differed according to a) the type of EMA transparency measure (i.e., whether the package was released under the EMAs standard procedure or under the exceptional measures for Covid-19 medicines), b) the type of EMA approval procedure or c) the medical specialty concerned. Differences between two groups were tested with the Mann-Whitney test. Differences between three or more groups were tested with the Kruskal-Wallis test, which, in case of statistically significant differences between groups were followed by pair-wise Mann-Whitney tests corrected for multiple testing using Bonferroni correction. P-values were two-tailed, and we used a P < 0.05 threshold for statistical significance.
      We used the freely available statistical software R [

      R Core Team. R: A language and environment for statistical computing. Vienna, Austria: : R Foundation for Statistical Computing 2019. https://www.R-project.org/.

      ] for all analyses.

      2.4 Patient and public involvement

      No patients were involved in the planning or conduct of this study.

      3. Results

      3.1 Characteristics and contents of clinical data packages

      We identified and included 148 clinical data packages released by the EMA. 141 (95.3%) of the packages were released during the period October 2018 to December 2018 and 7 (4.7%) of the packages were released during the period October 2020 to July 2021; all the latter packages were released as part of the EMAs transparency measure for COVID-19 medicines. No packages were released between December 2018 and October 2020. One additional data package, for the drug migalastat, was initially released but has since been withdrawn due to current litigation and was therefore not included. The clinical data packages concerned six different types of approval procedures: 84 (56.8%) were concerning initial marketing authorizations, 45 (30.4%) concerned extensions of indication, 5 (3.4%) concerned line extensions, one (0.7%) concerned an Article 58 application (i.e., an application for use of medicines outside the EU), 2 (1.4%) were labelled as “workshare” (i.e., an application that aims to change multiple approvals) and 11 (7.4%) of the packages concerned withdrawn applications. The clinical data packages were related to 18 different specialties, the most common of which were oncology (n = 46 (31.1%)) and infectious diseases (n = 29 (19.6%)). Details on the remaining specialties can be seen in Table 1. Six drugs appeared in more than one package (Humira, n = 4; Opdivo, n = 4; Imbruvica, n = 2; Keytruda, n = 2; Kyprolis, n = 2; and Veklury, n = 2). For ATC codes, drug classes, and indications for included drugs, see Supplemental Figure 1 and our full datasets available online.
      Table 1Characteristics of included packages.
      Initial marketing authorisation (N (%))Extension of indication (N (%))Line extension (N (%))Work-share (N (%))Withdrawn (N (%))Article 58 (N (%))Total (N (%))
      Oncology23 (15.5)19 (12.8)1 (0.7)None3 (2.0)None46 (31.1)
      Infectious diseases21 (14.2)4 (2.7)1 (0.7)None3 (2.0)None29 (19.6)
      Endocrinology5 (3.4)4 (2.7)1 (0.7)2 (1.4)NoneNone12 (8.1)
      Cardiology9 (6.1)2 (1.4)NoneNoneNoneNone11 (7.4)
      Rheumatology5 (3.4)3 (2.0%)NoneNone2 (1.4)None10 (6.8)
      Neurology4 (2.7)2 (1.4%)NoneNone1 (0.7)None7 (4.7)
      Gastroenterology3 (2)2 (1.4)1 (0.7)NoneNoneNone6 (4.1)
      Hematology5 (3.4)1 (0.7%)NoneNoneNoneNone6 (4.1)
      Immunology1 (0.7)3 (2.0%)NoneNone1 (0.7)None5 (3.4)
      Dermatology1 (0.7)2 (1.4%)NoneNoneNoneNone3 (2.0)
      OphthalmologyNone3 (2.0%)NoneNoneNoneNone3 (2.0)
      Pulmonology3 (2.0)NoneNoneNoneNoneNone3 (2.0)
      Nephrology2 (1.4)NoneNoneNoneNoneNone2 (1.4)
      PsychiatryNoneNone1 (0.7)None1 (0.7)None2 (1.4)
      Obstetrics1 (0.7)NoneNoneNoneNoneNone1 (0.7)
      PediatricsNoneNoneNoneNoneNone1 (0.7)1 (0.7)
      Radiology1 (0.7)NoneNoneNoneNoneNone1 (0.7)
      Total84 (56.8)45 (30.4)5 (3.4)2 (1.4)11 (7.4)1 (0.7)148 (100.0)
      N, Number.
      The clinical data packages contained a total of 7279 documents. The median (IQR) number of documents for packages was 18 (IQR 8 to 39). The packages contained a total of 198 files filed in the folder Clinical Overview, 448 files in the folder Clinical Summary, 6482 files in the folder Clinical Study Reports, and 151 files in the folder Anonymization Report. Additional details on the types of files in the packages are available in Appendix 1.
      By examining files in the Clinical Study Reports folder, we identified a total of 1005 unique clinical trials, of which 940 figured in only one package, 45 figured in two packages, and 20 figured in three packages; 605 (60.2%) of the trials were randomized, 262 (26.1%) were placebo-controlled and 261 (26%) were phase three trials (the clinical development phases of all trials can be seen in Supplemental Table 1). The 1005 trials were contained in 124 clinical data packages while the remaining 24 data packages contained either no clinical study reports folder or the files in the folder did not describe any clinical trials. For the 124 packages with any trials, the median number of trials was 4 (interquartile range (IQR) two to eight), ranging from one to 98 trials and the median number of randomized trials was 3 (IQR 1 to 6), ranging from 1 to 67 trials. Eighty-eight (59.4%) of the packages contained at least one phase three study and the median number of phase 3 studies per package was 2 (IQR 1 to 4), ranging from one to 36 studies .
      Full CSRs were available for 913 (90.8%) of the identified trials, abridged CSRs were available for 42 (4.2%) of the trials, and interim CSRs were available for 27 (2.7%) of the trials. No CSR was available for the remaining 25 (2.5%) of the trials. A clinical trial registry ID (either clinicaltrials.gov, EUCTR, or UTN) was available in CSRs for 198 (19.7%) of the trials.

      3.2 Time to clinical data publication

      For two of the clinical data packages, it was not possible to determine the EC decision date as the procedures could not be identified in the EC's Union Register of Medicinal Products. For the remaining 146 packages, the median overall time from the EC's decision to clinical data publication was 511 (IQR 411 to 574) days. The median time to publication was 523 (IQR 425 to 575) days for the 139 packages released under EMA's standard procedure and 119 (IQR 61 to 160) days for the seven packages released under their transparency measures for COVID-19 medicines. The time to publication by type of procedure can be seen in Table 2. Only two (1.4%) of the packages, both released under the EMA's transparency measures for COVID-19 medicines, were published within the EMA's planned timeline; one concerned an initial marketing authorization application, which was published 55 days after the EC decision, and one concerned a withdrawn application published 49 days after publication of the withdrawal letter. The time to publication appears to have increased over time since the implementation of Policy 0070 (Supplemental Figure 2). The time to clinical data publication for each EMA transparency measure is shown in Fig. 1 and the distribution of time to publication can be seen in Fig. 2. For the 144 packages that were published after the EMA planned timeline, the delay corresponded to a median of 446 (IQR 352 to 505) days relative to EMA's planned timeline. The delayed packages encompassed a total of 996 trials of which 405 were labelled as either phase 2, 2/3 or 3 trials (Fig. 3). CSRs were available for 974 trials published after the EMA's planned timeline and 908 of those were full CSRs (Fig. 2). There was a statistically significant difference in the delay time between packages released under EMA's transparency measures for COVID-19 medicines (median 77 (IQR 59-122) days) and those released according to EMA's standard practice (median 450 (IQR 364-508) days) (W = 0, P = 0.002). The delay time to publication by approval procedure is shown in Supplemental Figure 3. The Kruskal-Wallis test suggested the time to publication differed between types of approval procedures overall (Χ2 = 12.5, df = 5, P = 0.03) and when excluding packages released under EMA's transparency measures for COVID-19 medicines (Χ2 = 12.0, df = 5, P = 0.04). However, post-hoc pair-wise analyses with Bonferroni correction were not statistically significant in either case (all Padjust > 0.26 and Padjust > 0.32, respectively). The delay time to clinical data publication for each specialty is shown in Supplemental Figure 4. The delay time to publication neither differed between clinical specialties overall ((Χ2 = 20.3, df = 16, P = 0.21) nor when excluding clinical data packages released under EMA's transparency measures for COVID-19 medicines (Χ2 = 17.8, df = 16, P = 0.33).
      Table 2Time to publication and delay by procedure category.
      Application procedure typeTime to publicationDelay relative to EMA timeframe
      NMedian days (IQR)NMedian days (IQR)
      Initial marketing authorisation84470 (377-554)83412 (324-494)
      Extension of indication43508 (453-595)43448 (393-535)
      Line extension5564 (554-630)5504 (494-570)
      Workshare2574 (574-574)2514 (514-514)
      Withdrawn11580 (503-616)10443 (387-466)
      Article 581544 (544-544)1394 (394-394)
      N, number; IQR, Interquartile range.
      Fig 1
      Fig. 1Time to publication for each EMA transparency measure (For interpretation of the references to color in this figure legend, the reader is referred to the Web version of this article.)
      Fig 2
      Fig. 2Distribution of time to publication of clinical data packages (For interpretation of the references to color in this figure legend, the reader is referred to the Web version of this article.)
      Fig 3
      Fig. 3Number of delayed trials (For interpretation of the references to color in this figure legend, the reader is referred to the Web version of this article.)

      4. Discussion

      We here investigated the time to publication for all clinical data packages released by the EMA under their Policy 0070, and their characteristics. We found that among a total of 148 clinical data packages, the median time to publication was nearly one and a half years overall and that only two clinical data packages released under the EMA's transparency measures for COVID-19, and none released under their standard transparency measures, were published within the time frame outlined by the EMA. The clinical data packages published after the EMA's planned timeline encompassed nearly a thousand clinical trials, most of which were reported in full CSRs.
      Our study is the first to include all clinical data packages published by the EMA under their Policy 0070. A previous study assessed the delays in publication of clinical data for targeted therapy and immunotherapy trials for cancer only and found that delays were substantial [
      • Paludan-Müller AS
      • Créquit P
      • Boutron I.
      Reporting of harms in oncological clinical study reports submitted to the European medicines agency compared to trial registries and publications—a methodological review.
      ]. Although such an approach could theoretically have given rise to selection bias, our finding of comparable publication delays across specialties indicates this was not likely. By including all clinical data packages, not a sample, we nonetheless eliminated any risk of selection bias and thus were able to show that the issue of substantial delays in publication of clinical data appears to be a general issue.
      Our findings have several implications. First, our findings show that the EMA does not meet their planned timeline for clinical data publication. In no instances were clinical data processed according to their standard transparency procedure published within their timeline and the delays were substantial. As the release of data by the EMA may indeed enable public scrutiny and the development of new knowledge in the interest of public health, the main objectives of the Policy 0070, and ultimately impact patient care and policy making, timely publication is essential. Otherwise unavailable clinical data such as that included in the CSRs released by the EMA may for example provide more accurate or complete information [
      • Wieseler B
      • Kerekes MF
      • Vervoelgyi V
      • McGauran N
      • Kaiser T
      Impact of document type on reporting quality of clinical drug trials: a comparison of registry reports, clinical study reports, and journal publications.
      ] on important outcomes, including harms [
      • Paludan-Müller AS
      • Créquit P
      • Boutron I.
      Reporting of harms in oncological clinical study reports submitted to the European medicines agency compared to trial registries and publications—a methodological review.
      ,
      • Hodkinson A
      • Gamble C
      • Smith CT.
      Reporting of harms outcomes: a comparison of journal publications with unpublished clinical study reports of orlistat trials.
      ], when conducting systematic reviews, the gold standard for assessing the benefits and harms of drugs, reducing the risk of reporting bias [
      • Rohner E
      • Grabik M
      • Tonia T
      • Jüni P
      • Pétavy F
      • Pignatti J
      • et al.
      Does access to clinical study reports from the European Medicines Agency reduce reporting biases? A systematic review and meta-analysis of randomized controlled trials on the effect of erythropoiesis-stimulating agents in cancer patients.
      ]. Delays in their publication may thus mean delays in any impact the clinical data packages may have. The clinical data packages released by the EMA have obvious relevance for research synthesis and, hence, to patients, as well as for policy making; data on nearly a thousand clinical trials of which many were phase-3 studies, were available, and in most cases in the form of a CSR.
      The delay in publication time was considerably smaller for clinical data packages released following EMA's transparency measures for COVID-19 than for packages published following their standard procedure. While this could potentially be due to important differences in the packages released under the different transparency measures, we found no difference in the number of trials or CSR pages per package between transparency measures and no relation between neither the total number of CSR pages or the total number of trials included in each package and the time to its publication (data not shown), indicating this was not the case. A more likely explanation for the shorter publication times is the prioritizing by the EMA to release clinical data for Covid-19 treatments and vaccines []. Other possible explanations include the fact that no other packages needed to be processed and that both the EMA and sponsors had experience with the process. Even for those packages, however, the time to publication exceeded the EMA's planned timeline in most cases. Nonetheless, the finding suggests it is indeed possible to publish clinical data in a timelier manner, close to the specified timeframe, given priority.
      Reducing the time to publication of clinical data should be a priority, not only for data related to COVID-19, but in general. The EMA has effectively not shared data for almost 3 years, initially because of the implementation of the third phase of EMA's ‘business continuity plan’ set in place due to Brexit and the publication of clinical data now remains suspended due to ‘ongoing business continuity linked to the COVID-19 pandemic (https://www.ema.europa.eu/en/human-regulatory/marketing-authorisation/clinical-data-publication/support-industry-clinical-data-publication). Thus, we found that the only clinical data published since December 2018 were seven data packages relevant to COVID-19, and of these, only five concerned initial marketing authorizations. While there may be legitimate reasons for the EMA not to do so, the objectives of the Policy 0070 can of course only be met by publishing clinical data. This is contingent on the EMA resuming prospective publication of clinical data packages in accordance with Policy 0070 and releasing the comparative information for EC decisions regarding approval applications made since the suspension of publication in late 2018.
      Our findings might indicate that the EMAs role in promoting transparency and access to data in a timely manner is currently not given top priority. This assumption is supported by the fact that other researchers have called out the EMA, for example, for failing to ensure that outcome data is reported in the EU clinical trial register in a timely manner [
      • Goldacre B
      • DeVito NJ
      • Heneghan C
      • Irving F
      • Bacon S
      • Fleminger J
      • et al.
      Compliance with requirement to report results on the EU Clinical Trials Register: cohort study and web resource.
      ] and for failing to ensure transparency in device regulation [
      • Fraser AG
      • Butchart EG
      • Szymanski P
      • Caiani EG
      • Crosby S
      • Kearney P
      • et al.
      The need for transparency of clinical evidence for medical devices in Europe.
      ].
      A strength of our study was that we examined all the clinical study reports released by the EMA as part of Policy 0070, eliminating selection bias when quantify and characterizing the released data. Also, by retrieving information on all clinical trials contained in the clinical data packages, we were able to provide clinical context and importance to our findings of delayed publication of clinical data by the EMA. A potential limitation of our study is that our sample, although encompassing all published packages under Policy 0070, is relatively small and the timeframe is relatively short. Starting a new policy such as Policy 0070, that requires changes to the practices for both the EMA and sponsors, may to some extent be expected to come with a learning curve, with processing times improving over time. However, we found that the time to publication of packages released under Policy 0070 conversely increased over time. It is possible, of course, that with increased experience, the time to publication will be reduced. It can also be considered a limitation that we only considered packages published by the EMA. Although the EMA has stated in their business continuity plan that all applications submitted before 1 August 2018 are to be processed until completion, it is possible that some of those have not been published yet even though they fall under the scope of Policy 0070. The time to publication in the sample of packages included in our study may thus constitute an underestimate. Another limitation in our study was that we did not extract data in duplicate; we did, however, follow a pre-planned data extraction protocol, employed a pre-piloted data extraction form, and extensively mined our dataset for potential data extraction errors. Our dataset is available online. Finally, our statistical analyses were not pre-specified and should therefore be interpreted with caution.

      5. Conclusion

      The clinical data packages released by the EMA according to Policy 0070 contain large amounts of data of relevance for public health but are published with considerable delay relative to the timeline set forth by the EMA, limiting their potential impact on reporting bias. It should be a priority for the EMA to first resume publication of clinical data according to Policy 0070, which remains suspended except for packages relevant to COVID-19, and to reduce the time to publication of all data.

      Declarations

      Availability of data and materials

      All extracted data and statistical code is available on the Open Science Framework (doi:10.17605/OSF.IO/EHMS7)

      Author's contributions

      Asger Sand Paludan-Müller: Conceptualization, Methodology, Formal Analysis, Investigation, Writing – Original Draft, Writing – Review and Editing. Ingrid Rose Maclean-Nyegaard: Methodology, Investigation, Writing – Review and Editing. Klaus Munkholm: Conceptualization, Methodology, Software, Formal Analysis, Investigation, Writing – Review and Editing, Visualization, Supervision.

      Transparency declaration

      Asger Sand Paludan-Müller affirms that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained

      Appendix. Supplementary materials

      References

        • Schmucker C
        • Schell LK
        • Portalupi S
        • Oeller P
        • Cabrera L
        • Bassler D
        • et al.
        Extent of non-publication in cohorts of studies approved by research ethics committees or included in trial registries.
        PLOS ONE. 2014; 9e114023https://doi.org/10.1371/journal.pone.0114023
        • Chan A-W
        • Hróbjartsson A
        • Haahr MT
        • Gøtzsche PC
        • Altman DG
        Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles.
        JAMA. 2004; 291: 2457-2465https://doi.org/10.1001/jama.291.20.2457
        • Glasziou P
        • Altman DG
        • Bossuyt P
        • Boutron I
        • Clarke M
        • Julious S
        • et al.
        Reducing waste from incomplete or unusable reports of biomedical research.
        Lancet. 2014; 383: 267-276https://doi.org/10.1016/S0140-6736(13)62228-X
        • Wieseler B
        • Kerekes MF
        • Vervoelgyi V
        • McGauran N
        • Kaiser T
        Impact of document type on reporting quality of clinical drug trials: a comparison of registry reports, clinical study reports, and journal publications.
        BMJ. 2012; 344: d8141https://doi.org/10.1136/bmj.d8141
        • Paludan-Müller AS
        • Créquit P
        • Boutron I.
        Reporting of harms in oncological clinical study reports submitted to the European medicines agency compared to trial registries and publications—a methodological review.
        BMC Med. 2021; 19: 1-15https://doi.org/10.1186/s12916-021-01955-0
        • Lucey M
        • Clark J
        • Glasziou P.
        Public availability of trial protocols.
        Lancet. 2017; 390: e54-e55https://doi.org/10.1016/S0140-6736(17)33255-5
        • Chan A-W
        • Song F
        • Vickers A
        • Jefferson T
        • Dickersin K
        • Gøtzsche PC
        • et al.
        Increasing value and reducing waste: addressing inaccessible research.
        The Lancet. 2014; 383: 257-266https://doi.org/10.1016/S0140-6736(13)62296-5
        • Doshi P
        • Jefferson T.
        Clinical study reports of randomised controlled trials: an exploratory review of previously confidential industry reports.
        BMJ Open. 2013; 3https://doi.org/10.1136/bmjopen-2012-002496
        • Maund E
        • Tendal B
        • Hróbjartsson A
        • Jørgensen KJ
        • Lundh A
        • Schroll J
        • et al.
        Benefits and harms in clinical trials of duloxetine for treatment of major depressive disorder: comparison of clinical study reports, trial registries, and publications.
        BMJ. 2014; 348: g3510https://doi.org/10.1136/bmj.g3510
        • Schroll JB
        • Penninga EI
        • Gøtzsche PC.
        Assessment of adverse events in protocols, clinical study reports, and published papers of trials of orlistat: a document analysis.
        PLoS Med. 2016; 13e1002101https://doi.org/10.1371/journal.pmed.1002101
        • Jefferson T
        • Doshi P
        • Boutron I
        • Golder S
        • Heneghan C
        • Hodkinson A
        • et al.
        When to include clinical study reports and regulatory documents in systematic reviews.
        BMJ Evidence-Based Med. 2018; 23: 210-217https://doi.org/10.1136/bmjebm-2018-110963
        • Ferran J-M
        • Nevitt SJ.
        European medicines agency Policy 0070: an exploratory review of data utility in clinical study reports for academic research.
        BMC Med Res Methodol. 2019; 19: 204https://doi.org/10.1186/s12874-019-0836-3
      1. European Medicines Agency. European medicines agency policy on publication of clinical data for medicinal products for human use - Policy 0070. 2019. https://www.ema.europa.eu/en/documents/other/european-medicines-agency-policy-publication-clinical-data-medicinal-products-human-use_en.pdf.

      2. European Medicines Agency. Clinical data publication. European Medicines Agency. 2018. https://www.ema.europa.eu/en/human-regulatory/marketing-authorisation/clinical-data-publication (accessed 29 Jun 2020).

      3. Pinho AC. Transparency: exceptional measures for COVID-19 medicines. European Medicines Agency. 2020. https://www.ema.europa.eu/en/human-regulatory/overview/public-health-threats/coronavirus-disease-covid-19/treatments-vaccines/transparency-exceptional-measures-covid-19-medicines (accessed 22 Mar 2021).

      4. European Medicines Agency. Clinical data publication. 2018. https://www.ema.europa.eu/en/human-regulatory/marketing-authorisation/clinical-data-publication (accessed 19 Mar 2021).

      5. R Core Team. R: A language and environment for statistical computing. Vienna, Austria: : R Foundation for Statistical Computing 2019. https://www.R-project.org/.

        • Hodkinson A
        • Gamble C
        • Smith CT.
        Reporting of harms outcomes: a comparison of journal publications with unpublished clinical study reports of orlistat trials.
        Trials. 2016; 17: 207https://doi.org/10.1186/s13063-016-1327-z
        • Rohner E
        • Grabik M
        • Tonia T
        • Jüni P
        • Pétavy F
        • Pignatti J
        • et al.
        Does access to clinical study reports from the European Medicines Agency reduce reporting biases? A systematic review and meta-analysis of randomized controlled trials on the effect of erythropoiesis-stimulating agents in cancer patients.
        PLoS ONE. 2017; 12e0189309https://doi.org/10.1371/journal.pone.0189309
        • Goldacre B
        • DeVito NJ
        • Heneghan C
        • Irving F
        • Bacon S
        • Fleminger J
        • et al.
        Compliance with requirement to report results on the EU Clinical Trials Register: cohort study and web resource.
        The BMJ. 2018; 362 (In press)https://doi.org/10.1136/bmj.k3218
        • Fraser AG
        • Butchart EG
        • Szymanski P
        • Caiani EG
        • Crosby S
        • Kearney P
        • et al.
        The need for transparency of clinical evidence for medical devices in Europe.
        The Lancet. 2018; 392 (In press): 521-530https://doi.org/10.1016/S0140-6736(18)31270-4