Abstract
Background
Methods
Results
Conclusion
Keywords
- •Cox regression using a calendar time scale and Poisson regression adjusted for calendar time using restricted cubic splines were unbiased in simulations of COVID-19 vaccine effectiveness and safety.
Key Findings:
- •The highly time-varying incidence of SARS-CoV-2 infection and the rapid uptake of COVID-19 vaccines may bias analyses regarding COVID-19 vaccine effectiveness
- •When using calendar time as the underlying time scale in Cox regression, the researcher does not need to make any parametric assumptions regarding the instantaneous risk for the outcome of interest.
What this adds to what was known?
- •Observational analyses regarding COVID-19 vaccine effectiveness or safety should adjust for calendar time, preferably using flexible methods such as restricted cubic splines or by using calendar time as the underlying time scale in Cox regression.
- •If Cox regression is used in such analyses, the underlying time scale should be stated explicitly.
What is the implication and what should change now?
1. Introduction
- Haas E.J.
- Angulo F.J.
- McLaughlin J.M.
- Anis E.
- Singer S.R.
- Khan F.
- et al.
2. Methods
2.1 Simulation
2.2 Exposure
2.3 Outcomes
- 1)An effectiveness outcome with a high cumulative incidence proportion at the end of the year (10%) resembling the outcome of documented SARS-CoV-2 infection. For this, we obtained daily case numbers of testing positive for SARS-CoV-2 in Denmark during 2021 [[8]], divided these by the total size of the adult population (4.7 million) and adjusted the daily risk to sum up to a cumulative incidence proportion of 0.1 on day 365.
- Ritchie H.
- Mathieu E.
- Rodés-Guirao L.
- Appel C.
- Giattino C.
- Ortiz-Ospina E.
- et al.
Coronavirus pandemic (COVID-19).https://ourworldindata.org/covid-casesDate accessed: February 3, 2022 - 2)For the safety outcome, we simulated a rare event that exhibits seasonal variation with a winter peak. Examples of such outcomes are myocarditis or Guillain-Barré syndrome which have previously been associated with COVID-19 immunization [[2],9,10,11,12]. In detail, we obtained the daily risk of the outcome based on a normal distribution with a mean of 30 days and a standard deviation of 50 days, which was added to a constant baseline rate. To convert the obtained densities to a daily risk, the density for each day was standardized to sum up to one and then multiplied by the desired cumulative incidence proportion of 1/5,000. As the described normal distribution provided densities for days before day 0 (January 1st) but practically no densities for the end of the year (day 330-365), we considered the densities for days before day 0 to occur during the end of the year, i.e., the density for day −1 was set to day 364, the density for day −2 to 363, and so on. This creates a wrap around of the daily hazard. For a graphical depiction of the daily hazard and cumulative incidence of the simulated outcomes and exposure between day 1 and 365, see Figure 1.Fig. 1Daily simulated hazard and simulated cumulative incidence of the effectiveness outcome, safety outcome and vaccination for the study period among unvaccinated individuals.Fig. 2Mean incidence rate ratios (IRRs) and hazard ratios obtained by analyzing simulated data. Bold horizontal lines represent the point estimate ±1 empirical standard error (SE), thin lines the point estimate ±2 SE. The dashed line represents the true, simulated incidence rate ratio. ∗ Using a time under observation scale.
2.4 Follow-up
2.5 Scenarios
- Hansen C.H.
- Schelde A.B.
- Moustsen-Helm I.R.
- Emborg H.D.
- Krause T.G.
- Mølbak K.
- et al.
2.6 Estimators
2.7 Empirical data analysis
- Haas E.J.
- Angulo F.J.
- McLaughlin J.M.
- Anis E.
- Singer S.R.
- Khan F.
- et al.
3. Results
3.1 Simulation


Estimator | Mean | Bias (95% UI) | Log bias (MCSE) | SE (MCSE) | RMSE (95% UI) | Coverage, % (MCSE) |
---|---|---|---|---|---|---|
Outcome: Effectiveness (IRR=0.1) | ||||||
Poisson | 0.26 | 0.16 (0.15, 0.16) | 0.94 (0.00) | 0.02 (0.00) | 0.94 (0.94, 0.94) | 0 (0) |
Poisson, adjusted | 0.10 | 0.00 (−0.00, 0.00) | 0.00 (0.00) | 0.02 (0.00) | 0.02 (0.02, 0.02) | 95 (1) |
Cox | 0.38 | 0.28 (0.28, 0.28) | 1.33 (0.00) | 0.02 (0.00) | 1.33 (1.32, 1.33) | 0 (0) |
Cox, adjusted | 0.13 | 0.03 (0.03, 0.03) | 0.23 (0.00) | 0.12 (0.00) | 0.26 (0.26, 0.27) | 54 (2) |
Cox, calendar time | 0.10 | −0.00 (−0.00, 0.00) | −0.00 (0.00) | 0.02 (0.00) | 0.02 (0.02, 0.02) | 95 (1) |
Cox, matching | 0.10 | −0.00 (−0.00, 0.00) | −0.00 (0.00) | 0.04 (0.00) | 0.04 (0.04, 0.04) | 95 (1) |
Outcome: Effectiveness (IRR=0.5) | ||||||
Poisson | 1.26 | 0.76 (0.76, 0.76) | 0.93 (0.00) | 0.01 (0.00) | 0.93 (0.93, 0.93) | 0 (0) |
Poisson, adjusted | 0.50 | −0.00 (−0.00, −0.00) | −0.00 (0.00) | 0.01 (0.00) | 0.01 (0.01, 0.01) | 95 (1) |
Cox | 1.94 | 1.44 (1.43, 1.44) | 1.35 (0.00) | 0.01 (0.00) | 1.35 (1.35, 1.35) | 0 (0) |
Cox, adjusted | 0.62 | 0.12 (0.11, 0.12) | 0.21 (0.00) | 0.07 (0.00) | 0.22 (0.22, 0.22) | 10 (1) |
Cox, calendar time | 0.50 | 0.00 (−0.00, 0.00) | 0.00 (0.00) | 0.01 (0.00) | 0.01 (0.01, 0.01) | 95 (1) |
Cox, matching | 0.50 | 0.00 (−0.00, 0.00) | 0.00 (0.00) | 0.02 (0.00) | 0.02 (0.02, 0.02) | 95 (1) |
Outcome: Effectiveness (IRR=1.0) | ||||||
Poisson | 2.50 | 1.50 (1.50, 1.50) | 0.91 (0.00) | 0.01 (0.00) | 0.92 (0.91, 0.92) | 0 (0) |
Poisson, adjusted | 0.99 | −0.01 (−0.01, −0.00) | −0.01 (0.00) | 0.01 (0.00) | 0.01 (0.01, 0.01) | 93 (1) |
Cox | 3.94 | 2.94 (2.94, 2.94) | 1.37 (0.00) | 0.01 (0.00) | 1.37 (1.37, 1.37) | 0 (0) |
Cox, adjusted | 1.21 | 0.21 (0.20, 0.21) | 0.19 (0.00) | 0.05 (0.00) | 0.19 (0.19, 0.20) | 4 (1) |
Cox, calendar time | 1.00 | 0.00 (−0.00, 0.00) | 0.00 (0.00) | 0.01 (0.00) | 0.01 (0.01, 0.01) | 95 (1) |
Cox, matching | 1.00 | 0.00 (−0.00, 0.00) | 0.00 (0.00) | 0.02 (0.00) | 0.02 (0.02, 0.02) | 96 (1) |
Outcome: Safety (IRR=1.0) | ||||||
Poisson | 0.58 | −0.42 (−0.43, −0.41) | −0.55 (0.01) | 0.23 (0.01) | 0.59 (0.58, 0.61) | 31 (1) |
Poisson, adjusted | 1.05 | 0.05 (0.03, 0.08) | 0.05 (0.01) | 0.37 (0.01) | 0.38 (0.36, 0.39) | 95 (1) |
Cox | 0.55 | −0.45 (−0.46, −0.44) | −0.59 (0.01) | 0.23 (0.01) | 0.64 (0.62, 0.65) | 24 (1) |
Cox, adjusted | 1.02 | 0.02 (−0.06, 0.10) | 0.01 (0.04) | 1.27 (0.03) | 1.27 (1.21, 1.33) | 94 (1) |
Cox, calendar time | 1.04 | 0.04 (0.01, 0.06) | 0.04 (0.01) | 0.39 (0.01) | 0.39 (0.37, 0.41) | 95 (1) |
Cox, matching | 0.95 | −0.05 (−0.15, 0.05) | −0.06 (0.05) | 1.73 (0.04) | 1.73 (-, 2.58) | 97 (1) |
Outcome: Safety (IRR=2.0) | ||||||
Poisson | 1.16 | −0.84 (−0.85, −0.83) | −0.54 (0.01) | 0.18 (0.00) | 0.57 (0.56, 0.58) | 14 (1) |
Poisson, adjusted | 2.01 | 0.01 (−0.03, 0.05) | 0.01 (0.01) | 0.31 (0.01) | 0.31 (0.30, 0.33) | 96 (1) |
Cox | 1.12 | −0.88 (−0.90, −0.87) | −0.58 (0.01) | 0.18 (0.00) | 0.61 (0.60, 0.62) | 11 (1) |
Cox, adjusted | 2.13 | 0.13 (0.01, 0.26) | 0.06 (0.03) | 0.95 (0.02) | 0.95 (0.90, 0.99) | 95 (1) |
Cox, calendar time | 2.06 | 0.06 (0.01, 0.10) | 0.03 (0.01) | 0.34 (0.01) | 0.34 (0.33, 0.36) | 96 (1) |
Cox, matching | 2.10 | 0.10 (0.04, 0.17) | 0.05 (0.02) | 0.51 (0.01) | 0.51 (0.48, 0.54) | 98 (0) |
Outcome: Safety (IRR=5.0) | ||||||
Poisson | 2.91 | −2.09 (−2.12, −2.07) | −0.54 (0.00) | 0.15 (0.00) | 0.56 (0.55, 0.57) | 6 (1) |
Poisson, adjusted | 4.72 | −0.28 (−0.36, −0.20) | −0.06 (0.01) | 0.27 (0.01) | 0.27 (0.26, 0.29) | 95 (1) |
Cox | 2.85 | −2.15 (−2.18, −2.12) | −0.56 (0.00) | 0.15 (0.00) | 0.58 (0.57, 0.59) | 5 (1) |
Cox, adjusted | 5.33 | 0.33 (0.12, 0.55) | 0.06 (0.02) | 0.66 (0.01) | 0.66 (0.63, 0.69) | 95 (1) |
Cox, calendar time | 5.12 | 0.12 (0.02, 0.22) | 0.02 (0.01) | 0.31 (0.01) | 0.31 (0.29, 0.32) | 96 (1) |
Cox, matching | 5.36 | 0.36 (0.21, 0.52) | 0.07 (0.01) | 0.45 (0.01) | 0.46 (0.43, 0.49) | 97 (1) |
Outcome: SARS-CoV-2 infection | Outcome: Myocarditis | |||
---|---|---|---|---|
BNT162b2 | Unvaccinated | mRNA-1273 | Unvaccinated | |
Individuals (N) | 261,964 | 417,076 | 488,616 | 4,224,150 |
Person years | 42,512 | 196,641 | 72,861 | 1,786,933 |
Events | 1,311 | 21,367 | 18 | 92 |
Incidence rate/1000 PY | 31 | 109 | 0.25 | 0.05 |
Estimators | ||||
Poisson | 0.28 (0.27-0.30) | (ref.) | 4.80 (2.90-7.95) | (ref.) |
Poisson, adjusted | 0.13 (0.12-0.14) | (ref.) | 3.45 (1.96-6.06) | (ref.) |
Cox | 0.40 (0.38-0.43) | (ref.) | 7.84 (3.62-17.0) | (ref.) |
Cox, adjusted | 0.22 (0.15-0.32) | (ref.) | NR | (ref.) |
Cox, calendar time | 0.14 (0.13-0.15) | (ref.) | 2.65 (1.42-4.93) | (ref.) |
Cox, matching | 0.13 (0.11-0.14) | (ref.) | NR | (ref.) |
3.2 Empirical data analysis
4. Discussion
- Haas E.J.
- Angulo F.J.
- McLaughlin J.M.
- Anis E.
- Singer S.R.
- Khan F.
- et al.
- Haas E.J.
- Angulo F.J.
- McLaughlin J.M.
- Anis E.
- Singer S.R.
- Khan F.
- et al.
5. Conclusions
Supplementary data
- Supplementary
References
- Impact and effectiveness of mRNA BNT162b2 vaccine against SARS-CoV-2 infections and COVID-19 cases, hospitalisations, and deaths following a nationwide vaccination campaign in Israel: an observational study using national surveillance data.Lancet. 2021; 397: 1819-1829
- SARS-CoV-2 vaccination and myocarditis or myopericarditis: population based cohort study.BMJ. 2021; 375: e068665
- Israel’s rapid rollout of vaccinations for COVID-19.Isr J Health Policy Res. 2021; 10: 6
- Statistics » COVID-19 Vaccinations.(Available at:)https://www.england.nhs.uk/statistics/statistical-work-areas/covid-19-vaccinations/Date accessed: March 8, 2023
- Changing characteristics over time of individuals receiving COVID-19 vaccines in Denmark: A population-based descriptive study of vaccine uptake.Scand J Public Health. 2022; 50: 686-692
- Use of alternative time scales in Cox proportional hazard models: implications for time-varying environmental exposures.Statist Med. 2012; 31: 3320-3327
- Modern epidemiology.(Fourth edition) Wolters Kluwer/Lippincott Williams & Wilkins, Philadelphia2021
- Coronavirus pandemic (COVID-19).(Available at:)https://ourworldindata.org/covid-casesDate accessed: February 3, 2022
- Neurological complications after first dose of COVID-19 vaccines and SARS-CoV-2 infection.Nat Med. 2021; 27: 2144-2153
- Risks of myocarditis, pericarditis, and cardiac arrhythmias associated with COVID-19 vaccination or SARS-CoV-2 infection.Nat Med. 2022; 28: 410-422
- SARS-CoV-2 vaccination and myocarditis in a nordic cohort study of 23 million residents.JAMA Cardiol. 2022; 7: 600-612
- Association between covid-19 vaccination, SARS-CoV-2 infection, and risk of immune mediated neurological events: population based cohort and self-controlled case series analysis.BMJ. 2022; 376: e068373
- Vaccine effectiveness against SARS-CoV-2 infection with the Omicron or Delta variants following a two-dose or booster BNT162b2 or mRNA-1273 vaccination series: A Danish cohort study.(Available at:)https://www.medrxiv.org/content/early/2021/12/23/2021.12.20.21267966Date accessed: May 12, 2022
- Effectiveness of mRNA BNT162b2 COVID-19 vaccine up to 6 months in a large integrated health system in the USA: a retrospective cohort study.Lancet. 2021; 398: 1407-1416
- Regression Modeling Strategies: With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis.Springer International Publishing, Cham2015
- The active comparator, new user study design in pharmacoepidemiology: historical foundations and contemporary application.Curr Epidemiol Rep. 2015; 2: 221-228
- Immortal time bias in pharmaco-epidemiology.Am J Epidemiol. 2008; 167: 492-499
- Using simulation studies to evaluate statistical methods.Stat Med. 2019; 38: 2074-2102
- The Danish civil registration system.Scand J Public Health. 2011; 39: 22-25
- Representatives the MB of. The Danish microbiology database (MiBa) 2010 to 2013.Eurosurveillance. 2014; 19: 20667
- The Danish vaccination register.Euro Surveill. 2012; 17: 20155
- Safety and Efficacy of the BNT162b2 mRNA covid-19 vaccine.New Engl J Med. 2020; 383: 2603-2615
- Efficacy and safety of the mRNA-1273 SARS-CoV-2 vaccine.New Engl J Med. 2021; 384: 403-416
- Effectiveness of BNT162b2 vaccine against delta variant in adolescents.N Engl J Med. 2021; 385: 2101-2103
- BNT162b2 mRNA covid-19 vaccine in a nationwide mass vaccination setting.N Engl J Med. 2021; 384: 1412-1423
Article info
Publication history
Publication stage
In Press Journal Pre-ProofFootnotes
Funding: None.
Conflicts of interest: Lars Christian Lund reports participation in research projects funded by Menarini Pharmaceuticals and LEO Pharma, all with funds paid to the institution where he was employed (no personal fees) and with no relation to the current work. Anton Pottegård and Jesper Hallas report participation in research projects funded by Alcon, Almirall, Astellas, Astra-Zeneca, Boehringer-Ingelheim, Novo Nordisk, Servier and LEO Pharma, all regulator-mandated phase IV-studies, all with funds paid to the institution where he was employed (no personal fees) and with no relation to the work reported in this paper. Henrik Støvring has received personal consulting fees from Bristol-Myers-Squibb, Novartis, Roche, Merck and Pfizer outside the submitted work. He has received teaching fees from Atrium. Morten Andersen declares that he has previously participated in research projects funded by Pfizer, Janssen, AstraZeneca, H Lundbeck and Mertz, and Novartis with grants received by Karolinska Institutet (no personal fees); he has personally received fees for teaching from Atrium, the Danish Association of the Pharmaceutical Industry; the Pharmacovigilance Research Center is supported by a grant from the Novo Nordisk Foundation (NNF15SA0018404) to the University of Copenhagen.
Ethical approval: This study was approved by the local data protection board at the University of Southern Denmark (SDU-RIO), approval no. 11.247, and the Danish health data authority, project number FSEID-00005447. According to Danish law, ethical approval is not necessary for studies conducted using only registry data.
Author contributions: Lars Christian Lund: Conceptualization, Methodology, Software, Validation, Formal analysis, Data curation, Writing–Original Draft, Writing–Review & Editing, and Visualization. Henrik Støvring: Conceptualization, Methodology, Software, Validation, Formal analysis, Writing–Original Draft, Writing–Review & Editing, and Supervision. Anton Pottegård: Methodology, Writing–Original Draft, Writing–Review & Editing, and Supervision. Morten Andersen: Methodology, Writing–Original Draft, Writing–Review & Editing, and Supervision. Jesper Hallas: Conceptualization, Methodology, Writing–Original Draft, Writing–Review & Editing, and Supervision.
Identification
Copyright
User license
Creative Commons Attribution (CC BY 4.0) |
Permitted
- Read, print & download
- Redistribute or republish the final article
- Text & data mine
- Translate the article
- Reuse portions or extracts from the article in other works
- Sell or re-use for commercial purposes
Elsevier's open access license policy