Advertisement

Noninterventional studies in the COVID-19 era: methodological considerations for study design and analysis

  • Anne M. Butler
    Correspondence
    Corresponding author: Division of Infectious Diseases, John T. Milliken Department of Medicine, Washington University School of Medicine, 4523 Clayton Avenue, MSC 8051-0043-15, St. Louis, MO 63110, Tel.: +1 314 273 1975; fax: +1 314 454 5392.
    Affiliations
    Department of Medicine, Division of Infectious Diseases, Washington University School of Medicine, St. Louis, MO, USA

    Department of Surgery, Division of Public Health Sciences, Washington University School of Medicine, St. Louis, MO, USA
    Search for articles by this author
  • Mehmet Burcu
    Affiliations
    Department of Epidemiology, Merck & Co., Inc., Rahway, NJ, USA
    Search for articles by this author
  • Jennifer B. Christian
    Affiliations
    Center for Advanced Evidence Generation, Real World Solutions, IQVIA

    Clinical Epidemiology and Health Services Research, Weill Cornell Medicine
    Search for articles by this author
  • Fang Tian
    Affiliations
    Office of Surveillance and Epidemiology, Center for Drug Evaluation and Research, U.S. Food and Drug Administration, Silver Spring, MD, USA
    Search for articles by this author
  • Kathleen M. Andersen
    Affiliations
    Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA

    Center for Drug Safety and Effectiveness, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA
    Search for articles by this author
  • William A. Blumentals
    Affiliations
    Department of Epidemiology and Benefit-Risk, Sanofi, Cambridge, MA 02142, USA
    Search for articles by this author
  • Karen E. Joynt Maddox
    Affiliations
    Department of Medicine, Division of Cardiology, Washington University School of Medicine, St. Louis, MO, USA

    Center for Health Economics and Policy, Institute for Public Health at Washington University, St. Louis, MO, USA
    Search for articles by this author
  • G. Caleb Alexander
    Affiliations
    Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA

    Center for Drug Safety and Effectiveness, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA

    Division of General Internal Medicine, Johns Hopkins Medicine, Baltimore, MD, USA

    Division of Endocrinology, Johns Hopkins Medicine, Baltimore, MD, USA
    Search for articles by this author
Published:November 15, 2022DOI:https://doi.org/10.1016/j.jclinepi.2022.11.011

      Abstract

      The global COVID-19 pandemic has generated enormous morbidity and mortality, as well as large health system disruptions including changes in use of prescription medications, outpatient encounters, emergency department admissions, and hospitalizations. These pandemic-related disruptions are reflected in real-world data derived from electronic medical records, administrative claims, disease or medication registries, and mobile devices. We discuss how pandemic-related disruptions in healthcare utilization may impact the conduct of noninterventional studies designed to characterize the utilization and estimate the effects of medical interventions on health-related outcomes. Using hypothetical studies, we highlight consequences that the pandemic may have on study design elements including participant selection and ascertainment of exposures, outcomes, and covariates. We discuss the implications of these pandemic-related disruptions on possible threats to external validity (participant selection) and internal validity (for example, confounding, selection bias, missing data bias). These concerns may be amplified in populations disproportionately impacted by COVID-19, such as racial/ethnic minorities, rural residents, or people experiencing poverty. We propose a general framework for researchers to carefully consider during the design and analysis of noninterventional studies that use real-world data from the COVID-19 era.

      Keywords

      What is new?

        Key findings

      • Using hypothetical studies, we highlight consequences that the pandemic may have on study design elements including participant selection and ascertainment of exposures, outcomes, and covariates.

        What this adds to what is known?

      • The authors discuss how pandemic-related disruptions in healthcare utilization may impact the conduct of noninterventional studies designed to characterize the utilization and estimate the effects of medical interventions on health-related outcomes.
      • These concerns may be amplified in studies of populations that have been disproportionately impacted by COVID-19, such as racial/ethnic minorities, rural residents, or people experiencing poverty.

        What is the implication and what should change now?

      • We propose a general framework for researchers to carefully consider during the design and analysis of noninterventional studies that use real-world data from the COVID-19 era.

      1. Introduction

      The global COVID-19 pandemic [] has caused staggering morbidity and mortality as well as substantial disruption to the healthcare sector including loss of health insurance [
      • Agarwal S.D.
      • Sommers B.D.
      Insurance coverage after job loss - the importance of the ACA during the covid-associated recession.
      ,
      • Blumenthal D.
      • Fowler E.J.
      • Abrams M.
      • Collins S.R.
      Covid-19 - implications for the health care system.
      ], decreases in preventive care [
      • Whaley C.M.
      • Pera M.F.
      • Cantor J.
      • Chang J.
      • Velasco J.
      • Hagg H.K.
      • et al.
      Changes in health services use among commercially insured US populations during the COVID-19 pandemic.
      ], deferred elective and specialty care [
      • Jazieh A.R.
      • Akbulut H.
      • Curigliano G.
      • Rogado A.
      • Alsharm A.A.
      • Razis E.D.
      • et al.
      Impact of the COVID-19 pandemic on cancer care: a global collaborative study.
      ,
      • Meredith J.W.
      • High K.P.
      • Freischlag J.A.
      Preserving elective surgeries in the COVID-19 pandemic and the future.
      ], delays in disease diagnoses [
      • Kaufman H.W.
      • Chen Z.
      • Niles J.K.
      • Fesko Y.A.
      Changes in newly identified cancer among US patients from before COVID-19 through the first full year of the pandemic.
      ], changes in patient care-seeking behavior [
      • Alexander G.C.
      • Tajanlangit M.
      • Heyward J.
      • Mansour O.
      • Qato D.M.
      • Stafford R.S.
      Use and content of primary care office-based vs telemedicine care visits during the COVID-19 pandemic in the US.
      ], and reductions in medical utilization and expenditures [
      • McWilliams J.M.
      • Russo A.
      • Mehrotra A.
      Implications of early health care spending reductions for expected spending as the COVID-19 pandemic evolves.
      ]. For example, early in the course of the pandemic, there were increases in telemedicine programs [
      • Patel S.Y.
      • Mehrotra A.
      • Huskamp H.A.
      • Uscher-Pines L.
      • Ganguli I.
      • Barnett M.L.
      Trends in outpatient care delivery and telemedicine during the COVID-19 pandemic in the US.
      ] but decreases in prescription refills [
      • Vaduganathan M.
      • van Meijgaard J.
      • Mehra M.R.
      • Joseph J.
      • O'Donnell C.J.
      • Warraich H.J.
      Prescription fill patterns for commonly used drugs during the COVID-19 pandemic in the United States.
      ,
      • Selke Krulichová I.
      • Selke G.W.
      • Bennie M.
      • Hajiebrahimi M.
      • Nyberg F.
      • Fürst J.
      • et al.
      Comparison of drug prescribing before and during the COVID-19 pandemic: a cross-national European study.
      ], in-person outpatient visits [
      • Baum A.
      • Kaboli P.J.
      • Schwartz M.D.
      Reduced in-person and increased telehealth outpatient visits during the COVID-19 pandemic.
      ], emergency department visits [
      • Sokolski M.
      • Gajewski P.
      • Zymliński R.
      • Biegus J.
      • Berg J.M.T.
      • Bor W.
      • et al.
      Impact of coronavirus disease 2019 (COVID-19) outbreak on acute admissions at the emergency and cardiology departments across europe.
      ,
      • Nijman R.G.
      • Honeyford K.
      • Farrugia R.
      • Rose K.
      • Bognar Z.
      • Buonsenso D.
      • et al.
      Presentations of children to emergency departments across Europe and the COVID-19 pandemic: a multinational observational study.
      ], and hospitalizations relative to prior years [
      • Whaley C.M.
      • Pera M.F.
      • Cantor J.
      • Chang J.
      • Velasco J.
      • Hagg H.K.
      • et al.
      Changes in health services use among commercially insured US populations during the COVID-19 pandemic.
      ,
      • Jacobs E.A.
      • Ogedegbe O.
      • Fihn S.D.
      Elective care and health services research in the COVID-19 era.
      ].
      These pandemic-related disruptions will have important implications for the design and conduct of noninterventional studies that use real-world data, including those derived from electronic medical records (EMRs), administrative claims, disease/medication registries, and mobile devices. Researchers seeking to perform noninterventional studies with real-world data and policymakers seeking to understand them will need to consider how the pandemic may create new challenges and exacerbate perennial challenges inherent to observational research. Given changes in healthcare-seeking behavior during the pandemic, several issues are important to consider in the design and conduct of studies using real-world data from the pandemic era. These concerns may be amplified in studies of populations that have been disproportionately impacted by COVID-19, such as racial/ethnic minorities, rural residents, or people experiencing poverty [
      • Butler S.M.
      Four COVID-19 lessons for achieving health equity.
      ,
      • Metzl J.M.
      • Maybank A.
      • De Maio F.
      Responding to the COVID-19 pandemic: the need for a structurally competent health care system.
      ].
      We focus on the importance of considering pandemic-related healthcare disruptions during the planning stage of noninterventional studies that use real-world data. We use hypothetical study examples to demonstrate the implications of pandemic-related changes on study design elements, and consequently on study validity. We focus on the following study design elements: participant selection, exposure assessment, outcome assessment, covariate assessment, and accounting for competing risks. We discuss the implications of these pandemic-related disruptions on possible threats to external validity (participant selection) and internal validity (for example, confounding, selection bias, missing data bias), and propose considerations and approaches to characterize or mitigate these potential biases (Table 1). We refer broadly to pandemic-related disruptions, but recognize that the timing and magnitude of these disruptions varies by geography and setting of health care delivery–some disruptions subsided early in the pandemic whereas others persisted [
      • Coughlin S.S.
      • Yiğiter A.
      • Xu H.
      • Berman A.E.
      • Chen J.
      Early detection of change patterns in COVID-19 incidence and the implementation of public health policies: a multi-national study.
      ]. The proposed framework extends beyond recent work focused exclusively on using real-world data to study COVID-19-related treatments [
      • Pottegård A.
      • Kurz X.
      • Moore N.
      • Christiansen C.F.
      • Klungel O.
      Considerations for pharmacoepidemiological analyses in the SARS-CoV-2 pandemic.
      ,
      • Franklin J.M.
      • Lin K.J.
      • Gatto N.M.
      • Rassen J.A.
      • Glynn R.J.
      • Schneeweiss S.
      Real-world evidence for assessing pharmaceutical treatments in the context of COVID-19.
      ], and builds upon previous work addressing the use of real-world data to study non–COVID-19-related topics [
      • Reynolds M.W.
      • Petruski-Ivleva N.
      • Samant M.
      • Trivedi V.
      • Dreyer N.A.
      • Graff J.
      • et al.
      The Impact of COVID-19 on Real-World Health Data and Research: Understanding the Implications of the Pandemic on Real-World Health Research: Key Considerations and Solutions [White paper].
      ,
      • Webster-Clark M.
      Ways COVID-19 may impact unrelated pharmacoepidemiologic research using routinely collected data.
      ]. While not exhaustive of all potential study design considerations, our work proposes a general framework reflective of an evolving conceptualization of the conduct of noninterventional studies using real-world data from the COVID-19 era. This manuscript was endorsed by the International Society for Pharmacoepidemiology (ISPE).
      Table 1Considerations and Approaches to Address COVID-19 Pandemic-Induced Threats To Validity
      COVID-19 pandemic-induced considerationsPossible threats to validityApproaches to characterize or mitigate bias
      Participant selection: how has the ability to sample a target population changed over time?
      • Stay-home orders led to missed or delayed healthcare encounters altogether, resulting in underdiagnoses and undertreatment
      • Patient populations who sought treatment via healthcare encounters during pandemic may comprise nonrandom subset of target population (for example, more sick/frail)
      • Some populations may be disproportionately impacted by COVID-19 (for example, racial/ethnic minorities, rural residents, low-income)
      • Changes over time in patient populations (for example, telemedicine, Medicaid-insured)
      • Reduced generalizability (external validity) due to inability to identify representative sample of target population
      • Assess changes over time of baseline characteristics by exposure group, such as pre-COVID-19 vs. COVID-19 eras
      • Account for calendar time via restriction or stratification
      • Use direct standardization (g-formula, adjustment formula) or inverse probability of sampling weights to estimate the population average treatment effect when study sample is not random sample of target population
      • Conduct robust feasibility assessment of database options to select fit-for-purpose data sources, including careful consideration of ability to study historically marginalized populations (overall or subgroup analyses)
      • Leverage additional data sources
        Leveraging additional data sources can enrich existing data in several ways (e.g., capture structured and unstructured data fields; include various types of healthcare encounters and communications such as office visits, telehealth visits, phone/email communications; inclusion of a different population).
      Exposure assessment: can exposures be accurately identified?
      • Stay-home orders resulted in stockpiling of drugs, resulting in limited ability to accurately identify start and stop dates of prescription drugs
      • Increased flexibility in take-home scheduling for scheduled medications
      • Possible decreases in medication adherence due to pandemic-related barriers to health care access
      • Exposure misclassification via inability to identify new-users of drugs or drug discontinuation
      • Exposure misclassification due to nonadherence
      • Missing data bias
      • Assess temporal trends in prevalence of exposure
      • Account for calendar time via restriction or stratification
      • Apply two-stage g-computation designs for handling missing exposure information
      • Conduct sensitivity analyses to vary exposure definitions and quantify potential impact of exposure misclassification
      • Assess adherence using refill gap method, anniversary model, proportion of days covered, medication possession ratio
      • Use inverse probability weighting, instrumental variable estimation, or g-estimation to reduce bias from nonadherence
      • Leverage additional data sources
        Leveraging additional data sources can enrich existing data in several ways (e.g., capture structured and unstructured data fields; include various types of healthcare encounters and communications such as office visits, telehealth visits, phone/email communications; inclusion of a different population).
      Outcome assessment: can outcomes be accurately identified?
      • Incomplete ascertainment of outcomes with lower severity of illness (that did not require healthcare encounter)
      • Prolonged time between visits delay initiation of treatmncompent progression
      • Loss of employment/insurance coverage may result in loss-to-follow-up/missing outcomes
      • Some populations may be disproportionately impacted by COVID-19 (for example, racial/ethnic minorities, rural residents, low-income)
      • Delay in treatment initiation impacts outcomes regardless of treatment
      • Outcome misclassification
      • Selection bias due to differential loss-to-follow-up
      • Missing data bias
      • Consider outcome definitions that are robust to pandemic-related changes in healthcare utilization
      • Assess and describe temporal trends in prevalence of outcome
      • Account for calendar time via restriction, stratification, matching, weighting, or multivariable adjustment
      • Sensitivity analyses varying algorithms to define outcomes and quantify potential impact of outcome misclassification
      • Leverage additional data sources
        Leveraging additional data sources can enrich existing data in several ways (e.g., capture structured and unstructured data fields; include various types of healthcare encounters and communications such as office visits, telehealth visits, phone/email communications; inclusion of a different population).
      Covariate assessment: identifying potential confounders and effect measure modifiers
      • Stay-home orders resulted in missed or delayed healthcare encounters altogether, resulting in the absence of recorded data
      • Increased telemedicine leads to missing data on clinical measurements (for example, height, weight, blood pressure) and routine laboratory results (for example, hemoglobin A1c)
      • Confounding due to unmeasured or poorly measured covariates
      • Confounding and/or effect measure modification by COVID-19 status
      • Missing data bias
      • Assess temporal trends in prevalence of potential confounders
      • Define potential confounders using all-available covariate assessment period
      • Consider strategies for missing confounders such as complete-case analysis, last observation carried forward, the missingness pattern approach, multiple imputation, and inverse-probability-of-missingness weighting
      • Use negative controls to quantify uncontrolled confounding
      • Restrict/stratify study population by COVID-19 infection history
      • Censor patient follow-up on COVID-19 infection diagnosis date
      • Account for COVID-19 status as a confounder via restriction, stratification, matching, weighting, or multivariable adjustment
      • Account for calendar time via restriction, stratification, matching, weighting, or multivariable adjustment
      • Use restriction, stratification, matching, weighting, or multivariable adjustment to account for factors associated with differential impact of COVID-19 (for example, race/ethnicity, income)
      • Sensitivity analyses varying algorithms to define covariates (for example, duration of look-back period) and quantify potential impact of confounding
      • Replicate analyses among different data sources
      • Leverage additional data sources
        Leveraging additional data sources can enrich existing data in several ways (e.g., capture structured and unstructured data fields; include various types of healthcare encounters and communications such as office visits, telehealth visits, phone/email communications; inclusion of a different population).
      Accounting for competing risks
      • COVID-19 resulted in high mortality, particularly in certain subpopulations
      • Mortality is a competing event which precludes occurrence of many outcomes of interest
      • Inflation of risk estimates when competing events are treated as censoring events
      • Assess frequency of competing events
      • Use methods to estimate risk accounting for competing events
      • Compare results to those obtained using analytic simplifications commonly used to handle competing events, such as treating competing events as censoring events
      a Leveraging additional data sources can enrich existing data in several ways (e.g., capture structured and unstructured data fields; include various types of healthcare encounters and communications such as office visits, telehealth visits, phone/email communications; inclusion of a different population).

      2. Participant selection: how has the ability to sample a target population changed over time?

      External validity refers to the extent to which an internally valid effect measured in a study sample is an unbiased estimator of the treatment effect in the target population [
      • Cole S.R.
      • Stuart E.A.
      Generalizing evidence from randomized clinical trials to target populations: the ACTG 320 trial.
      ]. An important component of external validity is generalizability, which concerns making inference from a possibly nonrandom sample back to the target population [
      • Lesko C.R.
      • Buchanan A.L.
      • Westreich D.
      • Edwards J.K.
      • Hudgens M.G.
      • Cole S.R.
      Generalizing study results: a potential outcomes perspective.
      ,
      • Stuart E.A.
      • Cole S.R.
      • Bradshaw C.P.
      • Leaf P.J.
      The use of propensity scores to assess the generalizability of results from randomized trials.
      ,
      • Pearl J.
      Generalizing experimental findings.
      ]. Representative sampling is not required for studies of causal inference [
      • Rothman K.J.
      • Gallacher J.E.
      • Hatch E.E.
      Why representativeness should be avoided.
      ], but is key for interpretation of results with respect to generalizability. During the pandemic, identification of a random sample of the target population using real-world data may be challenging due to substantial changes in patterns of health-seeking behavior, healthcare utilization, and insurance enrollment. For example, systematic differences between the target population and those who sought care through a healthcare encounter, from which the study population is sampled [
      • Jazieh A.R.
      • Akbulut H.
      • Curigliano G.
      • Rogado A.
      • Alsharm A.A.
      • Razis E.D.
      • et al.
      Impact of the COVID-19 pandemic on cancer care: a global collaborative study.
      ,
      • Cutler D.
      How will COVID-19 affect the health care economy?.
      ], may threaten external validity.
      Consider a cohort study using a database with comprehensive pharmacy dispensing data that sought to evaluate the risk of antibiotic treatment failure among women coded for urinary tract infection (UTI) in the outpatient setting and dispensed a same-day antibiotic prescription. Over-the-phone prescriptions likely increased during COVID, and consequently, fewer women would meet study eligibility criteria in some countries due to absence of a UTI code typically assigned during an in-person encounter. If patients with worse symptoms are more likely to seek in-person care, cohort restriction to in-person encounters may result in a sicker study population, who may also be at higher risk of the outcome. This change may result in a different sample average treatment effect than the average treatment effect in the target population.
      Another important consequence of pandemic-related disruptions is their disproportionate effect on specific populations. For example, Medicaid discontinuity decreased during the pandemic due to increased Medicaid enrollment (via unemployment and loss of private insurance) as well as prohibitions on Medicaid disenrollment [
      • Corallo BM S.
      2022;Pages. Accessed at kaiser family foundation.
      ]; consequently, studies of Medicaid-insured individuals may include study populations with different characteristics before vs. after the pandemic. In another example, widespread and urgent implementation of telemedicine programs nationwide in response to the pandemic may have disproportionately served younger, well-resourced patients with stable internet access and comfort with technology [
      • Whaley C.M.
      • Pera M.F.
      • Cantor J.
      • Chang J.
      • Velasco J.
      • Hagg H.K.
      • et al.
      Changes in health services use among commercially insured US populations during the COVID-19 pandemic.
      ,
      • Gmunder K.N.
      • Ruiz J.W.
      • Franceschi D.
      • Suarez M.M.
      Demographics associated with US healthcare disparities are exacerbated by the telemedicine surge during the COVID-19 pandemic.
      ]; consequently, studies of telemedicine may include a low proportion of older adults, racial/ethnic minorities, rural residents, or people experiencing poverty.

      2.1 Approaches to characterize or mitigate bias related to participant selection

      Given concern that the pandemic may result in a lower proportion of in-person healthcare encounters but among a sicker or frailer sample of the population, it is important to assess temporal changes in key patient-level and provider-level characteristics. For example, researchers can investigate changes in severity of illness via proxies, such as receipt of kidney imaging to rule out ascension of the infection from the bladder among patients with UTI. To address changing study population characteristics, analyses may account for calendar time via stratification or other adjustment methods.
      Under certain conditions, direct standardization (via g-formula [
      • Robins J.
      A new approach to causal inference in mortality studies with sustained exposure periods - application to control of the healthy worker survivor effect.
      ] or adjustment formula [
      • Pearl J.
      Causality.
      ,
      • Maldonado G.
      • Greenland S.
      Estimating causal effects.
      ]), or inverse probability of sampling weights, can be used to estimate the population average treatment effect when the study sample is not a random sample of the target population. These estimators use data from the study sample on the exposure—outcome relationship and data from the target population on either (a) the distribution of pretreatment covariates for the g-formula estimator or (b) the sampling probabilities conditional on pretreatment covariates for the inverse probability of sampling estimator [
      • Lesko C.R.
      • Buchanan A.L.
      • Westreich D.
      • Edwards J.K.
      • Hudgens M.G.
      • Cole S.R.
      Generalizing study results: a potential outcomes perspective.
      ]. Details and illustrative examples are available in Lesko et al. [
      • Lesko C.R.
      • Buchanan A.L.
      • Westreich D.
      • Edwards J.K.
      • Hudgens M.G.
      • Cole S.R.
      Generalizing study results: a potential outcomes perspective.
      ].
      When possible, researchers should seek and demand data sources (directly or via data linkages) that enable study of historically marginalized populations. For example, study inclusion of low-income populations or individuals living in residential facilities (for example, jails/prisons) can enable results that generalize to important subpopulations, and ultimately contribute to overcoming a long history of underrepresentation in epidemiology and health services research [
      • Kaviani P.
      • Landi S.N.
      • McKethan A.
      • Brookhart M.A.
      • McGrath L.J.
      Who are we missing? Underrepresentation of data sources used for pharmacoepidemiology research in the United States.
      ]. Robust feasibility assessments of data options are needed to identify and select fit-for-purpose data sources [
      • Gatto N.M.
      • Campbell U.B.
      • Rubinstein E.
      • Jaksa A.
      • Mattox P.
      • Mo J.
      • et al.
      The structured process to identify fit-for-purpose data: a data feasibility assessment framework.
      ]. Additionally, collaborations and initiatives across sectors such as government, clinical practice, epidemiology, and informatics can improve the presence or accuracy of variables (for example, race/ethnicity, socioeconomic status) critical for studying historically marginalized populations.

      3. Exposure assessment: can exposures be accurately identified?

      Typically, studies of medications define the start date of the treatment exposure window based on the dispensing date of the first or second prescription of interest, after requiring a washout period without documentation of drug dispensing. Similarly, discontinuation dates are often defined as the dispensing date of the last prescription date plus the sum of the days’ supply and a grace period [
      • Stürmer T.
      • Wang T.
      • Golightly Y.M.
      • Keil A.
      • Lund J.L.
      • Jonsson Funk M.
      Methodological considerations when analysing and interpreting real-world data.
      ]. However, studies that use real-world data to define treatment exposure are susceptible to bias from unobservable factors including drug initiation, timing of initiation, and adherence. Furthermore, studies reflecting the pandemic era may be susceptible to additional biases related to exposure misclassification given disruptions to prescription drug utilization such as drug stockpiling [
      • Vaduganathan M.
      • van Meijgaard J.
      • Mehra M.R.
      • Joseph J.
      • O'Donnell C.J.
      • Warraich H.J.
      Prescription fill patterns for commonly used drugs during the COVID-19 pandemic in the United States.
      ,
      • Selke Krulichová I.
      • Selke G.W.
      • Bennie M.
      • Hajiebrahimi M.
      • Nyberg F.
      • Fürst J.
      • et al.
      Comparison of drug prescribing before and during the COVID-19 pandemic: a cross-national European study.
      ,
      • Reynolds M.W.
      • Petruski-Ivleva N.
      • Samant M.
      • Trivedi V.
      • Dreyer N.A.
      • Graff J.
      • et al.
      The Impact of COVID-19 on Real-World Health Data and Research: Understanding the Implications of the Pandemic on Real-World Health Research: Key Considerations and Solutions [White paper].
      ], increased flexibility in take-home scheduling (for example, methadone [
      Substance abuse and mental health services administration.
      ]), and decreased adherence to chronic medications [
      • Clement J.
      • Jacobi M.
      • Greenwood B.N.
      Patient access to chronic medications during the Covid-19 pandemic: evidence from a comprehensive dataset of US insurance claims.
      ].
      For example, dates of drug initiation and discontinuation may be less reliable given drug stockpiling. Prescription fills for lisinopril (angiotensin-converting enzyme [ACE] inhibitor) and losartan (angiotensin receptor blocker [ARB]) peaked the week after the March 13, 2020 declaration of a national emergency by the U.S [
      • Vaduganathan M.
      • van Meijgaard J.
      • Mehra M.R.
      • Joseph J.
      • O'Donnell C.J.
      • Warraich H.J.
      Prescription fill patterns for commonly used drugs during the COVID-19 pandemic in the United States.
      ]. In a new-user study of ACE inhibitor users vs. ARB users—restricted to patients who did not use either drug during the washout period—it is plausible that continuous users who stockpiled ACE inhibitors or ARBs may be misclassified as new users. In this scenario, inclusion of prevalent users may introduce prevalent user biases, particularly if the proportion of person-time is substantial among prevalent users. This “depletion of susceptibles” phenomenon occurs because prevalent users only represent the “survivor” subset of all initiators; it excludes individuals who became nonadherent due to experiencing events during the early period of pharmacotherapy, which may lead to substantial bias.

      3.1 Approaches to characterize or mitigate bias related to exposure assessment

      Assessing temporal trends in the prevalence of exposure by calendar time relative to COVID-19 may be useful for identifying the impact of pandemic-related changes in healthcare utilization, such as declines in laboratory tests to inform medication prescribing [
      • Meredith J.W.
      • High K.P.
      • Freischlag J.A.
      Preserving elective surgeries in the COVID-19 pandemic and the future.
      ,
      • Alexander G.C.
      • Tajanlangit M.
      • Heyward J.
      • Mansour O.
      • Qato D.M.
      • Stafford R.S.
      Use and content of primary care office-based vs telemedicine care visits during the COVID-19 pandemic in the US.
      ]. Approaches are also available to address misclassification or partial missingness of exposure. In some settings, restricting the study period to the prepandemic era may reduce misclassification. Also, stratifying analyses by calendar time relative to COVID-19 will provide effect estimates by time period, providing insight into the robustness of results. These analyses can be performed as primary, secondary, or sensitivity analyses, as applicable. To assess the possible impact of drug stockpiling on exposure misclassification, investigators can vary the washout period duration (for example, 6, 12, or 18 months) to identify new users or vary the grace period that defines date of discontinuation. To evaluate the extent of pandemic-related nonadherence to medications (that is, persistence and implementation), investigators can assess individual-level drug dispensing data by applying common methods such as the refill gap method, the anniversary model, the proportion of days covered, and the medication possession ratio [
      • Rasmussen L.
      • Wettermark B.
      • Steinke D.
      • Pottegård A.
      Core concepts in pharmacoepidemiology: measures of drug utilization based on individual-level drug dispensing data.
      ]. In the setting of comparative effectiveness research, inverse probability weighting, g-estimation, and instrumental variable estimation can reduce bias introduced by nonadherence [
      • Hernán M.A.
      • Hernández-Díaz S.
      Beyond the intention-to-treat in comparative effectiveness research.
      ].
      There are also additional approaches, which are outside of the main scope of this paper. For example, in scenarios when exposure information is partly missing, two-stage g-computation designs offer principled approaches to handle the missingness [
      • Breger T.L.
      • Edwards J.K.
      • Cole S.R.
      • Westreich D.
      • Pence B.W.
      • Adimora A.A.
      Two-stage g-computation: evaluating treatment and intervention impacts in observational cohorts when exposure information is partly missing.
      ]. Two-stage g-computation estimators leverage partially observed information on the full study sample and complete exposure information on a subset to estimate causal effects. Additional methods available to address exposure misclassification include simple bias analysis [
      • Lash T.L.V.T.
      • Haneuse S.
      • Rothman K.J.
      ,
      • Lash T.L.
      • Fox M.
      • MacLehose R.
      ,
      • Schneeweiss S.
      Sensitivity analysis and external adjustment for unmeasured confounders in epidemiologic database studies of therapeutics.
      ], probabilistic bias analysis [
      • Lash T.L.V.T.
      • Haneuse S.
      • Rothman K.J.
      ,
      • Lash T.L.
      • Fox M.
      • MacLehose R.
      ,
      • Lash T.L.
      • Fink A.K.
      Semi-automated sensitivity analysis to assess systematic errors in observational data.
      ,
      • Fox M.P.
      • Lash T.L.
      • Greenland S.
      A method to automate probabilistic sensitivity analyses of misclassified binary variables.
      ,
      • Greenland S.
      Basic methods for sensitivity analysis of biases.
      ], Bayesian bias analysis [
      • Lash T.L.V.T.
      • Haneuse S.
      • Rothman K.J.
      ,
      • MacLehose R.F.
      • Olshan A.F.
      • Herring A.H.
      • Honein M.A.
      • Shaw G.M.
      • Romitti P.A.
      Bayesian methods for correcting misclassification: an example from birth defects epidemiology.
      ,
      • Chu H.
      • Wang Z.
      • Cole S.R.
      • Greenland S.
      Sensitivity analysis of misclassification: a graphical and a Bayesian approach.
      ,
      • McCandless L.C.
      • Gustafson P.
      • Levy A.
      Bayesian sensitivity analysis for unmeasured confounding in observational studies.
      ], modified maximum likelihood [
      • Rosner B.
      • Willett W.C.
      • Spiegelman D.
      Correction of logistic regression relative risk estimates and confidence intervals for systematic within-person measurement error.
      ,
      • Neuhaus J.
      Bias and efficiency loss due to misclassified responses in binary regression.
      ,
      • Lyles R.H.
      • Tang L.
      • Superak H.M.
      • King C.C.
      • Celentano D.D.
      • Lo Y.
      • et al.
      Validation data-based adjustments for outcome misclassification in logistic regression: an illustration.
      ,
      • Magder L.S.
      • Hughes J.P.
      Logistic regression when the outcome is measured with uncertainty.
      ], multiple imputation for measurement error [
      • Bang H.
      • Chiu Y.L.
      • Kaufman J.S.
      • Patel M.D.
      • Heiss G.
      • Rose K.M.
      Bias Correction Methods for Misclassified Covariates in the Cox Model: comparison offive correction methods by simulation and data analysis.
      ,
      • Cole S.R.
      • Chu H.
      • Greenland S.
      Multiple-imputation for measurement-error correction.
      ], and regression calibration [
      • Lunt M.
      • Glynn R.J.
      • Rothman K.J.
      • Avorn J.
      • Stürmer T.
      Propensity score calibration in the absence of surrogacy.
      ,
      • Sturmer T.
      • Glynn R.J.
      • Rothman K.J.
      • Avorn J.
      • Schneeweiss S.
      Adjustments for unmeasured confounders in pharmacoepidemiologic database studies using external information.
      ,
      • Sturmer T.
      • Schneeweiss S.
      • Avorn J.
      • Glynn R.J.
      Adjusting effect estimates for unmeasured confounding with validation data using propensity score calibration.
      ]. Funk and Landi summarize these methods and provide examples [
      • Funk M.J.
      • Landi S.N.
      Misclassification in administrative claims data: quantifying the impact on treatment effect estimates.
      ].

      4. Outcome assessment: can outcomes be accurately identified?

      In noninterventional studies using real-world data, outcomes are commonly defined based on diagnoses, procedures, prescriptions, or healthcare encounters recorded by clinicians in routine care or by health insurance companies for billing purposes. Pandemic-induced changes to patient care-seeking behaviors, healthcare utilization, and (in some countries) insurance coverage may impact the accuracy, completeness, and timeliness of outcome assessment. Thus, outcome misclassification may occur more commonly in studies using real-world data generated in the pandemic. Consider a cohort study comparing risk of venous thromboembolism (VTE) between initiators of different oral contraceptives, where VTE is defined by the presence of diagnosis codes and radiology reports. Compared to the prepandemic era, VTE diagnosis among patients with mild/moderate symptoms may occur less frequently due to declines in in-person physical exams and diagnostic testing [
      • Ishigami J.
      • Kou M.
      • Ding N.
      • Matsushita K.
      Cardiovascular disease and coronavirus disease 2019: epidemiology, management, and prevention.
      ]. Typically, nondifferential outcome misclassification results in bias toward the null and underestimates of VTE risk, whereas differential outcome misclassification (that is, differential VTE underdiagnosis by formulation) may result in bias that either exaggerates or underestimates VTE risk [
      • Funk M.J.
      • Landi S.N.
      Misclassification in administrative claims data: quantifying the impact on treatment effect estimates.
      ].
      Selection bias due to differential loss-to-follow-up, also known as informative censoring [
      • Hernan M.A.
      • Hernandez-Diaz S.
      • Robins J.M.
      A structural approach to selection bias.
      ], represents another common threat to internal validity in cohort studies. The current standard in epidemiological research is to treat loss-to-follow-up, such as health plan disenrollment, as a censoring event that occurs independently of the outcome, and thus as a type of noninformative censoring. However, recent evidence of differential risk of disenrollment by patient- and health-plan level characteristics raises the possibility that treating health plan disenrollment as an independent censoring event may bias descriptive statistics as well as estimates of causal effect [
      • Howe C.J.
      • Cole S.R.
      • Lau B.
      • Napravnik S.
      • Eron Jr., J.J.
      Selection bias due to loss to follow up in cohort studies.
      ,
      • Butler A.M.
      • Todd J.V.
      • Sahrmann J.M.
      • Lesko C.R.
      • Brookhart M.A.
      Informative censoring by health plan disenrollment among commercially insured adults.
      ]. Consequently, selection bias may be more pronounced in studies using real-world data collected during the pandemic, since pandemic-related increases in unemployment and subsequent health plan disenrollment resulted in more loss-to-follow-up in specific subpopulations.
      In addition, due to changes in healthcare utilization patterns during the pandemic, the length of time between disease diagnosis and treatment initiation may increase due to delays in receiving medical care and treatments [
      • Jazieh A.R.
      • Akbulut H.
      • Curigliano G.
      • Rogado A.
      • Alsharm A.A.
      • Razis E.D.
      • et al.
      Impact of the COVID-19 pandemic on cancer care: a global collaborative study.
      ,
      • Carrieri D.
      • Peccatori F.A.
      Seeing beyond COVID-19: understanding the impact of the pandemic on oncology, and the importance of preparedness.
      ]. Consider a cohort study designed to quantify progression-free and overall survival among patients with resectable pancreatic cancer receiving neoadjuvant chemotherapy followed by surgery. But, during the pandemic, cancer patients may experience longer delays in initiation of treatment, impacting patient outcomes regardless of treatment type (due to delayed treatment initiation). Therefore, time from diagnosis to treatment initiation may differ by time relative to the pandemic (for example, prepandemic vs. pandemic era).

      4.1 Approaches to characterize or mitigate bias related to outcome assessment

      Little is known about the performance of algorithms using data from the COVID-19 era, thus, validation studies are critically needed. In addition, other standard approaches are available to address biases related to outcome assessment. Restricting or stratifying analyses by calendar time relative to COVID-19 or leveraging additional data may improve the validity and completeness of outcomes. For example, the accuracy of an algorithm to identify VTE may be enhanced by leveraging pharmacy claims or unstructured EHR notes. Furthermore, EHR data can be linked with additional data sources, such as the master death file, hospital data and/or the national death index. Rivera et al. and Pratt et al. provide guidance on the appropriateness, feasibility, evaluation, and reporting of data linkages [
      • Pratt N.L.
      • Mack C.D.
      • Meyer A.M.
      • Davis K.J.
      • Hammill B.G.
      • Hampp C.
      • et al.
      Data linkage in pharmacoepidemiology: a call for rigorous evaluation and reporting.
      ,
      • Rivera D.R.
      • Gokhale M.N.
      • Reynolds M.W.
      • Andrews E.B.
      • Chun D.
      • Haynes K.
      • et al.
      Linking electronic health data in pharmacoepidemiology: appropriateness and feasibility.
      ].
      To mitigate outcome misclassification, researchers ought to select outcomes robust to pandemic-related changes, such as outcomes that avoid reliance on outpatient and emergency department encounters that declined during the pandemic. For example, in the aforementioned UTI study, selecting a primary outcome that almost exclusively requires hospitalization, such as pyelonephritis, may be advantageous. Also, sensitivity analyses are needed to quantify the impact of outcome misclassification on estimates and the uncertainty around these estimates. Available methods are described above [
      • Lash T.L.V.T.
      • Haneuse S.
      • Rothman K.J.
      ,
      • Lash T.L.
      • Fox M.
      • MacLehose R.
      ,
      • Schneeweiss S.
      Sensitivity analysis and external adjustment for unmeasured confounders in epidemiologic database studies of therapeutics.
      ,
      • Lash T.L.
      • Fink A.K.
      Semi-automated sensitivity analysis to assess systematic errors in observational data.
      ,
      • Greenland S.
      Basic methods for sensitivity analysis of biases.
      ,
      • MacLehose R.F.
      • Olshan A.F.
      • Herring A.H.
      • Honein M.A.
      • Shaw G.M.
      • Romitti P.A.
      Bayesian methods for correcting misclassification: an example from birth defects epidemiology.
      ,
      • McCandless L.C.
      • Gustafson P.
      • Levy A.
      Bayesian sensitivity analysis for unmeasured confounding in observational studies.
      ,
      • Rosner B.
      • Willett W.C.
      • Spiegelman D.
      Correction of logistic regression relative risk estimates and confidence intervals for systematic within-person measurement error.
      ,
      • Magder L.S.
      • Hughes J.P.
      Logistic regression when the outcome is measured with uncertainty.
      ,
      • Bang H.
      • Chiu Y.L.
      • Kaufman J.S.
      • Patel M.D.
      • Heiss G.
      • Rose K.M.
      Bias Correction Methods for Misclassified Covariates in the Cox Model: comparison offive correction methods by simulation and data analysis.
      ,
      • Cole S.R.
      • Chu H.
      • Greenland S.
      Multiple-imputation for measurement-error correction.
      ,
      • Funk M.J.
      • Landi S.N.
      Misclassification in administrative claims data: quantifying the impact on treatment effect estimates.
      ].
      To mitigate selection bias, advanced statistical methods are available to handle outcome missingness or loss-to-follow-up. First, it is important to understand which individuals are at greatest risk of missing outcome data or loss-to-follow-up during COVID-19, and whether or not data are missing at random, missing completely at random, or missing not at random. Multiple imputation is an appropriate framework for dealing with data when missing at random or completely at random, and offers protection in some missing not at random contexts [
      • Harel O.
      • Mitchell E.M.
      • Perkins N.J.
      • Cole S.R.
      • Tchetgen Tchetgen E.J.
      • Sun B.
      • et al.
      Multiple imputation for Incomplete data in epidemiologic studies.
      ,
      • Schafer J.L.
      • Graham J.W.
      Missing data: our view of the state of the art.
      ,
      • Rubin D.B.
      Multiple imputation after 18+ years.
      ]. Other analytic approaches, such as Bayesian imputation or doubly robust estimators may be explored [
      • Erler N.S.
      • Rizopoulos D.
      • Jaddoe V.W.
      • Franco O.H.
      • Lesaffre E.M.
      Bayesian imputation of time-varying covariates in linear mixed models.
      ,
      • Erler N.S.
      • Rizopoulos D.
      • Rosmalen J.
      • Jaddoe V.W.
      • Franco O.H.
      • Lesaffre E.M.
      Dealing with missing covariates in epidemiologic studies: a comparison between multiple imputation and a full Bayesian approach.
      ,
      • Perkins N.J.
      • Cole S.R.
      • Harel O.
      • Tchetgen Tchetgen E.J.
      • Sun B.
      • Mitchell E.M.
      • et al.
      Principled approaches to missing data in epidemiologic studies.
      ], and sensitivity analysis may be conducted to evaluate the robustness of the different approaches. In scenarios with high potential for selection bias, appropriate methods to address selection bias should be applied including standard regression adjustment, joint modeling, and inverse probability of censoring-weighted estimation [
      • Hernan M.A.
      • Hernandez-Diaz S.
      • Robins J.M.
      A structural approach to selection bias.
      ,
      • Howe C.J.
      • Cole S.R.
      • Lau B.
      • Napravnik S.
      • Eron Jr., J.J.
      Selection bias due to loss to follow up in cohort studies.
      ,
      • Perkins N.J.
      • Cole S.R.
      • Harel O.
      • Tchetgen Tchetgen E.J.
      • Sun B.
      • Mitchell E.M.
      • et al.
      Principled approaches to missing data in epidemiologic studies.
      ,
      • Sun B.
      • Perkins N.J.
      • Cole S.R.
      • Harel O.
      • Mitchell E.M.
      • Schisterman E.F.
      • et al.
      Inverse-probability-weighted estimation for monotone and nonmonotone missing data.
      ,
      • Robins J.M.
      • Finkelstein D.M.
      Correcting for noncompliance and dependent censoring in an AIDS Clinical Trial with inverse probability of censoring weighted (IPCW) log-rank tests.
      ,
      • Hernan M.A.
      • McAdams M.
      • McGrath N.
      • Lanoy E.
      • Costagliola D.
      Observation plans in longitudinal studies with time-varying treatments.
      ,
      • Daniel R.M.
      • Kenward M.G.
      • Cousens S.N.
      • de Stavola B.L.
      Using causal diagrams to guide analysis in missing data problems.
      ,
      • Westreich D.
      Berkson's bias, selection bias, and missing data.
      ,
      • Cole S.R.
      • Hernan M.A.
      Constructing inverse probability weights for marginal structural models.
      ,
      • Howe C.J.
      • Cole S.R.
      • Chmiel J.S.
      • Munoz A.
      Limitation of inverse probability-of-censoring weights in estimating survival in the presence of strong selection bias.
      ,
      • Greenland S.
      • Pearl J.
      Adjustments and their consequences—collapsibility analysis using graphical models.
      ,
      • Hernan M.A.R.J.
      Causal Inference: What If.
      ].
      To understand differential delays between disease diagnosis and treatment between comparator groups if selected from different time periods, descriptive analyses may be conducted. For example, the time between diagnosis and treatment initiation can be assessed to detect irregularities during the pandemic; this parameter can be used as a descriptive statistic or a proxy measure to obtain confounding control. Sensitivity analyses are needed to address calendar-time (via restriction, stratification, modeling, matching, or weighting).

      5. Covariate assessment: identifying potential confounders and effect modifiers

      Confounding bias arises in the presence of nonexchangeability across exposure groups—due to the imbalance of (typically) causes of the outcome across levels of the exposure [
      • Westreich D.
      Epidemiology by Design: A Causal Approach to the Health Sciences.
      ]—and is an inherent limitation of noninterventional studies due to lack of randomization. Missing data on covariates that affect the exposure and independently affect the outcome may bias effect estimates due to residual confounding. The ability to ascertain information on potential confounders and account for confounding in studies using real-world data may be particularly challenging during COVID-19. Pandemic-related stay-home orders resulted in a decline of in-person healthcare encounters and an increase in telemedicine visits, resulting in absence of routinely collected clinical measurements (for example, body mass index, blood pressure) and laboratory results (for example, hemoglobin A1c). This is problematic in real-world data studies because the absence of a documented diagnosis or procedure is typically interpreted as the absence of the condition. And, the extent of missingness may be differential by specialty or disease/indication due to varying impacts of the pandemic on healthcare delivery. Thus, estimates of treatment effects may be subject to more residual confounding during COVID-19 due to missing information on a variety of potential confounders.
      A new consideration for studies using pandemic-era data is the possibility for COVID-19 infection history or COVID-19 vaccination status to be confounders of the exposure-outcome relationship under study. The ability for COVID-19 infection history or vaccination status to bias treatment effect estimates will be wide-ranging given the substantial COVID-19 burden of acute clinical manifestations and postacute sequelae [
      • Al-Aly Z.
      • Xie Y.
      • Bowe B.
      High-dimensional characterization of post-acute sequelae of COVID-19.
      ]. For example, in a study aiming to estimate the comparative risk of cardiovascular events for several antidiabetic medications, COVID-19 infection history may be a potential confounder if recipients of a particular antidiabetic medication have a differential history of COVID-19 infection and differential risk of the outcome. COVID-19 infection history and COVID-19 vaccination status may also be effect measure modifiers, resulting in treatment effect heterogeneity.

      5.1 Approaches to characterize or mitigate bias related to covariate assessment

      There are several rigorous epidemiologic study design decisions that can reduce or detect the potential for confounding by measured and unmeasured confounders. First, in studies of the comparative effects of medical interventions, use of an active comparator new-user study design is a powerful tool to balance the treatment groups with respect to patient characteristics and to reduce measured and unmeasured confounding by indication and frailty bias [
      • Lund J.L.
      • Richardson D.B.
      • Sturmer T.
      The active comparator, new user study design in pharmacoepidemiology: historical foundations and contemporary application.
      ,
      • D'Arcy M.
      • Stürmer T.
      • Lund J.L.
      The importance and implications of comparator selection in pharmacoepidemiologic research.
      ]. The active comparator aspect of the design requires restriction of the study population to patients who receive the medical intervention of interest or an alternative intervention commonly administered to patients with the same indication and without contraindications. Second, assessment of baseline confounders during an all-available covariate assessment period is generally preferable to the fixed-duration approach (for example, 6 months preindex) because it performs similarly or reduces overall confounder misclassification by increasing the sensitivity of confounder measurement [
      • Connolly J.G.
      • Schneeweiss S.
      • Glynn R.J.
      • Gagne J.J.
      Quantifying bias reduction with fixed-duration versus all-available covariate assessment periods.
      ,
      • Nakasian S.S.
      • Rassen J.A.
      • Franklin J.M.
      Effects of expanding the look-back period to all available data in the assessment of covariates.
      ,
      • Conover M.M.
      • Stürmer T.
      • Poole C.
      • Glynn R.J.
      • Simpson Jr., R.J.
      • Pate V.
      • et al.
      Classifying medical histories in US Medicare beneficiaries using fixed vs all-available look-back approaches.
      ]. Third, since missingness of potential confounders likely varies over calendar time, study designs that account for time (for example, calendar-time specific propensity scores, matching on index date) may mitigate differences between exposure groups [
      • Greifer N.
      • Stuart E.A.
      Matching methods for confounder adjustment: an addition to the epidemiologist's toolbox.
      ,
      • Mack C.D.
      • Glynn R.J.
      • Brookhart M.A.
      • Carpenter W.R.
      • Meyer A.M.
      • Sandler R.S.
      • et al.
      Calendar time-specific propensity scores and comparative effectiveness research for stage III colon cancer chemotherapy.
      ]. Fourth, negative control exposures and outcomes can detect, quantify, and correct for uncontrolled confounding [
      • Lipsitch M.
      • Tchetgen E.
      • Cohen T.
      Negative controls: a tool for detecting confounding and bias in observational studies.
      ,
      • Tchetgen Tchetgen E.
      The control outcome calibration approach for causal inference with unobserved confounding.
      ,
      • Li K.Q.
      • Shi X.
      • Miao W.
      • Tchetgen E.T.
      Double negative control inference in test-negative design studies of vaccine effectiveness.
      ]. Lastly, propensity score calibration or data linkages can reduce missingness of potential confounders.
      In addition, standard statistical approaches, such as multivariable regression models or propensity score methods, can be employed to account for confounding when potential confounders are adequately measured. Analytic approaches to handle missing data on potential confounders include complete-case analysis, last observation carried forward, the missingness pattern approach, multiple imputation, and inverse-probability-of-missingness weighting [
      • Leyrat C.
      • Carpenter J.R.
      • Bailly S.
      • Williamson E.J.
      Common methods for handling missing data in marginal structural models: what works and why.
      ]. Instrumental variable methods can address uncontrolled confounding when potential confounders are not adequately measured, as long as a suitable instrument exists [
      • Brookhart M.A.
      • Rassen J.A.
      • Schneeweiss S.
      Instrumental variable methods in comparative safety and effectiveness research.
      ,
      • Brookhart M.A.
      • Schneeweiss S.
      Preference-based instrumental variable methods for the estimation of treatment effects: assessing validity and interpreting results.
      ]. However, the use of calendar time as an instrument—a common choice in comparative effectiveness studies—will likely violate the instrumental variable assumptions during COVID-19, since calendar time may affect an outcome, such as mortality, in ways other than through the exposure [
      • Mack C.D.
      • Brookhart M.A.
      • Glynn R.J.
      • Meyer A.M.
      • Carpenter W.R.
      • Sandler R.S.
      • et al.
      Comparative effectiveness of oxaliplatin versus 5-flourouricil in older adults: an instrumental variable analysis.
      ].
      Sensitivity analyses are needed to quantify the impact of covariate misclassification on estimates and the uncertainty around these estimates. Available methods include regression calibration [,
      • Spiegelman D.
      • McDermott A.
      • Rosner B.
      Regression calibration method for correcting measurement-error bias in nutritional epidemiology.
      ] as well as those described above [
      • Lash T.L.V.T.
      • Haneuse S.
      • Rothman K.J.
      ,
      • Lash T.L.
      • Fox M.
      • MacLehose R.
      ,
      • Schneeweiss S.
      Sensitivity analysis and external adjustment for unmeasured confounders in epidemiologic database studies of therapeutics.
      ,
      • Lash T.L.
      • Fink A.K.
      Semi-automated sensitivity analysis to assess systematic errors in observational data.
      ,
      • Fox M.P.
      • Lash T.L.
      • Greenland S.
      A method to automate probabilistic sensitivity analyses of misclassified binary variables.
      ,
      • Greenland S.
      Basic methods for sensitivity analysis of biases.
      ,
      • MacLehose R.F.
      • Olshan A.F.
      • Herring A.H.
      • Honein M.A.
      • Shaw G.M.
      • Romitti P.A.
      Bayesian methods for correcting misclassification: an example from birth defects epidemiology.
      ,
      • McCandless L.C.
      • Gustafson P.
      • Levy A.
      Bayesian sensitivity analysis for unmeasured confounding in observational studies.
      ,
      • Rosner B.
      • Willett W.C.
      • Spiegelman D.
      Correction of logistic regression relative risk estimates and confidence intervals for systematic within-person measurement error.
      ,
      • Bang H.
      • Chiu Y.L.
      • Kaufman J.S.
      • Patel M.D.
      • Heiss G.
      • Rose K.M.
      Bias Correction Methods for Misclassified Covariates in the Cox Model: comparison offive correction methods by simulation and data analysis.
      ,
      • Cole S.R.
      • Chu H.
      • Greenland S.
      Multiple-imputation for measurement-error correction.
      ,
      • Lunt M.
      • Glynn R.J.
      • Rothman K.J.
      • Avorn J.
      • Stürmer T.
      Propensity score calibration in the absence of surrogacy.
      ,
      • Sturmer T.
      • Schneeweiss S.
      • Avorn J.
      • Glynn R.J.
      Adjusting effect estimates for unmeasured confounding with validation data using propensity score calibration.
      ,
      • Funk M.J.
      • Landi S.N.
      Misclassification in administrative claims data: quantifying the impact on treatment effect estimates.
      ].
      To address the possibility of confounding by COVID-19 infection history, investigators may treat documented COVID-19 infection as: a) an exclusion criterion if diagnosed on or before the index date; b) a censoring event if diagnosed during the follow-up period; or c) a confounder in the analysis. Treatment effect heterogeneity by COVID-19 infection history should also be evaluated, particularly in populations with high COVID-19 prevalence. However, these approaches may be limited by variability in testing capacity for SARS-CoV-2 infection and validity of COVID-19 diagnostic codes across settings and calendar time [
      • Bodilsen J.
      • Leth S.
      • Nielsen S.L.
      • Holler J.G.
      • Benfield T.
      • Omland L.H.
      Positive predictive value of ICD-10 diagnosis codes for COVID-19.
      ,
      • Wu G.
      • D'Souza A.G.
      • Quan H.
      • Southern D.A.
      • Youngson E.
      • Williamson T.
      • et al.
      Validity of ICD-10 codes for COVID-19 patients with hospital admissions or ED visits in Canada: a retrospective cohort study.
      ,
      • Lynch K.E.
      • Viernes B.
      • Gatsby E.
      • DuVall S.L.
      • Jones B.E.
      • Box T.L.
      • et al.
      Positive predictive value of COVID-19 ICD-10 diagnosis codes across calendar time and clinical setting.
      ].

      6. Accounting for competing risks

      In time-to-event analyses, individuals are observed from the start of the follow-up period until the occurrence of the event of interest, a competing event, or a censoring event. Competing events preclude the outcome of interest from occurring. For example, patients who die before experiencing a stroke will never be observed to have the event of interest.
      Given that COVID-19 caused substantial excess mortality, it is crucial to treat mortality as a competing event in studies with populations who experienced high COVID-19-related mortality, such as older adults or institutionalized individuals. Even though competing events are ubiquitous in epidemiological data, researchers frequently simplify analyses by treating competing events as censoring events, thus generating estimates of conditional risk defined as the risk that would be observed if all competing events were prevented without altering the hazard of the event of interest [
      • Lash T.L.V.T.
      • Haneuse S.
      • Rothman K.J.
      ,
      • Prentice R.L.
      • Kalbfleisch J.D.
      • Peterson Jr., A.V.
      • Flournoy N.
      • Farewell V.T.
      • Breslow N.E.
      The analysis of failure times in the presence of competing risks.
      ,
      • Murray C.J.
      • Salomon J.A.
      • Mathers C.D.
      • Lopez A.D.
      • et al.
      World Health Organization
      Summary measures of population health : concepts, ethics, measurement and applications / edited by Christopher J. L. Murray ... [‎et al.]‎.
      ]. However, the assumptions necessary for interpreting conditional risks limits the utility of these estimates for measuring public health impact. Since the degree of inflation of the estimate is proportional to the incidence of the competing event, censoring the competing events when competing events are common can dramatically distort estimates of risk [
      • Edwards J.K.
      • Hester L.L.
      • Gokhale M.
      • Lesko C.R.
      Methodologic issues when estimating risks in pharmacoepidemiology.
      ].

      6.1 Approaches to characterize or mitigate bias related to competing risks

      The competing-risk approach provides an estimate of the total amount of the event of interest that will occur in the population, which may provide estimates with greater accuracy and precision for health care policy and planning, compared to censoring the competing events. Researchers should assess their data sources with regards to the identifiability and frequency of competing risks in their study sample (for example, mortality), and consider using analytic methods to account for competing risks. The Aalen-Johansen estimator is straightforward to implement and can be used to generate interpretable, policy-relevant estimates of risk in the presence of competing events [
      • Aalen O.O.
      • Johansen S.
      An Empirical Transition Matrix for Non-Homogeneous Markov Chains Based on Censored Observations.
      ]. Edwards et al. provide detail on competing risks methodology and provide applied examples from noninterventional studies using real-world data [
      • Edwards J.K.
      • Hester L.L.
      • Gokhale M.
      • Lesko C.R.
      Methodologic issues when estimating risks in pharmacoepidemiology.
      ].

      7. Discussion

      Noninterventional studies are important for augmenting RCT evidence and generating evidence from real-world populations, but designing high-quality studies in the COVID-19 era requires careful consideration. We outline several challenges inherent to noninterventional research, and discuss how disruptions to healthcare utilization and outcomes during the COVID-19-pandemic pose challenges to the validity of noninterventional studies. Our proposed framework addresses several important methodological considerations in the design of noninterventional studies using pandemic-era real-world data. Researchers may find the guidance particularly useful for studies of populations that have been disproportionately impacted by COVID-19, such as racial/ethnic minorities, rural residents, or people experiencing poverty. Detailed attention to study design and analytic decisions have broad implications for the quality of future studies using real-world data from the pandemic-era. Our recommendations will foster improvements in the design and conduct of future noninterventional studies using real-world data and enhance the ability of future studies to provide rigorous evidence that is critical to patients, caregivers, clinicians, payers, policymakers, and other stakeholders.

      References

        • World Health Organization
        (Available at:)
        • Agarwal S.D.
        • Sommers B.D.
        Insurance coverage after job loss - the importance of the ACA during the covid-associated recession.
        N Engl J Med. 2020; 383: 1603-1606
        • Blumenthal D.
        • Fowler E.J.
        • Abrams M.
        • Collins S.R.
        Covid-19 - implications for the health care system.
        N Engl J Med. 2020; 383: 1483-1488
        • Whaley C.M.
        • Pera M.F.
        • Cantor J.
        • Chang J.
        • Velasco J.
        • Hagg H.K.
        • et al.
        Changes in health services use among commercially insured US populations during the COVID-19 pandemic.
        JAMA Netw Open. 2020; 3e2024984
        • Jazieh A.R.
        • Akbulut H.
        • Curigliano G.
        • Rogado A.
        • Alsharm A.A.
        • Razis E.D.
        • et al.
        Impact of the COVID-19 pandemic on cancer care: a global collaborative study.
        JCO Glob Oncol. 2020; 6: 1428-1438
        • Meredith J.W.
        • High K.P.
        • Freischlag J.A.
        Preserving elective surgeries in the COVID-19 pandemic and the future.
        Jama. 2020; 324: 1725-1726
        • Kaufman H.W.
        • Chen Z.
        • Niles J.K.
        • Fesko Y.A.
        Changes in newly identified cancer among US patients from before COVID-19 through the first full year of the pandemic.
        JAMA Netw Open. 2021; 4e2125681
        • Alexander G.C.
        • Tajanlangit M.
        • Heyward J.
        • Mansour O.
        • Qato D.M.
        • Stafford R.S.
        Use and content of primary care office-based vs telemedicine care visits during the COVID-19 pandemic in the US.
        JAMA Netw Open. 2020; 3e2021476
        • McWilliams J.M.
        • Russo A.
        • Mehrotra A.
        Implications of early health care spending reductions for expected spending as the COVID-19 pandemic evolves.
        JAMA Intern Med. 2021; 181: 118-120
        • Patel S.Y.
        • Mehrotra A.
        • Huskamp H.A.
        • Uscher-Pines L.
        • Ganguli I.
        • Barnett M.L.
        Trends in outpatient care delivery and telemedicine during the COVID-19 pandemic in the US.
        JAMA Intern Med. 2021; 181: 388-391
        • Vaduganathan M.
        • van Meijgaard J.
        • Mehra M.R.
        • Joseph J.
        • O'Donnell C.J.
        • Warraich H.J.
        Prescription fill patterns for commonly used drugs during the COVID-19 pandemic in the United States.
        JAMA. 2020; 323: 2524-2526
        • Selke Krulichová I.
        • Selke G.W.
        • Bennie M.
        • Hajiebrahimi M.
        • Nyberg F.
        • Fürst J.
        • et al.
        Comparison of drug prescribing before and during the COVID-19 pandemic: a cross-national European study.
        Pharmacoepidemiol Drug Saf. 2022; 31: 1046-1055
        • Baum A.
        • Kaboli P.J.
        • Schwartz M.D.
        Reduced in-person and increased telehealth outpatient visits during the COVID-19 pandemic.
        Ann Intern Med. 2021; 174: 129-131
        • Sokolski M.
        • Gajewski P.
        • Zymliński R.
        • Biegus J.
        • Berg J.M.T.
        • Bor W.
        • et al.
        Impact of coronavirus disease 2019 (COVID-19) outbreak on acute admissions at the emergency and cardiology departments across europe.
        Am J Med. 2021; 134: 482-489
        • Nijman R.G.
        • Honeyford K.
        • Farrugia R.
        • Rose K.
        • Bognar Z.
        • Buonsenso D.
        • et al.
        Presentations of children to emergency departments across Europe and the COVID-19 pandemic: a multinational observational study.
        PLoS Med. 2022; 19e1003974
        • Jacobs E.A.
        • Ogedegbe O.
        • Fihn S.D.
        Elective care and health services research in the COVID-19 era.
        JAMA Netw Open. 2020; 3e2025731
        • Butler S.M.
        Four COVID-19 lessons for achieving health equity.
        JAMA. 2020; 324: 2245-2246
        • Metzl J.M.
        • Maybank A.
        • De Maio F.
        Responding to the COVID-19 pandemic: the need for a structurally competent health care system.
        JAMA. 2020; 324: 231-232
        • Coughlin S.S.
        • Yiğiter A.
        • Xu H.
        • Berman A.E.
        • Chen J.
        Early detection of change patterns in COVID-19 incidence and the implementation of public health policies: a multi-national study.
        Public Health Pract (Oxf). 2021; 2100064
        • Pottegård A.
        • Kurz X.
        • Moore N.
        • Christiansen C.F.
        • Klungel O.
        Considerations for pharmacoepidemiological analyses in the SARS-CoV-2 pandemic.
        Pharmacoepidemiol Drug Saf. 2020; 29: 825-831
        • Franklin J.M.
        • Lin K.J.
        • Gatto N.M.
        • Rassen J.A.
        • Glynn R.J.
        • Schneeweiss S.
        Real-world evidence for assessing pharmaceutical treatments in the context of COVID-19.
        Clin Pharmacol Ther. 2021; 109: 816-828
        • Reynolds M.W.
        • Petruski-Ivleva N.
        • Samant M.
        • Trivedi V.
        • Dreyer N.A.
        • Graff J.
        • et al.
        The Impact of COVID-19 on Real-World Health Data and Research: Understanding the Implications of the Pandemic on Real-World Health Research: Key Considerations and Solutions [White paper].
        2021 (Available at)
        • Webster-Clark M.
        Ways COVID-19 may impact unrelated pharmacoepidemiologic research using routinely collected data.
        Pharmacoepidemiol Drug Saf. 2021; 30: 400-401
        • Cole S.R.
        • Stuart E.A.
        Generalizing evidence from randomized clinical trials to target populations: the ACTG 320 trial.
        Am J Epidemiol. 2010; 172: 107-115
        • Lesko C.R.
        • Buchanan A.L.
        • Westreich D.
        • Edwards J.K.
        • Hudgens M.G.
        • Cole S.R.
        Generalizing study results: a potential outcomes perspective.
        Epidemiology. 2017; 28: 553-561
        • Stuart E.A.
        • Cole S.R.
        • Bradshaw C.P.
        • Leaf P.J.
        The use of propensity scores to assess the generalizability of results from randomized trials.
        J R Stat Soc Ser A Stat Soc. 2001; 174: 369-386
        • Pearl J.
        Generalizing experimental findings.
        J Causal Inference. 2015; 3: 259-266
        • Rothman K.J.
        • Gallacher J.E.
        • Hatch E.E.
        Why representativeness should be avoided.
        Int J Epidemiol. 2013; 42: 1012-1014
        • Cutler D.
        How will COVID-19 affect the health care economy?.
        Jama. 2020; 323: 2237-2238
        • Corallo BM S.
        2022;Pages. Accessed at kaiser family foundation.
        (Available at:)
        • Gmunder K.N.
        • Ruiz J.W.
        • Franceschi D.
        • Suarez M.M.
        Demographics associated with US healthcare disparities are exacerbated by the telemedicine surge during the COVID-19 pandemic.
        J Telemed Telecare. 2021; 1357633X211025939https://doi.org/10.1177/1357633X211025939
        • Robins J.
        A new approach to causal inference in mortality studies with sustained exposure periods - application to control of the healthy worker survivor effect.
        Math Model. 1986; 7: 1393-1512
        • Pearl J.
        Causality.
        Cambridge University Press, New York, NY2009
        • Maldonado G.
        • Greenland S.
        Estimating causal effects.
        Int J Epidemiol. 2002; 31: 422-429
        • Kaviani P.
        • Landi S.N.
        • McKethan A.
        • Brookhart M.A.
        • McGrath L.J.
        Who are we missing? Underrepresentation of data sources used for pharmacoepidemiology research in the United States.
        Pharmacoepidemiol Drug Saf. 2020; 29: 1494-1498
        • Gatto N.M.
        • Campbell U.B.
        • Rubinstein E.
        • Jaksa A.
        • Mattox P.
        • Mo J.
        • et al.
        The structured process to identify fit-for-purpose data: a data feasibility assessment framework.
        Clin Pharmacol Ther. 2022; 111: 122-134
        • Stürmer T.
        • Wang T.
        • Golightly Y.M.
        • Keil A.
        • Lund J.L.
        • Jonsson Funk M.
        Methodological considerations when analysing and interpreting real-world data.
        Rheumatology (Oxford). 2020; 59: 14-25
      1. Substance abuse and mental health services administration.
        (Available at:)
        • Clement J.
        • Jacobi M.
        • Greenwood B.N.
        Patient access to chronic medications during the Covid-19 pandemic: evidence from a comprehensive dataset of US insurance claims.
        PLoS One. 2021; 16e0249453
        • Rasmussen L.
        • Wettermark B.
        • Steinke D.
        • Pottegård A.
        Core concepts in pharmacoepidemiology: measures of drug utilization based on individual-level drug dispensing data.
        Pharmacoepidemiol Drug Saf. 2022; 31: 1015-1026
        • Hernán M.A.
        • Hernández-Díaz S.
        Beyond the intention-to-treat in comparative effectiveness research.
        Clin Trials. 2012; 9: 48-55
        • Breger T.L.
        • Edwards J.K.
        • Cole S.R.
        • Westreich D.
        • Pence B.W.
        • Adimora A.A.
        Two-stage g-computation: evaluating treatment and intervention impacts in observational cohorts when exposure information is partly missing.
        Epidemiology. 2020; 31: 695-703
        • Lash T.L.V.T.
        • Haneuse S.
        • Rothman K.J.
        Modern Epidemiology. Fourth ed. Wolters Kluwer, Philadelphia, PA2021
        • Lash T.L.
        • Fox M.
        • MacLehose R.
        Applying Quantitative Bias Analysis to Epidemiologic Data. Second ed. Springer International Publishing, Switzerland2021
        • Schneeweiss S.
        Sensitivity analysis and external adjustment for unmeasured confounders in epidemiologic database studies of therapeutics.
        Pharmacoepidemiol Drug Saf. 2006; 15: 291-303
        • Lash T.L.
        • Fink A.K.
        Semi-automated sensitivity analysis to assess systematic errors in observational data.
        Epidemiology. 2003; 14: 451-458
        • Fox M.P.
        • Lash T.L.
        • Greenland S.
        A method to automate probabilistic sensitivity analyses of misclassified binary variables.
        Int J Epidemiol. 2005; 34: 1370-1376
        • Greenland S.
        Basic methods for sensitivity analysis of biases.
        Int J Epidemiol. 1996; 25: 1107-1116
        • MacLehose R.F.
        • Olshan A.F.
        • Herring A.H.
        • Honein M.A.
        • Shaw G.M.
        • Romitti P.A.
        Bayesian methods for correcting misclassification: an example from birth defects epidemiology.
        Epidemiology. 2009; 20: 27-35
        • Chu H.
        • Wang Z.
        • Cole S.R.
        • Greenland S.
        Sensitivity analysis of misclassification: a graphical and a Bayesian approach.
        Ann Epidemiol. 2006; 16: 834-841
        • McCandless L.C.
        • Gustafson P.
        • Levy A.
        Bayesian sensitivity analysis for unmeasured confounding in observational studies.
        Stat Med. 2007; 26: 2331-2347
        • Rosner B.
        • Willett W.C.
        • Spiegelman D.
        Correction of logistic regression relative risk estimates and confidence intervals for systematic within-person measurement error.
        Stat Med. 1989; 8: 1051-1069
        • Neuhaus J.
        Bias and efficiency loss due to misclassified responses in binary regression.
        Biometrika. 1999; 86: 43-55
        • Lyles R.H.
        • Tang L.
        • Superak H.M.
        • King C.C.
        • Celentano D.D.
        • Lo Y.
        • et al.
        Validation data-based adjustments for outcome misclassification in logistic regression: an illustration.
        Epidemiology. 2011; 22: 589-597
        • Magder L.S.
        • Hughes J.P.
        Logistic regression when the outcome is measured with uncertainty.
        Am J Epidemiol. 1997; 146: 195-203
        • Bang H.
        • Chiu Y.L.
        • Kaufman J.S.
        • Patel M.D.
        • Heiss G.
        • Rose K.M.
        Bias Correction Methods for Misclassified Covariates in the Cox Model: comparison offive correction methods by simulation and data analysis.
        J Stat Theor Pract. 2013; 7: 381-400
        • Cole S.R.
        • Chu H.
        • Greenland S.
        Multiple-imputation for measurement-error correction.
        Int J Epidemiol. 2006; 35: 1074-1081
        • Lunt M.
        • Glynn R.J.
        • Rothman K.J.
        • Avorn J.
        • Stürmer T.
        Propensity score calibration in the absence of surrogacy.
        Am J Epidemiol. 2012; 175: 1294-1302
        • Sturmer T.
        • Glynn R.J.
        • Rothman K.J.
        • Avorn J.
        • Schneeweiss S.
        Adjustments for unmeasured confounders in pharmacoepidemiologic database studies using external information.
        Med Care. 2007; 45: S158-S165
        • Sturmer T.
        • Schneeweiss S.
        • Avorn J.
        • Glynn R.J.
        Adjusting effect estimates for unmeasured confounding with validation data using propensity score calibration.
        Am J Epidemiol. 2005; 162: 279-289
        • Funk M.J.
        • Landi S.N.
        Misclassification in administrative claims data: quantifying the impact on treatment effect estimates.
        Curr Epidemiol Rep. 2014; 1: 175-185
        • Ishigami J.
        • Kou M.
        • Ding N.
        • Matsushita K.
        Cardiovascular disease and coronavirus disease 2019: epidemiology, management, and prevention.
        Curr Epidemiol Rep. 2021; 8: 1-8
        • Hernan M.A.
        • Hernandez-Diaz S.
        • Robins J.M.
        A structural approach to selection bias.
        Epidemiology. 2004; 15: 615-625
        • Howe C.J.
        • Cole S.R.
        • Lau B.
        • Napravnik S.
        • Eron Jr., J.J.
        Selection bias due to loss to follow up in cohort studies.
        Epidemiology. 2016; 27: 91-97
        • Butler A.M.
        • Todd J.V.
        • Sahrmann J.M.
        • Lesko C.R.
        • Brookhart M.A.
        Informative censoring by health plan disenrollment among commercially insured adults.
        Pharmacoepidemiol Drug Saf. 2019; 28: 640-648
        • Carrieri D.
        • Peccatori F.A.
        Seeing beyond COVID-19: understanding the impact of the pandemic on oncology, and the importance of preparedness.
        Hist Philos Life Sci. 2020; 42: 57
        • Pratt N.L.
        • Mack C.D.
        • Meyer A.M.
        • Davis K.J.
        • Hammill B.G.
        • Hampp C.
        • et al.
        Data linkage in pharmacoepidemiology: a call for rigorous evaluation and reporting.
        Pharmacoepidemiol Drug Saf. 2020; 29: 9-17
        • Rivera D.R.
        • Gokhale M.N.
        • Reynolds M.W.
        • Andrews E.B.
        • Chun D.
        • Haynes K.
        • et al.
        Linking electronic health data in pharmacoepidemiology: appropriateness and feasibility.
        Pharmacoepidemiol Drug Saf. 2020; 29: 18-29
        • Harel O.
        • Mitchell E.M.
        • Perkins N.J.
        • Cole S.R.
        • Tchetgen Tchetgen E.J.
        • Sun B.
        • et al.
        Multiple imputation for Incomplete data in epidemiologic studies.
        Am J Epidemiol. 2018; 187: 576-584
        • Schafer J.L.
        • Graham J.W.
        Missing data: our view of the state of the art.
        Psychol Methods. 2002; 7: 147-177
        • Rubin D.B.
        Multiple imputation after 18+ years.
        J Am Stat Assoc. 1996; 91: 473-489
        • Erler N.S.
        • Rizopoulos D.
        • Jaddoe V.W.
        • Franco O.H.
        • Lesaffre E.M.
        Bayesian imputation of time-varying covariates in linear mixed models.
        Stat Methods Med Res. 2019; 28: 555-568
        • Erler N.S.
        • Rizopoulos D.
        • Rosmalen J.
        • Jaddoe V.W.
        • Franco O.H.
        • Lesaffre E.M.
        Dealing with missing covariates in epidemiologic studies: a comparison between multiple imputation and a full Bayesian approach.
        Stat Med. 2016; 35: 2955-2974
        • Perkins N.J.
        • Cole S.R.
        • Harel O.
        • Tchetgen Tchetgen E.J.
        • Sun B.
        • Mitchell E.M.
        • et al.
        Principled approaches to missing data in epidemiologic studies.
        Am J Epidemiol. 2018; 187: 568-575
        • Sun B.
        • Perkins N.J.
        • Cole S.R.
        • Harel O.
        • Mitchell E.M.
        • Schisterman E.F.
        • et al.
        Inverse-probability-weighted estimation for monotone and nonmonotone missing data.
        Am J Epidemiol. 2018; 187: 585-591
        • Robins J.M.
        • Finkelstein D.M.
        Correcting for noncompliance and dependent censoring in an AIDS Clinical Trial with inverse probability of censoring weighted (IPCW) log-rank tests.
        Biometrics. 2000; 56: 779-788
        • Hernan M.A.
        • McAdams M.
        • McGrath N.
        • Lanoy E.
        • Costagliola D.
        Observation plans in longitudinal studies with time-varying treatments.
        Stat Methods Med Res. 2009; 18: 27-52
        • Daniel R.M.
        • Kenward M.G.
        • Cousens S.N.
        • de Stavola B.L.
        Using causal diagrams to guide analysis in missing data problems.
        Stat Methods Med Res. 2012; 21: 243-256
        • Westreich D.
        Berkson's bias, selection bias, and missing data.
        Epidemiology. 2012; 23: 159-164
        • Cole S.R.
        • Hernan M.A.
        Constructing inverse probability weights for marginal structural models.
        Am J Epidemiol. 2008; 168: 656-664
        • Howe C.J.
        • Cole S.R.
        • Chmiel J.S.
        • Munoz A.
        Limitation of inverse probability-of-censoring weights in estimating survival in the presence of strong selection bias.
        Am J Epidemiol. 2011; 173: 569-577
        • Greenland S.
        • Pearl J.
        Adjustments and their consequences—collapsibility analysis using graphical models.
        Int Stat Rev/Revue Internationale de Statistique. 2011; 79: 401-426
        • Hernan M.A.R.J.
        Causal Inference: What If.
        Chapman & Hall/CRC, Boca Raton2020
        • Westreich D.
        Epidemiology by Design: A Causal Approach to the Health Sciences.
        Oxford University Press, New York, NY2019
        • Al-Aly Z.
        • Xie Y.
        • Bowe B.
        High-dimensional characterization of post-acute sequelae of COVID-19.
        Nature. 2021; 594: 259-264
        • Lund J.L.
        • Richardson D.B.
        • Sturmer T.
        The active comparator, new user study design in pharmacoepidemiology: historical foundations and contemporary application.
        Curr Epidemiol Rep. 2015; 2: 221-228
        • D'Arcy M.
        • Stürmer T.
        • Lund J.L.
        The importance and implications of comparator selection in pharmacoepidemiologic research.
        Curr Epidemiol Rep. 2018; 5: 272-283
        • Connolly J.G.
        • Schneeweiss S.
        • Glynn R.J.
        • Gagne J.J.
        Quantifying bias reduction with fixed-duration versus all-available covariate assessment periods.
        Pharmacoepidemiol Drug Saf. 2019; 28: 665-670
        • Nakasian S.S.
        • Rassen J.A.
        • Franklin J.M.
        Effects of expanding the look-back period to all available data in the assessment of covariates.
        Pharmacoepidemiol Drug Saf. 2017; 26: 890-899
        • Conover M.M.
        • Stürmer T.
        • Poole C.
        • Glynn R.J.
        • Simpson Jr., R.J.
        • Pate V.
        • et al.
        Classifying medical histories in US Medicare beneficiaries using fixed vs all-available look-back approaches.
        Pharmacoepidemiol Drug Saf. 2018; 27: 771-780
        • Greifer N.
        • Stuart E.A.
        Matching methods for confounder adjustment: an addition to the epidemiologist's toolbox.
        Epidemiol Rev. 2022; 43: 118-129
        • Mack C.D.
        • Glynn R.J.
        • Brookhart M.A.
        • Carpenter W.R.
        • Meyer A.M.
        • Sandler R.S.
        • et al.
        Calendar time-specific propensity scores and comparative effectiveness research for stage III colon cancer chemotherapy.
        Pharmacoepidemiol Drug Saf. 2013; 22: 810-818
        • Lipsitch M.
        • Tchetgen E.
        • Cohen T.
        Negative controls: a tool for detecting confounding and bias in observational studies.
        Epidemiology (Cambridge, Mass.). 2010; 21: 383-388
        • Tchetgen Tchetgen E.
        The control outcome calibration approach for causal inference with unobserved confounding.
        Am J Epidemiol. 2014; 179: 633-640
        • Li K.Q.
        • Shi X.
        • Miao W.
        • Tchetgen E.T.
        Double negative control inference in test-negative design studies of vaccine effectiveness.
        ArXiv. 2022; arXiv:2203.12509v2
        • Leyrat C.
        • Carpenter J.R.
        • Bailly S.
        • Williamson E.J.
        Common methods for handling missing data in marginal structural models: what works and why.
        Am J Epidemiol. 2021; 190: 663-672
        • Brookhart M.A.
        • Rassen J.A.
        • Schneeweiss S.
        Instrumental variable methods in comparative safety and effectiveness research.
        Pharmacoepidemiol Drug Saf. 2010; 19: 537-554
        • Brookhart M.A.
        • Schneeweiss S.
        Preference-based instrumental variable methods for the estimation of treatment effects: assessing validity and interpreting results.
        Int J Biostat. 2007; 3
        • Mack C.D.
        • Brookhart M.A.
        • Glynn R.J.
        • Meyer A.M.
        • Carpenter W.R.
        • Sandler R.S.
        • et al.
        Comparative effectiveness of oxaliplatin versus 5-flourouricil in older adults: an instrumental variable analysis.
        Epidemiology. 2015; 26: 690-699
        • Logan R.
        Spiegelman D 2012.
        (Available at)
        • Spiegelman D.
        • McDermott A.
        • Rosner B.
        Regression calibration method for correcting measurement-error bias in nutritional epidemiology.
        Am J Clin Nutr. 1997; 65: 1179S-1186S
        • Bodilsen J.
        • Leth S.
        • Nielsen S.L.
        • Holler J.G.
        • Benfield T.
        • Omland L.H.
        Positive predictive value of ICD-10 diagnosis codes for COVID-19.
        Clin Epidemiol. 2021; 13: 367-372
        • Wu G.
        • D'Souza A.G.
        • Quan H.
        • Southern D.A.
        • Youngson E.
        • Williamson T.
        • et al.
        Validity of ICD-10 codes for COVID-19 patients with hospital admissions or ED visits in Canada: a retrospective cohort study.
        BMJ Open. 2022; 12e057838
        • Lynch K.E.
        • Viernes B.
        • Gatsby E.
        • DuVall S.L.
        • Jones B.E.
        • Box T.L.
        • et al.
        Positive predictive value of COVID-19 ICD-10 diagnosis codes across calendar time and clinical setting.
        Clin Epidemiol. 2021; 13: 1011-1018
        • Prentice R.L.
        • Kalbfleisch J.D.
        • Peterson Jr., A.V.
        • Flournoy N.
        • Farewell V.T.
        • Breslow N.E.
        The analysis of failure times in the presence of competing risks.
        Biometrics. 1978; 34: 541-554
        • Murray C.J.
        • Salomon J.A.
        • Mathers C.D.
        • Lopez A.D.
        • et al.
        • World Health Organization
        Summary measures of population health : concepts, ethics, measurement and applications / edited by Christopher J. L. Murray ... [‎et al.]‎.
        World Health Organization, Geneva2002: 291-302 (Available at)
        https://apps.who.int/iris/handle/10665/42439
        Date accessed: December 6, 2022
        • Edwards J.K.
        • Hester L.L.
        • Gokhale M.
        • Lesko C.R.
        Methodologic issues when estimating risks in pharmacoepidemiology.
        Curr Epidemiol Rep. 2016; 3: 285-296
        • Aalen O.O.
        • Johansen S.
        An Empirical Transition Matrix for Non-Homogeneous Markov Chains Based on Censored Observations.
        1978: 141-150