Adjusting for Calendar Time in Analyses of COVID-19 Vaccine Safety and Effectiveness: A Simulation Study

Background: Observational data analyses regarding COVID-19 vaccines often compare event occurrence relative to vaccinated and unvaccinated person time. Accounting for the time-varying incidence of SARS-CoV-2 and the rapid uptake of COVID-19 vaccines may depend on choice of time scale in Cox regression analyses.

Objectives: To quantify bias in person-time based methods with and without adjustment for calendar time, using simulations.

Methods: We simulated 500,000 individuals who were followed for 365 days, and a point exposure (vaccination) with a cumulative incidence proportion of 80%. Exposure dates followed a normal distribution (mean 180, standard deviation [SD] 30) and exposure status was considered a time-varying covariate. Individuals were censored on event occurrence.
We generated an effectiveness outcome, emulating the daily risk of SARS-CoV-2 infection in Denmark during 2021 (cumulative incidence 10%), and a safety outcome with seasonal variation (myocarditis, risk 1/5000). For the effectiveness outcome, we tested true incidence rate ratios (IRR) of 0.1 and 0.5, and for the safety outcome we tested an IRR of 5.0. Multiple estimators were evaluated: 1) Poisson regression, 2) Cox regression with time-since-vaccination as the scale, and 3) Cox regression with a calendar time scale. For methods 1 and 2, we also obtained calendar time-adjusted estimates (restricted cubic splines).

Results: For the effectiveness outcome, mean hazard ratios (HR) with a time-since-vaccination scale were biased (true IRR 0.1, HR 0.38; true IRR 0.5, HR 1.94). The same was observed for Poisson regression (mean IRR 0.26 and 1.26). Adjustment for calendar time reduced bias in Cox (HR 0.17 and 0.74) and Poisson regression (IRR 0.10 and 0.48). Cox regression with calendar time as the underlying time scale yielded unbiased estimates (HR 0.10 and 0.50). For the safety outcome, adjusted estimators yielded results close to the simulated IRR (HRTime-in-study 5.28, HRCalendar time 5.14, IRR 4.69). All estimators were unbiased when analyzing a constant rate.

Conclusions: Adjusting for the time-varying hazard of outcomes related to the COVID-19 pandemic is crucial, as lack of adjustment may severely bias estimates. Adjusted Poisson- or Cox regression with a calendar scale are recommended.