(34) Test negative designs in the context of COVID-19 vaccine effectiveness: a systematic methodology review

Background: The test-negative design (TND) is an observational study design routinely used to investigate post-licensure seasonal flu vaccine effectiveness (VE). At the height of the global severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic, it was widely used to evaluate COVID-19 vaccine effectiveness. As proposed, the TND involves the recruitment of care-seeking individuals who meet a common clinical case definition. All participants are then tested for an infection. Under certain assumptions, logistic regression can then be used to estimate risk ratios for vaccine effectiveness against symptomatic or medically attended illness. Our hypothesis was that the emergency context of the pandemic may have affected how TND studies are conducted.

Objectives: To describe the methodology used in the design and analysis of the TND applied to the evaluation of COVID-19 vaccine effectiveness and the reporting for potential biases.

Methods: A systematic review was conducted. We searched five biomedical databases using defined search terms. Peer-reviewed and preprint articles published between January 1, 2020, and January 25, 2022, were included. Studies described as a TND or a test-negative case-control study in which cases and controls were identified based on test and laboratory results of SARS-CoV-2 infection were included. Only studies based on individual-level vaccination data were included. Simulation studies were excluded. Information regarding the setting and context of recruitment, source population, data source, clinical definition of cases and controls, exclusion criteria based on past or future positive or negative tests, vaccine types, analysis type and discussion of assumptions or concerns related to the interpretation of causality were extracted.

Results: Our search identified 99 studies, 37 of which met the defined criteria. Most studies were from North America (n=17/37) and targeted the adult populations (n=17/37). In contrast to pre-pandemic studies, the majority of included studies evaluating the VE of COVID-19 used retrospective administrative or hospital databases. Clinical case definitions were based primarily on COVID-19-like symptoms; however, several papers did not consider or specify symptoms. Whereas the conventional TND requires cases and controls to have the same inclusion criteria, this was only the case for half of the analyses (38/74). In addition, the conditions for selecting controls were not clearly described in some analyses (5/74). The selection of cases often required no previous positive test in a pre-specified period, no previous positive test at any point, or no restriction; approaches to the selection of controls were more diverse. Polymerase chain reaction testing was used in most papers (n=35/37). Vaccination status was generally classified by time since last dose, type of vaccine, number of vaccine doses and/or booster. Potential unmeasured confounding (n=21/37), misclassification of current SARS-CoV-2 infection (n=16/37) and selection bias (n=10/37) were mentioned by some studies.

Conclusion: In this systematic review, we observed deviations from the validated design in the application of the TND in the context of the pandemic and a lack of recognition of potential biases. We intend to investigate the potential impact of those deviations on the bias of VE estimates in a future study.