(162) Selective Sampling: A Novel Approach to Simulating Biased Samples for Methods Research

Publication Number: 1207

Background: Simulation is a powerful tool for evaluating and comparing epidemiologic methods but is often criticized for being overly simplistic. Additional approaches are needed to evaluate epidemiologic methods in controlled but realistic settings.

Objectives: Develop a method to a) create cohorts with known treatment effects but without simulating the exposure, outcome, or covariates, and b) evaluate its use for comparing epidemiologic analyses.

Methods: Using data from 5 randomized clinical trials in 3 therapeutic areas (i.e., HIV, metastatic colorectal cancer, T2DM), we simulated cohorts with clinically meaningful covariate imbalance patterns. For each trial, we identified a subset of covariates associated with the outcome and expected to influence treatment choice in real-world settings. A patient's probability of being selected into the simulated cohort was based on these covariates, investigator-specified parameters, and treatment-group specific thresholds. Parameter values for logistic regression models were defined to induce stronger or weaker selection. As a result of selecting only some patients from each treatment arm for inclusion in a cohort, the sample size was reduced by half leading to unstable effect estimates when the source trial was small (n < 800). We re-sampled the trial with replacement, such that the simulated cohort’s sample size approximated the original trial. G-computation was used to recover the true treatment effect in the simulated cohorts accounting for potential changes in the distribution of effect measure modifiers between the trial and the cohorts. Multiple confounding scenarios were created using different selection parameters, and 1000 iterations were simulated for each.

Results: A total of 106 unique scenarios were created: 51 from positive trials (mean true HR=0.77 [range: 0.51-0.87]) and 55 from null trials (HR=0.97 [0.93-1.10]). On average, the crude treatment effect in the simulated cohorts was HR=0.73 (0.55-0.99) in positive trials and HR=0.87 (0.63-1.18) in null trials. The average absolute bias of the lnHR was 0.22 (0.02-0.49) and 0.23 (0.03-0.40) for positive and null trials, respectively. Induction of confounding was difficult in settings where few measured covariates were strongly associated with the outcome. G-computation estimates of the true HR were consistent with those from the full RCT.

Conclusions: This approach was successful at creating simulated cohorts with known confounding structures. Using these resampled data in which the true effect was known, we were able to evaluate the performance of various sensitivity analyses to address confounding. Unlike in plasmode simulations where one or more individual elements are simulated, selective sampling alone was used to induce bias.