(158) The Autopsy of a Failed Instrumental Variable: Plasmode Simulation as a Tool to Assess Unconfoundedness of an IV

Publication Number: 489

Background: In UTMOST, a cohort study comparing partial vs total knee replacement , we were able to replicate the results of an RCT using propensity score stratification and inverse probability weighting. However, instrumental variables (IV) based on surgeon’s preference failed, yielding implausible estimates. This is despite these IVs fulfilling two pre-specified conditions: they were strong instruments and they balanced known confounders (all with an ASMD < 0.1).

Objectives: To estimate bias and coverage of preference-based IV analyses and of a Synthetic IV in plasmode simulated datasets. To explore if known confounders, all of them balanced in falsification tests, could be driving the bias.

Methods: Plasmode simulation, a re-sampling method with replacement that can keep the confounding structure of the data. Data was re-sampled from the original UTMOST effectiveness study. We used surgeon preference as present in the dataset as an IV and constructed a totally synthetic IV with a similar shape as the real-world IV. Both continuous IVs were dichotomised on median. Exposure was created as a function of the IV and known confounder effects, with different IV strengths. The outcome was created as a function of a pre-defined true treatment effect and the known confounder effects. We compared the bias and coverage of the real-world IV and the Synthetic IV on the estimation of treatment effects in a two-stage least squares regression. We explored the same metrics in the real-world IV method, in several simulations, with only introducing the effects of one confounder at a time (instead of the whole list of confounders) from a list of 24 hand-picked known confounders.

Results: Synthetic IV coverage was optimal for all IV strengths, with 96% (88% - 91%) coverage, but a relative bias of 100%, with an IV strength OR=50. The real-world IV coverage was systematically bad for all strengths, with 55% (52% to 58%) for the same OR=50 IV strength and a relative bias of 350%. After repeating the simulation of the real-world IV with one single confounder at a time both in the exposure and outcome generation, some variables introduced substantial bias and resulted in a coverage lower than 80%: for pre-operative knee score (221% bias), deprivation (54% bias) and surgeon output (33% bias).

Conclusions: The traditional falsification method based on covariate imbalance for known confounders failed to detect confounding. New methods, such as plasmode simulations evaluating the effect of one known confounder at a time could be a useful diagnostic to detect residual confounding. More research is needed to understand the relationships between IV and confounding, especially in surgical and device epidemiology.