How Choosing the Risk Ratio or the Risk Difference Changes Sufficient Adjustment Sets for External Validity

Background: Pharmacoepidemiologic research generalizing or transporting effect estimates from a study to a target population often suggests conditioning on all effect measure modifiers (EMM) on the scale of interest. EMM is known to be scale dependent, but there is also a rich literature on the nuances of EMM and collapsibility that is often neglected when considering external validity. More specific characterization of EMM and the mathematical nuances of effect measures like the risk ratio (RR) and risk difference (RD) is needed.

Objectives: To describe how the scale of an effect measure (i.e. RD vs RR) alters the types of covariates that must be balanced to generalize or transport internally valid effect estimates to an external target.

Methods: We defined two types of EMM: marginal EMM, where a (unconfounded) effect on the scale of interest differs across levels of a variable; and conditional EMM, where an effect differs across levels of a variable conditional on a set of other variables. These EMM types define three variable classes: Class 1 (conditional and marginal EMM), Class 2 (marginal, but not conditional EMM), and Class 3 (neither marginal nor conditional EMM, still associated with the outcome). We created a study and target population of 1000 individuals with two options for a treatment X, a binary outcome Y, and a binary covariate in Class 1 (M), Class 2 (N), and Class 3 (O) on a linear (when examining the RD) or log-linear (when examining the RR) scale. Each population had different probabilities of M (0.5 in the trial, 0.2 in the target), probabilities of N given M (0.7 if M=1, 0.3 if M=0 vs 0.3 if M=1, 0.7 if M=0), and O (0.3 vs 0.6). Based on hypothetical trials we calculated asymptotic effect estimates in 1) the target population, 2) the study population, 3) the study population weighted with inverse odds of sampling weights (IOSW) to resemble the target with respect to M, 4) the study population IOSW weighted with M and N, and 5) the study population IOSW weighted with M, N, and O.

Results: In the linear model for Y, the RD in the target was 12.0% and the RD in the study was 15.0%. Applying IOSW based on M to the study population resulted in an estimated RD of 12.0%, the true RD in the target. In the log-scale model for Y, the RR in the target population was 1.68 and the RR in the study population was 1.98. Reweighting the study based only on M resulted in a RR of 1.73. Both M and N needed to be included in the sampling weights to reach the target's true RR of 1.68. Class 3 variables like O were only needed to estimate risks or odds ratios, and not RDs and RRs, in the target population.

Conclusions: A richer definition of EMM is needed to identify adjustment sets for external validity. The types of variables that must be accounted for can differ greatly depending on whether the goal is to estimate a RD, RR, or another effect measure.