(132) Accounting for Time Trends in Classification Parameters Using the Adaptive Validation Approach

Publication Number: 463

Background: Validation substudies are an important component of pharmacoepidemiology studies using claims-based algorithms for outcome assessment. However, conventional approaches to validation substudy design (i.e., simple random sample, balanced design) recommend sampling participants once follow-up has been completed. The adaptive validation approach allows for prospective monitoring of validation data, enabling detection of a time trend and subsequent adjustment to the validation approach.

Objectives: To evaluate performance of the adaptive validation approach in detecting time trends in classification parameters and to develop recommendations for sampling validation data to account for time trends in outcome misclassification.

Methods: We simulated 10,000 datasets in which 100,000 cohort members were enrolled over a 5-year period and followed for two years for occurrence of an event. In each simulated cohort, we assumed an exposure prevalence of 20% and the true 2-year risk of the outcome in the exposed to be 10% and 5% in the unexposed. A time trend in outcome misclassification was induced in each simulated cohort by manipulating the degree of change in outcome misclassification over time and assuming that misclassification decreased over time. Blocks of 10 participants by observed outcome status within strata of the exposure were sampled in order of outcome ascertainment and classification parameters were computed using a beta-binomial Bayesian model to iteratively update the positive and negative predictive values (PPV and NPV, respectively).

Results: In our preliminary simulation results, we observed a positive time trend in the classification parameters, as expected. The ability of the adaptive validation approach to detect a time trend in the classification parameters was dependent on the degree of change in outcome misclassification.

Conclusions: Our simulation results support the need for analyzing changes in validation parameters over the follow-up period. When a time trend in classification parameters exists, conventional validation approaches using a single summary classification parameter value will misrepresent the classification accuracy over most of the study’s outcome ascertainment period.