(282) Validation to Estimate Sensitivity Along with Positive Predictive Value: A Case Study

Publication Number: 956

Background: To validate an algorithm (ALG) retrieving cases of a disease from real-world data sources, positive predictive value (PPV) is often estimated, while sensitivity is rarely assessed. When a sensitive auxiliary algorithm can identify all cases in the population of interest, estimation of sensitivity of ALG is possible, using a published analytical formula based on PPV of both algorithms. If this assumption does not hold, the true sensitivity of the algorithm is lower than the result of the formula, i.e., the formula estimates the best case scenario for the true sensitivity of ALG, only assuming that the validation can accurately differentiate between true cases and non cases

Objectives: To illustrate how PPV of an auxiliary algorithm can be used to estimate sensitivity of an algorithm retrieving cases of angioedema

Methods: Diagnosis of angioedema (AE) recorded as ICD9CM 9951 in the ER or during hospital admissions between 2016-2019 were extracted from an administrative data source of the Tuscany region in Italy. As an auxiliary algorithm, diagnoses were also extracted compatible with angioedema (hypersensitivity: HYP), ICD9CM: 9950, 99527, 7088, 7080, 7081, 7089, 37633, 37482, 47825, 4786, 47875, 5088, 7823, 9952. All records of AE and a sample of HYP were sampled for validation by two assessors to classify true cases of angioedema. However, due to COVID restrictions, assessors could not access medical charts. Diagnoses recorded during the hospitalisation, free text observations during ER, and all medicine dispensing recorded up to 30 days following diagnosis were used instead. PPV of AE and HYP was calculated as the proportion of true cases out of all cases. The best case scenario for the sensitivity of AE was calculated using the formula SE = C_AExPPV_AE/( C_HYPxPPV_HYP+C_AExPPV_AE), where C_AE and C_HYP were the number of cases retrieved by AE and HYP, respectively

Results: Cases identified for AE and HYP were 34 and 451, respectively. Validation resulted in 12 true cases out of 34 for AE (PPV: 35.3%) and 8 in a sample of 96 for HYP (PPV: 8.3%). Using the formula, we estimated that there were at least 451x8.3% = 37 cases of angioedema in the population that were false negative for AE: the best case for sensitivity of AE was 12/(12 + 37) = 24.5%

Conclusions: Based on this illustrative example, the sensitivity of AE in the ARS data source was not higher than 24.5%. This methodology can be leveraged in pharmacoepidemiologic studies to provide best case sensitivity of case-finding algorithms. This case study should be considered relevant from a methodological point of view, but not from a clinical perspective, due incomplete case adjudication