Validation of a Model for Predicting Surgical Site Infection after Pelvic Organ Prolapse Repair

Objectives: To externally validate a prediction model for surgical site infection (SSI) after pelvic organ prolapse (POP) surgery.

Methods: Previously our group developed a prediction model for SSI following POP repair based on the CMS 5% Limited Dataset. Two additional large data sets (Premier Healthcare Database and the National Surgical Quality Improvement Project data) were then used to validate the model. The original model included the following variables: age (modeled via a 5 knot restricted cubic spline), peripheral vascular disease, diabetes with complications, use of vaginal prosthesis, abdominal sacrocolpopexy, sling, blood transfusion, para- or hemi-plegia and intraoperative gastrointestinal tract injury. This model was then applied individually to the Premier dataset to calculate the risk of SSI 90-days and to the NSQIP dataset to calculate the risk of SSI 30 days after POP surgery. Discrimination (AUC, Brier) and calibration were assessed. When poor calibration was identified, different re-calibration approaches were considered; these included (1) re-estimation of the intercept, (2) re-calibration of model predictions using the slope of the logistic calibration curve, and (3) re-estimation of the model coefficients in the validation data set.

Results: Validation with NSQIP: The rate of SSI in the original (CMS) data set was 4.7%, and in NSQIP the rate of SSI was 1.5%, therefore the model exhibited poor ‘calibration-in-the-large’. A model with re-estimated intercept showed poor logistic calibration (slope = 0.74). After recalibration of the model, the optimism corrected AUC was 0.596 and Brier Score of 0.014, which was similar to the performance of the original, AUC=0.603. Decision curve analysis suggest an overall benefit of intervention for SSI was applied to those with a risk of infection between 1-2%. Re-estimation of the model using NSQIP did not result in a substantially better performing model.

Validation with Premier: The rate of SSI in Premier was 1.9%. The original model showed similar discrimination to that in the original data set (AUC = 0.605; Brier = 0.019) but exhibited poor logistic calibration (slope = 0.84). A model with re-estimated coefficients showed slightly better discrimination performance (internally validated AUC = 0.641; Brier Score = 0.019) and good logistic calibration (slope = 0.997). Decision curve analysis demonstrated that an SSI preventive strategy may be of benefit for risks up to 4%.

Conclusions: A prediction for SSI after POP surgery developed using CMS data shows discrimination performance that is as good or better when applied to a more general population within NSQIP up to 30 days after surgery and within Premier up to 90 days after surgery. That being said, the model did require re-calibration to improve the correspondence of observed and predicted probabilities. Prospective validation of SSI prediction using this model may help identify patients who may benefit most from a selective SSI prevention strategy.