Validity and Inter-rater Reliability of Keyword-based Operative Report Review for Disease Severity Assessment in Pediatric Appendicitis: Results from a Multi-center Pediatric Research Consortium

Background: Reliable and accurate severity assessment is essential for meaningful comparative performance reporting in pediatric appendicitis. In 2016, four intraoperative findings were identified by NSQIP-Pediatric to differentiate uncomplicated from complicated disease as a means to risk adjust hospital-level outcomes: 1) visible hole, 2) abscess, 3) extraluminal fecalith, and 4) diffuse fibrinopurulent exudate extending beyond the pelvis or right lower quadrant (DFE). While each intraoperative finding has been previously validated as independently associated with increased risk in organ space infection (OSI), variation in interpretation and inter-rater reliability (IRR) during operative report review has not been examined. The purpose of this study was three-fold: 1) Identify areas of subjectivity and disagreement with current NSQIP-Pediatric definitions; 2) Develop keyword-based definition criteria for intraoperative findings to standardize and reduce subjectivity in operative report review; and 3) Assess validity of the severity discrimination and IRR of proposed keyword-based definitions.

Methods: We conducted a retrospective analysis of children undergoing appendectomy for acute appendicitis from January 1, 2015 to December 31, 2020 using NSQIP-Pediatric data from 11 hospitals participating in the Eastern Pediatric Surgical Network research consortium. Operative reports were obtained for all cases. A preliminary review of 250 operative reports by two reviewers was conducted to identify areas of subjectivity/disagreement for the four NSQIP-Pediatric complicated intraoperative findings. Keyword-based definitions for operative report review were then developed using 53 keywords distinctive for intraoperative findings to address subjectivity and clarify definitions (Table 1). Validity of keyword-based definitions was assessed using multivariable regression to estimate the influence of each intraoperative finding on rates of OSI, postoperative abdominal imaging, and revisits and postoperative length of stay (LOS) after adjusting for patient characteristics and hospital-level clustering. IRR was assessed with the Kappa statistic generated from double review by 18 surgeons of 898 randomly selected operative reports.

Results: 6221 operative reports were severity assessed using keyword-based definitions. In the multivariable model, each of the four intraoperative findings as identified by keywords were associated with significant increase in odds of OSI, postoperative abdominal imaging, and increased LOS (Table 2). IRR agreement based on keyword criteria was near-perfect for differentiating complicated (presence of any of the four intraoperative findings) from uncomplicated appendicitis (Kappa 0.85 [95% CI 0.82, 0.89]), and ranged from substantial to near perfect for individual findings (Kappa ranging from 0.68 [95% CI 0.61, 0.77] to 0.83 [95% CI 0.76, 0.89]).

Conclusion: Use of keyword-based definitions was associated with high accuracy and inter-rater reliability in discriminating clinically meaningful disease severity in children with appendicitis. The results of this study may improve the validity of comparative performance assessment and provide a framework to standardize comparative effectiveness research reporting for pediatric appendicitis.