PC1020: Accuracy and Interrater Reliability of Point-of-care Ultrasonography Image Interpretation for Intussusception

Background: Point-of-care ultrasound (POCUS) use is increasing in pediatric emergency departments (EDs). Our objective was to determine the accuracy and interrater reliability of 1) POCUS image interpretation for identification of intussusception and 2) reliability of secondary signs associated with intussusception, among expert compared to novice POCUS reviewers.

Methods: We conducted a planned secondary analysis of a prospective, convenience sample of children aged 3 months to 6 years who were evaluated with POCUS for intussusception across 17 international pediatric emergency departments between October, 2018 and December, 2020. A random sample of 100 POCUS examinations were reviewed by novice and expert POCUS reviewers. The primary outcome was identification of any intussusception or no intussusception. Secondary outcomes included intussusception size, presence of trapped free fluid or echogenic foci. Accuracy was summarized by sensitivity and specificity, which were estimated via generalized mixed effects logistic regression. Interrater reliability was summarized via Light’s kappa statistics with bootstrapped standard errors (SE). Accuracy and reliability of expert and novice POCUS reviewers were compared.

Results: A total of 18 expert and 16 novice POCUS reviewers completed reviews. The average expert sensitivity was 94.5% (95% CI: 88.6, 97.5) and specificity was 94.3% (95% CI: 90.3, 96.7), which were significantly higher than average novice sensitivity of 84.7% (95% CI: 74.3, 91.4) and specificity of 80.4% (95% CI: 72.4, 86.7) (Table 1, Figure 1). Kappa was significantly greater for expert (0.679, SE 0.039) compared to novice POCUS reviewers (0.424, SE 0.044; difference 0.256, SE 0.033). For our secondary outcome measure of intussusception size, kappa was significantly greater for experts (0.661, SE 0.038) compared to novices (0.397, SE 0.041; difference 0.264, SE 0.029). Interrater reliability was weak for expert and minimal for novice reviewers regarding detection of trapped free fluid and echogenic foci.

Conclusion: Expert POCUS reviewers demonstrate high accuracy and moderate interrater reliability for identification of intussusception via image interpretation, and perform better than novice reviewers. In contrast, novice POCUS reviewers demonstrated modest accuracy and weak interrater reliability. Our results suggest that curricula emphasizing image interpretation, in addition to image acquisition and clinical integration, may be needed prior to implementation in clinical practice. Our results have implications for pediatric hospital systems seeking to expand POCUS use and training, as well as stakeholders in medical education.