(163) Development and Evaluation of The Algorithm Certainty Tool KIT (ACE-IT) to Evaluate Safety Outcomes Using Electronic Medical Record and Claims-Based Algorithms

Publication Number: 1208

Background: Electronic health record or medical claims-based algorithms (i.e., operational definitions) are used to evaluate safety outcomes using real world data. However existing tools do not allow researchers and decision makers to adequately appraise whether a particular algorithm is fit for purpose to support regulatory decisions.

Objectives: Our objective was to develop an ALgorithm CErtainty Tool KIT (ACE-IT Tool) to enable regulatory decision makers and other stakeholders to appraise whether an algorithm is fit for purpose for evaluation for outcome evaluation.

Methods: We drafted a tool comprising a set of questions informed by regulatory guidance documents, existing instruments, and publications. We conducted a scoping review to identify algorithms for ischemic stroke, an important safety endpoint which served as an exemplar. We conducted a three-round online Delphi panel to develop and refine the tool and achieve consensus on items (>70 % agreement). The panel was composed of regulators, researchers, methodologists, pharmacoepidemiologists, and cardiologists. We conducted a qualitative analysis of panel responses. Four pairs of reviewers independently evaluated two ischemic stroke algorithms to test its application and further refine the tool.

Results: The ACE-IT tool was developed. The panel reviewed and made changes to the initial list of 77 proposed items. The panel achieved consensus and stability of ratings on 38 items. These included items across the sections on internal validity, external validity, and design and conduct. The internal validity section comprises of items on objectives, data sources, population, outcomes, design and setting, statistical methods, reference standard, accuracy and strengths and limitations. The external validity section included items that assess the generalizability to a proposed target study. The section on design and conduct includes items on ethical conduct and reporting. Applying the tool to the two ischemic stroke studies demonstrated challenges in application, clarified concepts and refined the organizational structure of the tool. We developed a user guide, with explanation and elaboration for each item, essential and additional elements, and an illustrative example of a complete assessment. Stakeholder assessment of feasibility, acceptability and appropriateness is ongoing.

Conclusions: The ACE-IT Tool supports a structured, transparent, and flexible approach for decision makers to appraise whether electronic health record or medical claims-based algorithms are fit for purpose to evaluate safety outcomes. Reliability and validity testing using a larger sample of participants in other therapeutic areas is needed to further evaluate its utility for regulatory decision making.