AI coding

How to pilot AI coding on your own charts: a 30-day checklist

By Claimino · Updated · 2 min read

Short answer

A good AI coding pilot runs the tool on a sample of your own de-identified charts that have already been coded and audited, compares its codes with your final codes, and measures what matters to you: codes it got right, codes it missed, principal diagnosis and DRG agreement, and how long a coder takes to check each chart. Decide the sample, the metrics and the pass mark before you start.

Key facts

  • Test on your own charts, not the vendor's demo cases: your documentation style is what the tool has to handle.
  • Use charts that are already coded and audited, so there is a reference to compare against.
  • Measure both precision (suggested codes that were right) and recall (correct codes it found).
  • Review every disagreement: sometimes the AI is right and the original code was wrong.
  • De-identify charts first, or put a Business Associate Agreement in place before sharing PHI.

Before you start

  • Write down the goal: faster coding, fewer denials, better specificity, or coverage for a backlog.
  • Choose a sample that matches your real mix: inpatient and outpatient, your main specialties, simple and complex cases. A few dozen to a hundred charts is usually enough to see patterns.
  • De-identify the charts, or arrange a BAA with the vendor before any PHI is sent.
  • Agree the pass mark in advance, for example a level of principal diagnosis agreement, so the decision isn’t made on impressions.
  • Pick two or three coders to run the pilot, including at least one sceptic.

What should you measure?

MetricWhat it tells youHow to measure
PrecisionHow many suggested codes were rightCorrect suggestions ÷ all suggestions
RecallHow many of the right codes it foundCorrect suggestions ÷ codes on the audited chart
Principal diagnosisWhether it sequences the main condition correctlyShare of charts where it matches
DRG agreementThe financial effect of its codes (inpatient)Share of charts grouping to the same DRG
Review timeWhether checking its work is faster than codingMinutes per chart, with and without the tool
Evidence qualityWhether coders can verify a code quicklyCoder rating per chart

A week-by-week plan

  • Week 1: set up accounts, agree the metrics, prepare and de-identify the sample.
  • Week 2: run the sample through the tool without looking at the original codes.
  • Week 3: compare against the audited codes, and review every disagreement with a senior coder.
  • Week 4: score the metrics against the pass mark, collect coder feedback, and decide.

Red flags during a pilot

  • The vendor won't run the tool on your own charts.
  • Codes come without the evidence behind them, so you can't tell why a code was chosen.
  • It can't say which code-set version it used.
  • Patient data may be kept or used for training.
  • It suggests codes the documentation doesn’t support, especially CCs and MCCs.