AI coding
How to pilot AI coding on your own charts: a 30-day checklist
Short answer
A good AI coding pilot runs the tool on a sample of your own de-identified charts that have already been coded and audited, compares its codes with your final codes, and measures what matters to you: codes it got right, codes it missed, principal diagnosis and DRG agreement, and how long a coder takes to check each chart. Decide the sample, the metrics and the pass mark before you start.
Key facts
- Test on your own charts, not the vendor's demo cases: your documentation style is what the tool has to handle.
- Use charts that are already coded and audited, so there is a reference to compare against.
- Measure both precision (suggested codes that were right) and recall (correct codes it found).
- Review every disagreement: sometimes the AI is right and the original code was wrong.
- De-identify charts first, or put a Business Associate Agreement in place before sharing PHI.
Before you start
- Write down the goal: faster coding, fewer denials, better specificity, or coverage for a backlog.
- Choose a sample that matches your real mix: inpatient and outpatient, your main specialties, simple and complex cases. A few dozen to a hundred charts is usually enough to see patterns.
- De-identify the charts, or arrange a BAA with the vendor before any PHI is sent.
- Agree the pass mark in advance, for example a level of principal diagnosis agreement, so the decision isn’t made on impressions.
- Pick two or three coders to run the pilot, including at least one sceptic.
What should you measure?
| Metric | What it tells you | How to measure |
|---|---|---|
| Precision | How many suggested codes were right | Correct suggestions ÷ all suggestions |
| Recall | How many of the right codes it found | Correct suggestions ÷ codes on the audited chart |
| Principal diagnosis | Whether it sequences the main condition correctly | Share of charts where it matches |
| DRG agreement | The financial effect of its codes (inpatient) | Share of charts grouping to the same DRG |
| Review time | Whether checking its work is faster than coding | Minutes per chart, with and without the tool |
| Evidence quality | Whether coders can verify a code quickly | Coder rating per chart |
A week-by-week plan
- Week 1: set up accounts, agree the metrics, prepare and de-identify the sample.
- Week 2: run the sample through the tool without looking at the original codes.
- Week 3: compare against the audited codes, and review every disagreement with a senior coder.
- Week 4: score the metrics against the pass mark, collect coder feedback, and decide.
Red flags during a pilot
- The vendor won't run the tool on your own charts.
- Codes come without the evidence behind them, so you can't tell why a code was chosen.
- It can't say which code-set version it used.
- Patient data may be kept or used for training.
- It suggests codes the documentation doesn’t support, especially CCs and MCCs.