Getting Past The Test Drive: Getting RCM AI From Demo To Production
Buying anything with an engine is hard when the salesperson picks the route. Revenue cycle AI is no exception.
Every demo I've watched takes the same drive. Smooth pavement, sunny afternoon, a clean denial and a chart that answers every question on page one.
So what happens at mile 60,000, in February, on a road nobody mapped?
The questions on the lot
HFMA recently asked three hospital revenue cycle leaders for the one AI question their peers should be asking. Their answers deserve to be taped to every CFO's monitor.
Caris Newton at Corewell Health asked how you'll know, six or twelve months after go-live, that a tool is still delivering, and who's accountable for proving it. Sheila Augustine at Nebraska Medicine said to start with the business problem and a measured baseline, not the technology. Nikki Harper at Mayo Clinic asked how you decide what stays with human judgment, and how you keep governance running long after the contract is signed.
The research behind the piece is sobering. UPMC and KLAS interviewed 27 health system leaders, and 16 of them said their AI strategy is still developing.
One said advanced.
One.
The view from the other side of the table
Here's what I keep coming back to. Those are the right questions for a health system to ask of itself.
They aren't tough enough to ask of a vendor.
I spent years building AI on the payer side, the models that review claims and authorizations before a person ever sees them. Over there, nobody gets to show a sunny-day demo to the medical director and walk out with a signature. You get asked how the model behaves when policy changes underneath it, what it does with a chart that contradicts itself, and how many dollars it held.
Payers kick the tires like they're buying a used car from a cousin they don't trust.
Providers usually take the test drive and sign.
Seven questions to ask before you sign
Show me paid, not sent. Ask the vendor to report recovered dollars, not appeals submitted. An appeal that's clinically airtight and never paid is a write-off with extra steps. I ask every team the same thing: do you want to be right, or do you want to get paid?
What happens at month nine? Payers rewrite policies constantly, and a denial pattern you beat in March can come back in October with new language. Ask how the tool's knowledge stays current, who notices when overturn rates slip, and how fast it gets fixed. If the answer is "the model learns," ask again.
Who decides not to appeal? A tool that appeals everything is a cost engine wearing a revenue badge. Every denial belongs in one of three lanes: correct and resubmit, appeal, or write off. Ask to see the triage logic and how often it says no.
Can every sentence be traced? An appeal letter is an argument to someone who reads them for a living. Every clinical fact should point to a page in the chart, and every policy reference to the actual policy. One invented citation loses that appeal and your credibility on the next ten.
What does the human actually check? "Human in the loop" is on every slide. Ask what the reviewer sees, how long they get, and how often they reject the draft.
Does it know how the other side thinks? Payers adjudicate with policy logic, clinical criteria and, more and more, their own AI. A vendor that has never sat on that side is guessing what the reviewer needs to see. Ask who on their team has built or run adjudication.
Whose baseline is it? Measure your own denial rate, overturn rate, days to resolution and cost per appeal before go-live. If the vendor supplies the "before" number, they're grading their own test.
A reviewer who never says no isn't oversight.
It's a signature.
Bringing it home
At PromptWrx, we build this stuff for a living, so take this for what it is. I want health systems to buy AI. I just want them to buy it the way a payer would.
The leaders HFMA talked to are right that enthusiasm isn't the bottleneck. Discipline is, and discipline starts on the lot, before the keys change hands.
A demo is a test drive on a sunny day.
The baseline belongs to the buyer.
If you can't cite it, don't send it.
Paid dollars, current policy, triage, traceability, real oversight, payer fluency, your own baseline.
Next time, ask to drive it through a Nebraska February. Bring your own potholes.