The pattern is consistent enough to be predictable. A pilot is built in a few weeks. The demo is genuinely impressive. Everyone is enthusiastic. Then it spends nine months not quite reaching production, and eventually stops being mentioned.
The technology is rarely the reason.
It was demoed on the easy cases
Pilots get built against clean, representative examples. Production brings the tail: the scanned document that’s slightly rotated, the invoice in a second language, the email that’s three forwarded threads deep, the form someone filled in wrong.
A system that handles 80% of cases sounds close to done and often isn’t, because the remaining 20% needs detection, routing, human handling, and a way to feed corrections back. That machinery is most of the engineering, and it’s absent from every pilot.
What separates the ones that ship: they test against a random sample of real inputs from the start, including the messy ones, and they design the exception path before they optimise the happy path.
Nobody owned the last mile
Pilots are typically run by an innovation function, a data team, or an enthusiastic individual. Production requires an owner in the operational team — someone whose job improves if it works and who will chase the integration, the training, and the process change.
Without that person, the pilot is technically successful and organisationally orphaned. It has no route into anyone’s daily work.
What separates the ones that ship: the operational owner is named before the pilot starts, and is the person who defines what success means.
The integration was left until last
The AI part is often the easy part. Getting data out of the CRM, writing results back into the ERP, handling authentication, and dealing with the internal system whose API is a SOAP endpoint maintained by a supplier who no longer supports it — that’s where months go.
Teams underestimate this because the pilot ran on a CSV export.
What separates the ones that ship: integration is proven first, sometimes with a stub in place of the model. If you can’t write a result back into the system of record, the model’s quality is irrelevant.
Success was never defined
“Improve efficiency” cannot be evaluated, so the project cannot be declared finished or abandoned. It drifts until interest fades.
What separates the ones that ship: a number agreed in advance — a percentage of documents processed without intervention, an accuracy floor against the human baseline, a cost per unit — and an agreed date to judge it. A pilot that fails a clear test in six weeks is a good outcome, cheaply obtained.
The comparison was against perfection
AI outputs get scrutinised in a way the existing process never was. A model at 94% accuracy is rejected while the manual process it replaces — never measured — runs at 91% and nobody knows.
What separates the ones that ship: they measure the human baseline first. It reframes the discussion from “is this perfect” to “is this better than what we do now, at what cost.”
The review interface was an afterthought
Where a human checks output, the review experience determines the economics. If verifying takes nearly as long as doing the task, there’s no saving — and this is decided by interface design, not model quality. Showing the source document beside the extraction with the relevant field highlighted can turn a 90-second review into a 5-second one.
What separates the ones that ship: the review interface is treated as core product work, because it is.
Nobody planned for model change
Providers deprecate models and change behaviour. A prompt tuned against one version can degrade against its successor. Teams that pinned a version and never revisited it discover the problem when the version is retired.
What separates the ones that ship: an evaluation set — a few dozen real cases with known correct answers — that can be re-run whenever anything changes. It takes a day to build and makes every subsequent change safe.
The process underneath was broken
Sometimes the honest finding is that the task was only necessary because of an upstream problem. Automating the re-keying of data between two systems is worth less than making the two systems talk. Automating chasing for missing information is worth less than not letting the form be submitted incomplete.
What separates the ones that ship: they ask why the task exists before automating it, and occasionally conclude the task should be eliminated instead. That’s a better outcome, though it rarely gets counted as an AI success.
A short pre-flight checklist
Before starting, be able to answer:
- Who in operations owns this, and how does their job improve?
- What number defines success, and by when?
- What is the current human accuracy and time-per-unit?
- Have we tested against a random sample of real inputs, including bad ones?
- What happens to the cases the system can’t handle?
- Can we write results back into the system of record today?
- How will a reviewer verify output in seconds rather than minutes?
- How do we detect quality degradation after launch?
Most stalled pilots would have been stopped, redirected, or fixed early by questions 1, 2 and 6.
Frequently asked questions
How long should a pilot take? Weeks, not months. If it can’t be evaluated in six to eight weeks, the scope is too large to learn from.
Should we run several pilots at once? Better to complete one properly. Parallel pilots typically compete for the same scarce attention and all stall together.
What if accuracy isn’t good enough? Narrow the task. A model that struggles across twelve document types often performs well on the three that make up most of the volume. Automate those and route the rest to a person.
Is it worth restarting a stalled pilot? Often, if the failure was ownership, integration or definition rather than capability. Restart with the checklist above rather than resuming where it stopped.
Internal links: #15 where AI automation pays off, /services, /contact. External references: your provider’s model deprecation and versioning policy — worth reading before you build against a specific version. CTA: “Stalled between pilot and production? That gap is usually engineering, not AI.” → /contact