Our AI pilot failed. What now?

It demoed well. Six months later the licences are still being paid and nobody opens it.

This is the most common position a company is in when it calls us, and it is a better position than it feels like. You now know something about your own company that the people starting their first pilot do not, and the second attempt is usually cheaper, faster and more likely to work. It just has to be aimed at the right failure.

MIT's Project NANDA put a number on how common this is: 95% of organisations reported no measurable profit-and-loss impact from formal generative AI investment, against thirty to forty billion dollars spent. You are not in an unusual minority. You are in the majority, and the majority mostly does not find out why.

The four failures#

1. Nobody agreed what success was#

The most common, and the easiest to miss, because the pilot can look like a success and still be one of these. There was no number before the work started, so afterwards there was nothing to compare against, and the conversation about whether it worked became a conversation about whether people liked it.

How to tell. Ask what the metric was and when the baseline was taken. If the answer is a date after the project started, or there is no answer, this is your failure.

What it costs to fix. Almost nothing. Take the measurement properly, then decide. Some of these pilots turn out to have worked.

2. The wrong job was chosen#

The thing was built well and the job was not worth doing. Often the job was picked because it was visible, or because a vendor had a demo for it, rather than because it was where the hours went.

How to tell. Ask how the job was chosen. If the answer involves a vendor, a conference or an executive's idea, rather than somebody timing the work, this is it.

What it costs to fix. A proper look at where time actually goes, two to three weeks. The build that follows is usually smaller than the one that failed.

3. It worked and nobody used it#

The system does what it was meant to. The team carried on doing it the old way.

This is almost always because the new way was not actually faster for the person doing it, whatever the aggregate numbers said, or because it sat outside the tools they already have open.

How to tell. Ask the people who were supposed to use it. Not their manager. They will tell you in one sentence and it will be specific.

What it costs to fix. Sometimes very little. Moving a working thing into the tool somebody already has open is a small piece of work with a large effect.

4. It was never going to reach production#

A demonstration built against a copy of the data, with no evaluation, no monitoring and no path through your security review. It was always going to stop where it stopped.

How to tell. Ask what would have had to happen for it to go live, and whether anybody had written that down before the work started.

What it costs to fix. The most expensive of the four, because the answer is usually to build it again properly. The pilot was not a first version, it was a sales artefact.

Why the second attempt is cheaper#

Three reasons, and they are real rather than consoling.

You know which systems are hard to reach, because somebody already tried. You know who in the company will and will not adopt something, which is normally learned the expensive way. And you have an organisation that has stopped believing the demo, which sounds like a disadvantage and is the opposite: nobody will approve a second pilot without a number attached, which is the discipline that was missing the first time.

What not to do#

Do not start again with a different vendor and the same brief. The brief is what failed. A new supplier reading the same paragraph will build a different version of the same mistake.

Do not conclude that AI does not work here. Four failures with four different fixes have been collapsed into one verdict, and the verdict is usually wrong. The honest version is narrower: this job, done this way, did not pay.

Do not skip the measurement again. The second pilot is where the discipline either arrives or does not.

Where to start#

The cheapest next step is a day working out which of the four happened, because all four have different costs and three of them do not need a rebuild.

If it turns out to be the second failure, the wrong job, then the work is finding the right one, which is set out in how to decide what AI not to build. If it was the first, the fix is measurement, and how to measure a process nobody has ever measured covers taking a baseline on something that was never instrumented. If you are about to brief somebody for the second attempt, write it down differently this time.

Bring us the pilot that failed and we will tell you which of the four it was. Thirty minutes, and it costs nothing.