Your AI pilot probably doesn't need more training

You're reading The AI Directive — practical AI intelligence for UK business leaders. Subscribe here if someone forwarded this.

When an AI pilot starts to wobble, the usual answer is more training.

Another workshop. Another prompt guide. Another Teams session where someone shares their screen, asks Copilot to summarise a meeting, and everyone politely agrees this is the future.

Training has a place.

But a lot of the time, it is being used to dodge the more awkward question:

That is not a new AI problem.

It is an old software problem wearing a newer shirt.

I started in IT in the 1990s, and I am old enough to recognise the pattern. You get into delivery too quickly. Requirements are a bit woolly. Scope is still moving. The design feels clear enough to proceed, because everyone is keen to see something working.

Then it reaches test.

And suddenly the thing everyone thought they wanted is not quite the thing they need.

That is when the cost goes up.

There is an old idea often described as the cost-to-fix curve. Fix a requirement problem early, when it is still a conversation or a sketch, and the cost is small. Fix it later, once the product is built, integrated, tested, trained and politically committed, and the cost is enormous.

Small at the narrow end.

Huge once you are at the bottom with a built thing, a deadline, and a room full of people saying “can we just change it?”

AI pilots are running into exactly the same issue.

The language is different. In 2026 we talk about Copilot, agents, prompts, knowledge sources and adoption. In the 1990s we talked about requirements, scope, design, testing and change control.

The underlying mistake is familiar:

We start solving before we are clear what problem we are solving.

A lot of AI pilots do not fail loudly. They fade.

People try the tool. They are mildly impressed. A few enthusiasts keep experimenting. Everyone else returns to the old process because the new behaviour was never properly designed into the work.

Then the adoption dashboard looks disappointing, and someone says the business needs more training.

Maybe.

But if the use case is vague, training will not fix it.

If the workflow is not defined, training will not fix it.

If nobody owns the outcome, training will not fix it.

If the pilot is trying to prove “AI” rather than improve a specific piece of work, training will not fix it.

This is where the old IT scar tissue is useful.

A bad requirement discovered early is annoying.

A bad requirement discovered after launch is expensive.

An AI pilot attached to the wrong problem follows the same curve. The later you admit the scope is wrong, the more sunk cost, expectation, training, comms and reputational mess you have to unwind.

That is why I do not think the first rescue move should be “more training”.

The first rescue move should be a decision.

Kill it, narrow it, rescue it or scale it.

Do not leave it hovering in that awkward middle state where everyone agrees AI is important but nobody can point to the operational gain. That is how pilots become corporate houseplants: visible, occasionally watered, and mostly decorative.

There are four honest options.

Kill it if the problem is not worth solving, the pain is not real, or the expected value is too weak.

That is not failure. That is portfolio hygiene.

Narrow it if the pilot is too broad. “Use AI in sales” is not a pilot. “Turn call notes into CRM-ready follow-ups within 10 minutes” is closer.

Rescue it if the use case is right but the operating model is missing: unclear owner, weak source material, no review process, vague prompts, no safe boundary, no measurement.

Scale it only if the pilot has evidence: repeatable use, measurable benefit, named owner, known risks, training materials, support route and a plan for what happens when it is wrong.

Most organisations want to jump from experiment to scale because it sounds more impressive.

The mature move is often narrowing.

A narrow AI pilot is not less ambitious. It is more honest.

Last week I wrote about choosing the lightest Microsoft AI surface that can safely do the job. This is the same principle applied to pilots.

The goal is not “more AI”.

The goal is better work.

If the pilot is not making a specific piece of work better, stop admiring the technology and go back to the requirement.

Unfashionable, yes.

Usually where the answer is hiding.

If you have a stalled AI pilot, ask these questions before booking another training session.

*1. What problem are we actually solving?*

Not which tool are we using. Not which model is cleverest. Not which demo impressed the room.

What problem?

If you cannot write it in one plain sentence, the pilot is probably too vague.

Good: “reduce time spent turning meeting notes into agreed actions.”

Weak: “improve productivity in operations.”

One is a workflow. The other is a wish.

*2. What requirement did we assume rather than define?*

This is the old software-development trap.

Everyone thinks the requirement is obvious until test, launch, or adoption proves otherwise.

For AI, the hidden assumptions are often things like:

Some of those may be true.

The dangerous ones are the assumptions nobody wrote down.

*3. Who owns the outcome?*

Not who owns the software.

Who owns the business result?

An AI pilot owned only by IT often becomes a platform demonstration. A pilot owned only by the business may miss governance, security and integration issues. You usually need both, but one named business owner must care whether the work actually improves.

*4. What does success look like?*

Do not settle for “usage”.

Usage is a signal, not the goal. People can use bad workflows too. We have all seen spreadsheets that should have been pensioned off with dignity.

Look for outcomes:

*5. What is the review and approval model?*

If AI creates drafts, who checks them? If it summarises documents, who verifies important points? If it recommends actions, who approves them? If it touches external communication, where is the stop point?

No review model means the pilot is not ready to scale.

*6. What happens when it is wrong?*

This is where the grown-up conversation starts.

AI will be wrong. So will humans. The question is whether the process detects, contains and corrects the error before it matters.

If nobody has designed that, you do not have an operational pilot.

You have optimism with a login.

I am building more practical AI workflow content on the YouTube channel here:

*Toby Smith-Firth on YouTube*

The theme is not “look what AI can do”. There is enough of that.

The useful question is: what does it take to run AI properly in real work?

1. If a pilot is stuck, do not start with training. Start with the problem, workflow, owner and success measure.

2. Narrowing a pilot is usually a sign of maturity, not retreat. Broad pilots produce vague excitement. Narrow pilots produce evidence.

3. The cost-to-fix curve still applies. A fuzzy requirement is cheap to fix before you build around it. It is much more expensive once the pilot has users, comms, training and expectations attached.

“How long should an AI pilot run before we decide?”

— Impatient but correct

Long enough to see whether the workflow improves. Not long enough to let uncertainty become the operating model.

For a narrow internal workflow, you can often learn a lot in two to four weeks if the pilot is properly designed.

The key is to decide in advance what evidence you need.

At the end, make one of four calls:

What you should not do is let it drift into permanent “pilot” status, where it is too visible to abandon and too weak to matter.

That is not innovation.

That is limbo with a steering committee.

This is where AI work starts to become management work.

Not model selection. Not prompt libraries. Not another demo.

Management work: deciding what matters, who owns it, what good looks like, what risk is acceptable, and when to stop.

The technology is new. The management discipline is not.

I rather wish it were. It would save us all from relearning the same lesson with better branding.

Next week: AI agents are easy to demo and hard to run.

See you Tuesday.

— Toby

@TobySmithFirth on YouTube

These are tools I use personally. Affiliate links marked — I earn a small commission if you sign up, at no extra cost to you.

Some links in this issue are affiliate links. I only recommend tools I actually use.