How to tell if an AI feature is worth building

Most AI proposals we are asked to assess do not fail on feasibility. The model can usually do the task. They fail because nobody established what the task was worth doing in the first place.

Here is the test we apply during a feasibility engagement, in the order we apply it.

1. Is there a measured baseline?

Before anything else: what does the current manual process cost, in time and in errors?

Not an estimate. A measurement. If somebody spends part of their week on this task, find out how much of their week, over a real sample. If the process produces errors, find out the actual rate — usually by taking a hundred recent cases and checking them.

This is tedious and it is the single most valuable hour of a feasibility study, for two reasons.

The obvious reason is that you cannot calculate return on investment without it. The less obvious and more important reason is that a surprising number of AI proposals die at this step, and they die cheaply. We have measured processes that everyone assumed consumed a day a week and found they consumed about forty minutes. That is not a problem worth solving with a model.

2. What does a wrong answer cost?

Language models are wrong sometimes. This is not a defect to be engineered away; it is a property of the technology, and any vendor implying otherwise is either inexperienced or selling.

The right question is therefore not "will it be accurate?" but "what happens when it is wrong, and who finds out?"

Three broad categories, in descending order of how comfortable we are building them:

Wrong answers are caught cheaply. A human reviews output before it takes effect — extracted invoice fields shown for confirmation, a draft reply an operator edits before sending. The model saves the effort of creating; the human retains the decision. This is where most of the genuine value in business AI currently sits, and it is unglamorous.

Wrong answers are self-correcting. Search ranking, suggestions, recommendations. A poor result is a mild annoyance and the user simply moves on. Also fine.

Wrong answers propagate silently. Automatic posting to a ledger, unreviewed customer communication, anything feeding a downstream system that treats input as authoritative. Here the cost of a single bad output can exceed the entire value of the automation. We will build these, but only with constrained output schemas, confidence thresholds and an explicit human path for the uncertain cases — and we will say so up front rather than after the invoice.

3. Could you just write the rule down?

This is the question that saves clients the most money, and it is the one that gets asked least.

If a competent person could write the logic as a set of rules in an afternoon, write the rules. They will be faster, cheaper, fully deterministic, debuggable, and they will not cost anything per invocation. "Route the enquiry to the sales team if it mentions pricing" is an if statement, not a machine learning problem.

The genuine case for a model is judgement over unstructured input: text, images, documents, or messy real-world data where the rules are numerous, fuzzy, and change. Classifying free-text complaints into fourteen overlapping categories. Extracting fields from invoices that arrive in two hundred different layouts. Matching part descriptions that three different suppliers have each written their own way.

That last one is a good example of the boundary. Exact part-number matching is a database join. Matching "FRT BUMPER COVER PRIMED - CAPA" against "Bumper Cover, Front, Primed (CAPA certified)" is the kind of fuzzy judgement where a model genuinely earns its cost.

What a serious feasibility phase produces

If a proposal survives those three questions, the feasibility work should produce four things before any production build starts.

An evaluation set. Fifty to two hundred real cases from your data, with correct answers agreed by someone who knows the domain. Without this you cannot tell whether a change to a prompt or a model helped or hurt, and you will spend the whole project guessing. Building it is boring and it is not optional.

A measured accuracy number, on your data. Not benchmark figures, not a vendor's marketing. Your cases, your edge cases, your messy inputs.

A running-cost model. Tokens per operation, operations per month, cost per model tier. Usually this comes out small relative to the manual cost — but it should be calculated rather than assumed, because the arithmetic occasionally goes the other way at high volume.

A data-readiness and privacy position. Where the data goes, under what terms, whether it can be used for training, and whether any of it cannot leave your infrastructure at all. Settle this before the build, not during procurement review afterwards.

The uncomfortable conclusions

Two things we say to clients more often than they expect.

The first is that a meaningful share of AI proposals should be conventional software instead. Better data validation, a fixed integration, a well-designed form — these are less exciting and frequently the right answer.

The second is that when AI is the right answer, the hard part is almost never the model. It is the evaluation harness, the schema validation, the human review workflow, the error handling, and the integration with systems that were designed on the assumption that their inputs were always correct. The model is perhaps fifteen per cent of the work. Budget for the other eighty-five.