How do I compare two quotes that differ by half?

Ask both to quote the evaluation set, the failure handling, the cost controls and the handover as separate lines. The gap is almost always one or more of those four. The cheaper quote is not cheaper, it is a smaller scope, and the difference arrives as change requests once the feature is in front of customers.

Our agency says they have AI experience. How do I check?

Ask what their evaluation set for the last AI project contained, and how often it ran. A team that has done this answers immediately and specifically. A team that has shipped a demo talks about the model they used, which is the part that matters least.

Is it cheaper to just use our own engineers?

Over a year, usually yes, and the knowledge stays. The cost is the roadmap they are not shipping meanwhile, plus a quarter of platform work the first time round: cost controls, tracing, deployment. The second AI system your team builds is much cheaper than the first, which is worth knowing before you judge the first estimate.

Compare

Specialist AI team or a development agency?

The short answer

Use a development agency when the specification is clear and the work is software. They are cheaper per week, they scale to more hands, and a good one will build a well-tested product from a brief. Use a specialist when the hard part is not the software but the behaviour: whether the output is right often enough, what it costs per user at real volume, and what happens when it is wrong. The tell is in the quote. If the estimate has no line for an evaluation set, the agency has priced a feature and not a system.

Why this comes up

The situation, before the answer

01
You have a specification for an AI feature and two quotes, and one is roughly half the other.
02
The cheaper quote is from an agency you have worked with before and trust on ordinary software.
03
Neither quote mentions how anyone will know whether the thing is any good.
04
The feature will be in front of customers, so being wrong is not free.

Side by side

The comparison, on the dimensions that decide it

Shapes rather than figures. Salary bands, day rates and vendor prices move by an order of magnitude between markets, and a number here that you held us to would be worse than none.

Specialist AI team or a development agency?
Dimension	A specialist AI practiceFewer people, priced on the parts that make an AI system survive.	A development agencyMore hands, cheaper per week, strong on ordinary software.	Your own engineersThey know your systems and they own it afterwards.
Cost per week	Higher. Fewer people, and the estimate includes work the other columns leave out.	Lower, and genuinely lower rather than a trick. This is a real advantage.	Already paid for, and taken off whatever they were doing instead.
Building the software around it	Competent. It is not the reason to hire one.	Strong. This is what they are for, and a good agency does it better than a small specialist.	Strong, and they know where the exceptions in your data live.
Knowing whether the output is good enough	An evaluation set from your real inputs, run automatically, with a number somebody watches.	Usually manual testing against examples somebody chose. Ask to see the plan.	Depends whether they have shipped AI before. If not, this is the quarter nobody planned for.
Cost at real volume	Modelled before the build, including at five and ten times the forecast.	Frequently absent from the quote. It appears later as a bill.	Yours to discover, and you will.
What happens when it is wrong	Designed with the happy path, in the same estimate. Roughly half the work.	Often scoped as a follow-up ticket, which is where it stays.	Depends on how much support pain they have felt personally.
Who owns it in month four	You do, if the handover was in the contract. Insist on the runbook and the evaluation set.	Often the agency, on a retainer. Sometimes that suits you.	You, unambiguously. The strongest argument in this column.

Deciding

Which one, and when

01
A specialist AI practice
- The feature is in front of customers and being wrong costs you something.
- Nobody on the team has run an evaluation set before.
- The unit economics are not yet known and the feature is priced into a subscription.
- It takes actions rather than only producing text.
02
A development agency
- The specification is genuinely clear and the AI part is a small piece of a larger product.
- You need more hands than a specialist has.
- The feature is internal, read-only, and being wrong is cheap.
- You already have an evaluation set and can hold them to it.
03
Your own engineers
- Your engineers have shipped AI to production before.
- The systems involved are unusual enough that outside context would take months.
- This is the first of many, so the learning is worth more than the speed.

The limit

What this comparison gets wrong

We are the first column, so treat this as written by an interested party and check the agency column against your own experience. The comparison also implies you must choose, and the arrangement that works best is frequently neither: an agency builds the product, a specialist owns the evaluation and the cost model, and your engineers sit on both so the knowledge stays. That is harder to contract and it produces better systems. What none of this captures is that a strong agency with one person who has done this before beats a weak specialist, and no comparison table can see that.