The AI vendor scorecard: compare three firms on one page
A scoring template for comparing AI consultancies on the twelve things that predict whether an engagement works, with what each score means.
Comparing AI consultancies on capability produces three identical answers, because every firm can do the work. The twelve criteria below score the things that actually vary, which are accountability, specificity and what happens when the plan is wrong. Score each firm out of three on each line during the call. A firm scoring well on judgement and poorly on delivery is a different risk from the reverse, and the shape of the scores tells you more than the total.
A shortlist of three firms, scored on capability, comes back as a three-way tie. Every firm can build the thing. Capability is not the variable.
What varies is whether a firm will tell you something you do not want to hear, whether the person who impressed you does the work, and what the contract says when the number does not move. The twelve lines below score those. Fill it in during the calls rather than afterwards, because memory flatters whoever you spoke to last.
How to score
Three points each. 0 is no answer or a deflection. 1 is a general answer. 2 is specific. 3 is specific with an example that cost them something.
Thirty-six available. In practice a good firm lands in the mid-twenties and nobody scores full marks, because line 11 is hard.
The scorecard
| # | What you are scoring | What a 3 sounds like |
|---|---|---|
| 1 | What they last talked a client out of | A named example, with the reasoning, that lost them revenue |
| 2 | Whether "do nothing" is a possible output | Yes, with the last time it happened |
| 3 | The most likely reason this fails | Something dull and operational, like system access |
| 4 | Who does the work, by name | Names, and the same names in week three |
| 5 | Current capacity | A real number of concurrent engagements and who is on them |
| 6 | Price, on the first call | A band, without a discovery call first |
| 7 | What is excluded from the estimate | Integration, data cleanup, the second approach |
| 8 | Notice period and payment terms | Short notice, fees tied to delivery rather than signature |
| 9 | The number this moves, and when the baseline is taken | Before any code is written |
| 10 | What the closing document says if it did not work | They have written one, and will describe it |
| 11 | A client where it went badly | A yes, or a no with a reason that is not evasive |
| 12 | What happens after handover | A named owner on your side and documentation they will write |
What the shape of the scores tells you
High on 1, 2 and 10, low on 4 and 5. A firm with real judgement and thin delivery capacity. Good for an assessment, risky for a build. Consider buying only the first phase from them.
High on 4, 5 and 12, low on 1 and 2. A capable delivery shop that will build whatever you specify. Fine if you already know what to build and have tested that decision elsewhere. Dangerous if you are relying on them to tell you.
High on 6, 7 and 8, low on 9 and 10. Commercially straightforward and indifferent to outcomes. You will get what you ordered, on time, and nobody will tell you it was the wrong order.
Low on 3. Treat this as a strong negative regardless of the total. A firm that cannot name how a project like yours usually goes wrong has not done enough of them, or is managing you.
The three lines that matter most
If the call runs short, score 1, 9 and 10.
Line 1 tells you whether judgement is on offer. Line 9 tells you whether anyone will be able to prove the result. Line 10 tells you what happens when the honest answer is unflattering, which is the only moment the relationship is actually tested.
Everything else is useful and recoverable. Those three are not.
Using it with more than three firms
It gets worse above three. A longlist of six produces six hours of calls and a spreadsheet nobody trusts, because the early calls are scored differently from the late ones.
Score three. If none clears the mid-twenties, the problem is usually the brief rather than the market, and rewriting it produces better conversations than adding a fourth firm.
One limit worth stating
A scorecard is a defence against being charmed, and it has the matching weakness. Scored strictly it favours firms that are good at being assessed, which is a skill some sales teams have and some excellent small operators do not. The founder who answers line 4 with "me, obviously" and has no answer at all for line 5 may be the best choice on the list. Use the scores to find the questions worth pushing on rather than to pick a winner arithmetically.
The long versions
Each line expands. The reasoning behind the eleven questions is in questions to ask an AI consultancy before you sign, the money is in what AI consulting actually costs, and the overall framework is in how to choose an AI consultancy.
We will sit through all twelve on the first call, including line 11. Thirty minutes, no deck.
If you have a budget, a deadline, and no clear answer on which AI project deserves either, that is the conversation to have.
Book the callWritten by
Radwan Altaf
Radwan runs AISynq. Before that he delivered software inside enterprise programmes at DHL, AT&T, DirecTV and Accenture, which is where the habit of measuring a result against its baseline came from. More about the firm.
Read next
- The AI project brief template: fill this in before you ask for a quoteA one-page template with the nine fields that make three quotes comparable, what to write in each, and the two answers that are allowed to be "we do not know".
- How to brief an AI project so the quotes are comparableFive things to write down before you contact anyone, and why a vague brief produces three quotes that differ threefold for reasons nobody can explain.
- How to get budget approved for an AI projectWhat a finance director actually needs to see, the three numbers that carry an approval, and why the AI framing is usually what sinks it.
Work with us
Have a budget, a deadline, and no clear answer on which AI project deserves either?
Book the callGet the next one
New writing in Buying AI help as it goes up, roughly twice a month. One article per email and nothing else in it.