The number on the slide is not about you
Every AI vendor pitch eventually gets to the slide with a number on it. 94 percent accuracy. Outperforms GPT-4 on this benchmark. Best in class on that leaderboard. The number is real, in the sense that someone ran a test and got it. What it is not, almost always, is a measurement of how the system will perform on your documents, your questions, your edge cases.
This matters more than it sounds like it should, because the number is doing a specific job in the sales process. It is standing in for the question you actually care about, which is: will this work on my stuff. A published benchmark answers a different question: how did this model do on a fixed, public set of test questions that thousands of vendors have now trained against, tuned prompts against, and in some cases seen the answers to.
None of that makes the vendor dishonest. It makes the benchmark the wrong instrument for your decision, in the same way a car's advertised fuel economy is not a lie but also is not what you will get in city traffic with the air conditioning on. The fix is not to distrust every number. The fix is to build a small test of your own and run it before you sign anything.
Why a public benchmark tells you less than it looks like
A few structural problems show up in almost every published benchmark a vendor points to.
The questions are generic. Benchmarks built for research or marketing purposes are written to be broadly representative, which means they are specific to nothing. Your policy manual has its own vocabulary, its own exceptions, its own way of cross-referencing a clause in one document against a definition in another. A benchmark built from generic customer service transcripts or public legal text will not touch any of that.
The data may already be seen. Large language models are trained on enormous amounts of public text. If a benchmark is old enough and public enough, there is a real chance the model has encountered questions similar to it, or even the exact test set, during training. A high score on that benchmark tells you the model is good at that benchmark. It does not tell you the model is good at reasoning from a document it has never seen, which is what your use case actually requires.
The failure modes do not match yours. A benchmark score is an average across many questions. It hides the shape of the errors. A system can score well on average while being reliably wrong on the exact category of question that matters most to you, for instance anything involving a recent policy update, a number, or a comparison between two documents that partially contradict each other. Averages erase exactly the information you need.
Nobody is checking the vendor's methodology. Was the benchmark run on the same configuration you would actually get. Same retrieval setup, same prompt, same context window limits. Vendors are not required to disclose this, and most buyers do not ask. A benchmark run under generous conditions, with a hand-picked prompt and no production constraints, is not the same product you will receive.
One common view worth taking seriously is that model selection is only one ingredient in whether an AI system works, and not where most of the engineering effort actually goes. If that is right, then a benchmark that only measures the underlying model is measuring the smaller part of the problem. The retrieval quality, the prompt structure, the handling of edge cases and failures, the way the system behaves when it does not know the answer: none of that shows up in a leaderboard score, and all of it is where the real difference between vendors sits.
Build a small evaluation set instead of trusting the leaderboard
You do not need a data science team to test a vendor's claim against your own reality. You need somewhere between twenty and fifty real examples, pulled from your own operation, and a plain method for scoring the answers. This is inside the reach of an operations lead with a spreadsheet and an afternoon.
Pull real questions from real usage
Do not write hypothetical test questions. Go to whatever channel already carries this traffic: a shared inbox, a Slack channel, a help desk queue, a stack of past audit requests. Pull the actual questions people asked over the last few months. Include the awkward ones. The question that took three people and an email thread to resolve is more valuable to your test set than ten easy ones, because it is the awkward questions that separate a vendor who has actually engineered the retrieval and prompt logic from one who has wrapped a generic model around your files and called it done.
Write down the correct answer before you test anything
For each question, write the answer you would consider correct and note which document and passage it comes from. Do this before you run any vendor's system against the set. This step gets skipped constantly, and skipping it is how a demo ends up looking better than it is: it is easy to be generous grading an answer in the moment if you have not already committed to what a correct answer looks like.
Include questions the system should refuse to answer
A meaningful fraction of your test set, maybe a quarter, should be questions the system has no good answer to: something not covered in the documents, something ambiguous, something outdated. The right behavior here is not a wrong answer stated confidently. It is a system that says it does not know, or asks a clarifying question, or points to the closest relevant material without asserting something false. A system that always produces a confident answer, even to questions it has no basis for, will look impressive in a demo and will cause real damage in production. This is worth testing deliberately rather than discovering by accident three months in.
Score for citation, not just correctness
An answer without a traceable source is a claim you cannot check without redoing the work yourself, which defeats the purpose. For every answer, check whether the system points to the specific passage it drew from, and whether that passage actually supports the answer given. A vendor whose system cannot cite its source reliably is asking you to trust it blind, which is a different product than one built to be checked.
Run the same set against every vendor you are seriously considering
The value of a fixed evaluation set is comparative. Run the identical questions through each vendor's system, score them the same way, and you get something a benchmark cannot give you: a direct, apples to apples comparison on your own material. This also surfaces something benchmarks hide entirely, which is how a system behaves on your specific edge cases rather than on a generic average.
What to ask a vendor before you build anything
A vendor confident in their system should have no objection to a few direct questions.
- Can we bring our own twenty to fifty questions and test them before signing anything, using our own documents.
- What happens when the system does not have a good answer. Does it say so, or does it guess.
- Does every answer come with a citation back to the source document and passage.
- What was the benchmark run on. Same configuration a customer would actually get, or a more generous setup.
- How does the system behave when a document is updated or two documents conflict.
A vendor who resists letting you test on your own material with your own evaluation set is telling you something, whether or not they mean to. Confidence in a leaderboard score is not the same as confidence in the specific system that will sit in front of your staff or your clients. If you want a sense of what a running system built for this kind of question actually looks like once it is in production, our case study on answering policy questions from a firm's own documents describes one shape of the problem: staff asking plain language questions and getting answers cited back to source, which is exactly the property a good evaluation set is built to check for.
The point of the exercise
None of this replaces due diligence on price, support, or implementation timeline. It replaces one specific thing: the temptation to let a published number substitute for evidence about your own use case. A benchmark tells you how a model performed on someone else's test. A small evaluation set, built from your own questions and scored against answers you wrote down in advance, tells you how a system performs on yours. The second one is the only number that should change your decision.
