The pitch you will hear
At some point in a vendor demo, someone will tell you their model has a context window of a million tokens, and that this means you can just dump your whole document set in and ask it anything. It sounds like it solves the hard part of building an AI system: the part where you have to decide what the model actually gets to see. If the window is big enough, why decide at all?
Because a bigger window changes what the model can technically fit, not what it can reliably use. Those are different problems, and the second one is the one that determines whether your system gives correct answers in production.
What actually happens when you stuff the context
Put a hundred pages of policy documents, contracts, or process notes into a prompt and ask a specific question, and you are not asking the model to read carefully. You are asking it to search inside its own attention over a much larger space, while every irrelevant paragraph in that space competes for the same weight as the one paragraph that actually answers the question.
Three things tend to go wrong, in order of how often we see them:
- Position bias. Models are measurably better at using information near the start or end of a long context than information buried in the middle. If the answer sits on page 40 of an 80 page dump, the odds it gets used correctly drop, even though it is technically "in context."
- Distraction from near-miss content. A document that is topically close but factually wrong for this question (an old version of a policy, a similar but different clause) sits right next to the correct one. The model has to actively resist the wrong answer. It does not always win that fight.
- Cost and latency, not accuracy, but still a real cost. Every token you feed in gets paid for and adds time. A system that reads forty pages to answer a question that needed one paragraph is slower and more expensive for no accuracy gain, and often a small accuracy loss.
None of this is a claim about one model being worse than another. It is a property of how these systems handle long, mostly irrelevant input, and it holds up across the general class of frontier models, not just weaker ones.
Why the vendor talking point survives anyway
A bigger context window is a real capability and it is easy to market. It shows up as a single number on a spec sheet, it sounds like more is better, and it lets a vendor avoid saying the harder thing, which is that solving retrieval well takes engineering work specific to your documents and your questions. "Just paste it all in" is a much shorter sales conversation than "we need to understand your document structure, build a retrieval step, and test it against real questions from your team."
The window size answers a question nobody was asking: how much text can the model technically accept. The question that actually matters is: out of everything you have, what is the smallest, most relevant set of material that answers this specific question. Those are not the same question, and only the second one predicts whether the system is right.
What relevance-first retrieval looks like instead
The alternative is not a clever prompt trick. It is a retrieval step built before the model ever sees the question: something that ranks and selects the handful of passages actually relevant to the query, and passes only those in, along with enough structure that the model can cite where each fact came from.
Done properly, this involves work that a large context window does not remove:
- Chunking documents sensibly. A policy document split at arbitrary character counts produces chunks that cut a clause in half. Split by section or clause instead, and retrieval gets meaningfully better before you have touched the model at all.
- Ranking by relevance, not recency or keyword match alone. A search that returns "everything containing the word approval" is not doing the job. It needs to rank by how well a passage actually answers the question asked.
- Deciding how much is enough. Sometimes the right answer needs three passages. Sometimes it needs one table and a definition from a different document. The number is not fixed, and a system that always returns the same amount regardless of the question is guessing.
- Testing against real questions, not synthetic ones. The only way to know whether retrieval is working is to run it against the actual questions your staff ask, and check the citations against the source. This is the same discipline covered in how to tell if an AI vendor's benchmark has anything to do with your business: a spec sheet number tells you nothing about your documents.
This is more work than pointing a large context window at a folder. It is also the part that determines whether the system answers correctly or confidently wrong, which is a worse failure mode than a slow one.
Where a large window actually helps
It would be wrong to say context window size never matters. It matters when the task genuinely requires the model to reason across a large amount of connected material at once: reviewing a long contract for internal consistency, for instance, where the relevant content really is most of the document and there is no smaller relevant subset to retrieve. In that case, a larger window is doing real work, because the task is "read all of this," not "find the part that answers this."
The distinction is whether the task is a retrieval problem or a comprehension problem. Retrieval problems (answer this question from our documents) need relevance engineering regardless of window size. Comprehension problems (assess this one document as a whole) can genuinely benefit from more room. Most of the recurring questions a business system has to answer, "what's the approval threshold," "who signs off on this," "what does the contract say about renewal," are retrieval problems wearing a comprehension costume. Buying a bigger window does not change which kind of problem you have.
The question to ask a vendor instead
If a vendor's answer to "how do you make sure the system gives correct answers" is a token count, that is a sign the retrieval work has not been done, or that it has been quietly replaced with volume. The better question is narrower: show me, on our own documents, what the system retrieves for a specific question, and let me check that against the source myself. A system that can show its retrieved passages and their source is one you can audit. A system that just says "I read everything" is one you have to trust blind.
Relevance is the thing that gets tested and tuned. Volume is the thing that gets advertised. They are not the same axis, and only one of them moves accuracy.
