The question nobody asks until after the demo
Every AI system that reads across departments raises the same question, usually about three weeks after it works. Someone in finance asks the assistant a question, and the answer includes a detail from an HR file it should never have touched. Not because anyone did anything malicious. Because nobody decided, in a way the system could actually enforce, what it was allowed to read on that person's behalf.
This is not a model problem. The model will happily retrieve and summarize whatever it is given access to. The problem is that most teams building their first AI system do not have a permission model at all. They have a login. The system authenticates the user, then goes and reads from wherever its retrieval layer points, which is usually "everything we indexed." Nobody wired the system to know that the person asking is in accounts payable and should not see legal's contract drafts, or that a contractor's questions should not surface the same answers as a full time employee's.
None of this is exotic. It is the same access control problem every company already solved, or half solved, for its file shares and its CRM. The difference is that an AI agent is much better at finding the thing you forgot to lock down, and much better at explaining it back to someone in plain language. A permission gap that sat quietly in a shared drive for years becomes visible the first time someone asks the assistant a question that touches it.
What a permission boundary for an agent actually is
Strip away the vendor language and a permission model for an AI system has three pieces, and they are the same three pieces you would build for any system that reads sensitive data on someone's behalf.
- Identity. The system needs to know, for every request, who is asking. Not "a valid session token exists" but which person, in which role, in which department. This usually means the agent inherits the identity provider you already have (Azure AD, Okta, whatever issues your logins) rather than maintaining its own user table.
- Scope. For that identity, which documents, tables, or tools is it allowed to touch. This is the part people assume is free because "the model only sees what we gave it in the prompt." That is true and also the whole problem: someone has to decide, ahead of time, what gets fed into that prompt for this particular user, on this particular request.
- Enforcement point. Where the check actually happens. This is the piece that gets skipped most often, because it is the least visible one during a demo.
The mistake we see most is treating the third piece as covered by the first two. It is not. Knowing who someone is and what they are allowed to see does not automatically stop the retrieval layer from pulling in a document outside that scope. That check has to be enforced at query time, against the actual data store, not just described in a policy document or hoped for in a system prompt.
Where the checks live, and why the choice matters
There are three places you can put an access check, and each has a different failure mode.
In the prompt. You tell the model, in its instructions, what the user is and is not allowed to see, and trust it to comply. This is the cheapest option to build and the worst one to rely on for anything sensitive. A model following an instruction is not the same as a system enforcing a boundary. Prompt instructions can be overridden by a cleverly phrased question, by a long conversation that drifts the model off its original framing, or simply by the model making a retrieval call before it has fully "read" the constraint. Use this layer for tone and behavior. Do not use it as your only control for what the system can see.
In the retrieval layer. The access check happens before anything reaches the model, as a filter on the query against your document store or database. If a user's role does not include access to a folder, the retrieval call for that folder never runs for them, regardless of what they ask. This is the layer that actually matters, because it fails closed: if the filter is missing or misconfigured, the system returns nothing rather than returning the wrong thing (assuming you built it that way, which is a design choice, not a default).
In the underlying data store. The check is enforced by the database or document system itself, using the same row level or folder level permissions your existing applications already respect. This is the strongest option when it is available, because it means the AI system inherits controls you have already tested against your existing tools, rather than reimplementing them in a new codebase that has not been through the same scrutiny.
In practice, a defensible system uses the second and third together: the data store enforces its own permissions, and the retrieval layer respects them rather than using a service account that bypasses them to make indexing easier. The prompt layer is a courtesy on top, not the boundary itself.
The mistake that actually causes the incident
The failure we have described so far is not usually a design failure. It is an indexing shortcut. Someone building the retrieval system needs it to work quickly, so they give the indexing job a service account with broad read access, on the reasoning that the permission check will happen "later, at query time." Then the query time check gets built for the common cases, the ones that show up in testing, and the edge cases quietly fall through: a document that was moved between folders after indexing, a user whose role changed last month but whose old permissions are still cached, a group membership that synced incorrectly from the identity provider.
None of these show up in a demo, because a demo runs the same three questions from the same test account every time. They show up in production, three months in, when someone asks a question nobody scripted.
This is why the honest answer to "what does it cost to keep an agent from touching the wrong data" is not a licensing fee or a security tool. It is engineering time spent on the boring parts: mapping every source system's existing permission model before you touch retrieval, deciding what happens when a permission check fails (deny by default, and say so out loud, rather than letting it silently degrade to "return everything"), and testing with accounts that represent your actual range of roles, not just an admin account that can see everything and therefore never notices what is missing.
What this costs in practice
Expect the permission work to be a meaningful fraction of the build, not an afterthought bolted on before launch. On a system that reads across more than one department's data, mapping who should see what, and wiring the enforcement so it actually holds, is often comparable in effort to the retrieval and generation work everyone assumes is the hard part. The model call is the easy part. Knowing what the model is allowed to be fed, for this user, on this request, is the part that takes the time.
This is also why a system built to answer questions from one department's documents, of the kind described in a case study like ai-policy-copilot, is a meaningfully different engineering problem from one that reads across departments. A single-department assistant has a much smaller scope to get wrong. The moment a system is asked to serve ops, risk, legal, and customer support from the same interface, the permission model stops being a footnote and becomes the main design question.
If a vendor or a build team cannot describe, specifically, where the enforcement point is and what happens when it fails, that is the question to ask before anything else. Not "is the model secure." The model has no idea what your organization chart looks like. The system built around it either enforces that chart or it does not.
