Every team wants an LLM on its real work: the contracts, the support tickets, the patient notes (protected health information, or PHI), the source code. That work is also full of data that must never be pasted into a public model. The usual instinct is to ban it, and the usual result is shadow AI: people route around the block through personal accounts and browser extensions, and the data leaves anyway, now completely unlogged. Samsung learned this in 2023, with three separate leaks of proprietary code and internal notes within weeks of allowing ChatGPT, followed by a company-wide ban. “The vendor says it will not train on our data” is a contract, not a control.
In 2026 the problem got sharper. Exfiltration is no longer only an accident; it is an attack. A hidden instruction buried in an email, a document, or a chat message can make an AI assistant leak whatever it can see, with no malware and no stolen credentials, an attack known as indirect prompt injection. The attack surface is the text the model reads. So the brief for this reference build was blunt: let people run sensitive data through an LLM, and make control architectural rather than contractual.
The idea
One principle drives the whole design. The model should reason over data it is allowed to see, in a form that cannot hurt you if it leaks, inside a boundary you control, with a record of everything. Turned into engineering, that becomes a set of layers, each doing one job, with a single place to enforce them.
The core move is a sandwich: sensitive values never reach the model in the clear. They are detected and swapped for stable placeholders on the way in, and the real values are restored on the way out. The model reasons over “PERSON_1” and “ACCOUNT_2”; you keep the mapping.
How it works, layer by layer
The gateway. Every call goes through one proxy, so there is a single place to enforce policy. It authenticates the caller and attaches an identity, detects sensitive data with a stack of pattern matching and named-entity recognition (software that spots names, places, and account numbers in free text), and then either masks it (blanks the value for good, which anonymizes the data and can take it out of the scope of GDPR, Europe’s data-protection law) or tokenizes it (swaps it for a reversible placeholder, so the model can tell that “PERSON_1” in line one is the same person in line ten). Only sanitized text crosses to a public model. When the model genuinely needs the real value, the gateway routes the request to a private deployment instead. Every call is logged.
Permission-aware retrieval. Most enterprise assistants use retrieval, pulling in documents as context, and this is where access control quietly breaks. A naive vector store (the search index the assistant pulls from) ignores the folder permissions of the source systems, so the model can surface a document the user was never allowed to open. The fix is to enforce access at the retrieval layer, filtering to what the user may see before anything reaches the model, and to redact sensitive data at ingest so the index itself stays clean.
Guardrails. Inputs are scanned for injection attempts before they reach the model; outputs are scanned for leaks before they reach the user. One failure worth catching specifically: a model that invents a placeholder the system never issued. The gateway replaces it rather than risk leaking data across requests.
Deployment by sensitivity. Matching the control to the data is most of the design.
Governance underneath. None of this counts unless you can prove it. Every detection, tokenization, rehydration, and retrieval is written to a tamper-evident audit log where entries can only be added, never changed or deleted (the write-once pattern known as WORM), keyed with lineage, a record of where each piece of data came from and where it went, so that a deletion request can actually be honored across caches and downstream copies. Agents and services get least-privilege access, only the data and tools a task needs. And anything irreversible, moving money or deleting records, waits for a human.
Compliance, mapped to the build
The layers are not just good hygiene; they line up with what regulators now require. Under GDPR the model provider is a data processor (a vendor that handles your data on your instructions), so you need a processing agreement and, for EU data, residency or transfer safeguards: truly anonymized data leaves GDPR scope, while tokenized data stays in it. HIPAA, the US medical-privacy law, needs a signed agreement before any provider touches health data, or the data stays de-identified (stripped of anything that points to a person) or inside your perimeter. The EU AI Act’s high-risk rules, landing in 2026, map directly to the data governance, tamper-evident logging, and human oversight this build provides, with penalties large enough to make it a board-level question.
Outcome
The guarantees are architectural. Sensitive values never leave the perimeter in the clear. Retrieval respects the permissions that already exist. Injection and leak attempts are caught and logged rather than quietly succeeding. The most sensitive data runs where it legally must. And every step is on the record, so compliance is something you can show an auditor, not something you promise.
None of this costs you the LLM: the model still does the useful work, over data it is allowed to see. Which model, how much to redact versus keep private, and which regulations bite depend on your data and your stack, and that mapping is work we can do with you.