Most teams test prompt injection the way they once tested XSS: someone pastes a clever string into a chat box, it works, everybody panics, and a filter gets added. That is a demonstration, not a test. A prompt injection testing framework is the difference between knowing your assistant broke once and knowing, release over release, how often it breaks, under which attack classes, and whether the number is going down.
What follows is the structure I use when building this capability inside banks and other regulated enterprises. It assumes an LLM application with retrieval, tools, and real downstream actions — not a toy chatbot.
1. Start from a threat model, not an attack list
Prompt injection is not one vulnerability. It is a delivery mechanism. Before collecting payloads, write down where untrusted text enters the model's context and what the model can do once influenced. In practice there are four injection surfaces: direct user input; retrieved documents in a RAG index; tool and API responses fed back into context; and system-adjacent content such as file names, email subjects, ticket bodies or web pages the agent fetches. The last three are indirect prompt injection, and they are where real incidents live, because nobody is watching the SharePoint sync the way they watch the chat box.
Pair each surface with an impact class: data exfiltration (the model reveals content the requester is not entitled to), unauthorised action (the model calls a tool that moves money, grants access or sends mail), integrity (the model returns confidently wrong answers that a human acts upon), and policy bypass (the model produces output the organisation has committed not to produce). A test that cannot be mapped to one of these does not belong in the suite.
2. Build an attack corpus you own
Public jailbreak lists age badly and were mostly written against consumer chat products. Your corpus should have three tiers. Tier one is a generic baseline — instruction override, role reassignment, delimiter and encoding tricks, multilingual and homoglyph variants, markdown and HTML comment smuggling, and tool-call coercion. Tier two is application-specific: payloads written against your actual system prompt, your actual tool names, your actual document conventions. Tier three is scenario chains — multi-turn attacks where the first message is benign and the payload is planted in a retrieved document opened three turns later.
Every payload is a record, not a string: identifier, surface, attack class, target impact, the exact injected text, the expected safe behaviour, and a deterministic success condition. The success condition matters more than the payload. 'The model said something weird' is unscoreable. 'The response contains the canary token planted in document 7' is a test.
3. Canary tokens make exfiltration measurable
Seed the evaluation environment with unique, unguessable canary strings at each sensitivity level — one inside a restricted document, one in a tool response, one in the system prompt. Any appearance of a canary in an output the requester was not entitled to is an unambiguous failure, detectable by string match, with no judge model and no human in the loop. This single technique converts the majority of exfiltration testing from subjective review to a regression assertion that runs in CI.
4. Harness design: the application, not the model
Test through the same entry point your users hit. If the production path includes a retrieval layer, a re-ranker, an input classifier, a system prompt, tool schemas and an output filter, all of them are part of the system under test. A harness that calls the model API directly measures the model vendor's alignment, which you do not control and cannot fix. The harness should support seeded documents, mocked tools that record invocations rather than executing them, fixed sampling parameters where possible, and a fixed number of repetitions per payload — non-determinism means a single pass tells you nothing.
5. Scoring: attack success rate, per class
The headline metric is attack success rate (ASR): successful attacks divided by attempts, reported per attack class and per injection surface, not as one number. A system with 2% overall ASR that fails 40% of indirect injections through retrieved documents is not a 2% system. Report alongside it the false refusal rate on a benign control set, because every injection defence can be trivially improved by refusing more, and a helpful-but-hardened assistant is the actual goal. Track both across releases; a defence that cuts ASR by three points while doubling refusals on legitimate work will be switched off by the business within a month.
For impact classes that resist string matching — subtly biased or policy-violating prose — use a judge model with a written rubric, then human-review a stratified sample of its verdicts each cycle to keep the judge honest. Never let an unaudited judge model be the only thing standing between a regression and production.
6. Release gates and runtime feedback
Wire the suite into the deployment pipeline with explicit thresholds: zero tolerance for canary exfiltration and unauthorised tool invocation, a numeric ceiling for lower-impact classes, and a mandatory re-run whenever the system prompt, retrieval configuration, tool schema or model version changes. Model version changes are the one people forget, and they are the one most likely to silently move the numbers. Close the loop by feeding production refusals, anomalous tool calls and user reports back into the corpus so the suite grows from real traffic rather than imagination.
7. Mapping to governance
In a regulated environment this work has to be legible to people who will never read the harness. Map the framework onto NIST AI RMF (Measure and Manage functions), ISO/IEC 42001 controls, and the OWASP Top 10 for LLM Applications — LLM01 in particular — so that each test class traces to a stated control objective. The output risk committees actually want is one page: attack success rate by class, trend against previous release, open exceptions, and the date of the last full run.
What good looks like
A mature programme can answer four questions on demand: which untrusted content can reach the model, what the model can do once influenced, what percentage of known attacks currently succeed, and when that was last measured. Nothing about that requires exotic tooling. It requires treating prompt injection testing as a durable engineering asset with owners, thresholds and a changelog — rather than as an exercise someone repeats each time a journalist writes about jailbreaks.