AI Security
Protect models, data, tools and infrastructure
Defending AI systems against adversaries: prompt injection, data leakage, model and supply-chain compromise, and abuse of the tools and identities an AI system can reach.
AI Security & Safety · Research and practice
Securing AI Agents Before They Become Enterprise Attack Surfaces.
Exploring the security, governance, and risk-management challenges of agentic AI, retrieval-augmented generation, and AI-enabled enterprise workflows. Status labels distinguish completed work from proposed experiments.
Research focus
Security, Governance and Risk Management of Agentic AI Systems in Financial Services.
Seven themes structure the work below — from prompt injection trust boundaries to AI governance in financial services.
01
Prompt Injection and Trust Boundaries
02
RAG Security and Data Protection
03
Agentic AI Threat Modelling
04
AI Identity and Least Privilege
05
AI Safety and Human Oversight
06
Adversarial Machine Learning
07
AI Governance in Financial Services
Three disciplines
Security, safety and governance are different questions.
They overlap in practice, but each has its own owner, controls and evidence.
AI Safety
Reliable, intended behavior with human oversight
Making sure systems behave as intended even without an attacker: predictable outputs, bounded autonomy, human review of consequential actions, and prevention of harm.
AI Governance
Accountability, lifecycle risk and regulatory controls
Assigning ownership and evidence across the AI lifecycle: risk assessment before adoption, policy and standards, vendor due diligence, monitoring and regulatory alignment.
Domains
AI security and AI safety, side by side.
Security asks how AI systems are attacked; safety asks how they fail on their own. Enterprise adoption needs answers to both.
AI Security
- Large Language Model Security
- Prompt Injection
- Indirect Prompt Injection
- Retrieval-Augmented Generation Security
- Agentic AI Security
- AI Agent Identity and Access Management
- AI Tool and API Security
- Adversarial Machine Learning
- AI Data Leakage
- AI Supply Chain Security
- AI Red Teaming
- AI Threat Modelling
- Multi-Agent System Security
AI Safety
- AI Alignment and Misalignment
- Safe AI Deployment
- Human Oversight
- Human-in-the-Loop
- Autonomous Agent Safety
- AI Evaluation and Assurance
- Robustness and Reliability
- Harmful AI Behaviour
- AI Risk Assessment
- Responsible AI
- Systemic AI Risks
AI governance & frameworks
Frameworks, with an honest depth label.
Some frameworks I have implemented professionally, some I have assessed against, and some I study as research. The label on each card says which.
Applied professionally
NIST AI Risk Management Framework
Govern–Map–Measure–Manage structure behind the enterprise AI security standard.
Applied professionally
ISO/IEC 42001
AI management-system requirements reflected in AI adoption controls.
Applied professionally
OWASP Top 10 for LLM Applications
Threat taxonomy baseline for LLM and RAG assessments.
Assessed
MITRE ATLAS
Adversary techniques against AI systems, used in threat modelling.
Research & study
EU AI Act
Regulatory obligations tracked for AI governance in financial services.
Research agenda
Four themes in depth.
Agentic AI Security
“How can an agent's actions be constrained when its instructions cannot be fully trusted?”
- Attack surfaces
- Tool calls, delegated credentials, planning loops, memory.
- Candidate defenses
- Tool gateway, per-tool authorization, human approval for high-impact actions.
- Proposed evaluation
- Unauthorized action rate under adversarial task suites.
Prompt Injection and Trust Boundaries
“Where must trust boundaries sit when model input mixes instructions and data?”
- Attack surfaces
- User input, retrieved documents, tool output, email and web content.
- Candidate defenses
- Provenance labels, privilege separation, output validation.
- Proposed evaluation
- Attack success rate across a structured payload taxonomy.
RAG Security and Data Protection
“Can retrieval respect the same access controls as the source systems?”
- Attack surfaces
- Vector stores, ingestion pipelines, document permissions.
- Candidate defenses
- Permission-aware retrieval, ingestion validation, tenant isolation.
- Proposed evaluation
- Cross-user retrieval and poisoning tests against baseline and hardened builds.
AI Governance and Risk Management
“What minimum governance lets an enterprise adopt AI without losing accountability?”
- Attack surfaces
- Shadow AI, third-party models, unowned agents.
- Candidate defenses
- AI inventory, risk classification, release gates, reassessment.
- Proposed evaluation
- Coverage of inventoried systems with owners and approved permissions.
Featured project · In progress
Autonomous AI agent security
How can an enterprise constrain an AI agent's ability to execute unauthorized actions when exposed to malicious or untrusted instructions?
The language model proposes; it never authorizes. Every tool call crosses a server-side policy boundary where a gateway checks the agent's identity, the tool's allowed parameters and the provenance of the instruction. High-impact actions wait for a human. Prompt filtering and model-level instructions help, but they are never the only barrier.
Agentic AI threat areas · Research topics, not proven findings
Indirect prompt injection
Retrieval poisoning
Memory poisoning
Goal hijacking
Excessive agency
Insecure tool / MCP integrations
Multi-agent privilege escalation
Data exfiltration
Unsafe autonomous execution
Threat taxonomy
| Threat | Attack path | Preventive control | Detective control | Residual risk |
|---|---|---|---|---|
| Direct prompt injection | User instructs the model to ignore policy | Server-side authorization independent of the model | Policy-violation logging | Low impact if tools are gated |
| Indirect prompt injection | Instructions hidden in retrieved or tool content | Provenance labels; untrusted content cannot trigger privileged tools | Tool calls traced to untrusted context | Medium — content steering remains possible |
| Retrieval poisoning | Malicious document inserted into the index | Ingestion validation and source allow-lists | Index change auditing | Medium |
| Excessive agency | Agent holds broader permissions than the task needs | Task-scoped, least-privilege agent identities | Permission usage review | Low with scoping |
| Unauthorized tool invocation | Model emits a call outside its allowed set | Tool gateway with schema and parameter validation | Denied-call alerts | Low |
| Sensitive data exposure | Model returns data the user may not see | Permission-aware retrieval; DLP on output | Output classification logging | Medium |
| Cross-user or cross-tenant leakage | Shared memory or index across tenants | Tenant-partitioned stores and caches | Isolation tests in release gates | Low |
| Unsafe delegation | Agent passes authority to a sub-agent or external service | No credential forwarding; scoped tokens | Delegation chain audit | Medium |
| Inadequate monitoring | Agent actions not recorded or reviewed | Mandatory audit events per tool call | Audit coverage metric | Low |
| Insecure output handling | Model output executed or rendered unsafely | Treat output as untrusted; encode and validate | Downstream error and WAF signals | Low |
Defense in depth
- Least-privilege agent identities
- Tool-specific authorization
- Server-side policy enforcement
- Input provenance and trust labels
- Schema and parameter validation
- Human approval for high-impact actions
- Separation of planning and execution
- Restricted data access
- Secrets isolation
- Tamper-resistant audit records where feasible
- Runtime monitoring and incident response
- Explicit failure handling
RAG security lab · Proposed
Baseline versus hardened retrieval.
Planned scenarios: malicious retrieved instructions, cross-user document access, retrieval poisoning, unauthorized retrieval, sensitive disclosure and unsafe downstream tool calls.
Related work · Independent project
Making prompt-injection resistance measurable
An end-to-end evaluation framework for testing RAG and LLM applications against prompt injection, with a structured payload taxonomy and an auditable scoring model.
Evaluation methodology
Planned metrics
A sample schema, not data. Each metric will be reported with its test conditions and limitations.
| Metric | Definition | Denominator | Value |
|---|---|---|---|
| Unauthorized action rate | Tool calls executed outside policy | All attempted tool calls in adversarial runs | Not yet measured |
| Attack success rate | Payloads achieving their stated goal | Payloads executed | Not yet measured |
| Sensitive data exposure rate | Responses containing data above the user's clearance | Responses in test set | Not yet measured |
| Legitimate task completion rate | Benign tasks completed correctly | Benign tasks attempted | Not yet measured |
| False refusal rate | Benign tasks wrongly refused | Benign tasks attempted | Not yet measured |
| Human approval coverage | High-impact actions that passed approval | High-impact actions attempted | Not yet measured |
| Audit event coverage | Tool calls with a complete audit record | All tool calls | Not yet measured |
AI governance
A lifecycle, not a policy document.
- 01AI system inventory
- 02Model and component provenance
- 03AI bill of materials
- 04Data classification
- 05Risk classification
- 06Agent identity and permissions
- 07Human accountability
- 08Evaluation and release gates
- 09Runtime monitoring
- 10Incident response
- 11Periodic reassessment
- 12Retirement and decommissioning
References