Case study · Architect · 35 min

Prompt Injection Defense for Tool-Using Agents

Hardening an agent against direct and indirect prompt injection when it reads untrusted documents and calls powerful tools.

safetysecurityagents

Interview opener

Harden an app that puts untrusted documents into context.

1.Indirect injection via a retrieved PDF — mitigations?
2.Can the model escalate tool privileges via injection?
3.How do you test defenses continuously?

Strong signals

Instruction/data separation
Allowlisted tools + output validators
Red-team suite in CI

Hit 0/3

Weak signals (avoid)

Just a better system prompt
Trusting the model to 'not obey' malicious text

Requirements

Threats: jailbreaks, malicious PDF instructions, tool exfiltration
Assets: email send, CRM write, code execution sandbox
Goal: exploit success < 1% on red-team suite; zero high-impact tool abuse in prod
UX: minimize false refusals on benign support workflows

Back-of-envelope Estimation

Metric	Calculation	Result
Red-team cases	400	400
CI cost/run	400 × 3K × $2/1M	~$2.40
Prod QPS	20	20
Guard model cost	20 × 500 tok × 86400 × $0.2/1M	~$173/day
High-impact tools/day	~500	extra auth friction OK

API Design

Component	Method	Description
tool_gateway	RPC	Schema + policy engine before side effects
POST /v1/guard/classify	POST	Injection/jailbreak risk score
POST /v1/tools/{id}/confirm	POST	Human approval for high-risk tools

Data Model

Entity	Key fields	Notes
policies	tool_id, risk, requires_human	Versioned
untrusted_spans	doc_id, offset, label	Retrieved content tags
security_events	ts, rule, action	SIEM

High-level Design

Defense in depth

User prompt ──► allow/deny filters
Untrusted docs ─► sanitize + wrap in <untrusted> delimiters
Planner LLM ───► can propose tools only from allow-list
Tool gateway ──► validate args; strip URLs; enforce authz
High-risk ─────► human confirm OR dual control
Output ────────► DLP scan before send
All layers emit security_events

Deep Dive: Indirect Injection

Treat retrieved text as data, never as instructions — dual-channel prompts
Content detectors for "ignore previous" patterns (imperfect; still useful)
Disable tools when context includes untrusted HTML/email until cleared
Capability tokens: tools receive short-lived scoped credentials, not master keys

No silver bullet

Assume the model can be socially engineered. Architectural isolation beats prompt-only defenses.

Bottlenecks at Scale

Guard model latency — run parallel with retrieval; cache clean docs
False positives frustrate users — tiered risk, not blanket bans
Red-team drift — refresh suite monthly with new exploits

Failure Modes & Monitoring

SLO	Target	Alert
Red-team pass	≥ 99%	Block release
High-risk tool w/o confirm	0	Page
False refusal rate	< 3%	Warn UX
DLP hits	tracked	Weekly review

AI & FDE Interview Room