Chapter 10 · 6 lessons
Hooks, Security, Evals & Shipping
Deterministic guardrails with hooks, permission and sandbox design, reviewing agent output, evals for agent setups, CI and cloud agents, and rolling this out to a team.
Cursor · Claude Code · OpenAI Codex · GitHub Copilot
Show all 6 lessonsHide list
Lesson 1 of 6
Hooks: Prompts Suggest, Hooks Decide
Hooks are scripts the harness runs on lifecycle events — before a tool call, after a file edit, before a shell command, when the agent tries to stop. They are deterministic: a hook that blocks rm -rf blocks it every time, regardless of what the model 'intended'.
| Tool | Config | Example events |
|---|---|---|
| Claude Code | .claude/settings.json or plugin hooks/hooks.json | PreToolUse, PostToolUse, UserPromptSubmit, Stop, SubagentStop, SessionStart |
| Cursor | .cursor/hooks.json | preToolUse, beforeShellExecution, beforeMCPExecution, afterFileEdit, stop, subagentStop |
| Copilot | Hooks in CLI and cloud agent | Pre/post tool and session events |
- Block: dangerous commands, edits to protected paths, secrets in diffs
- Enforce: run formatter after edits; refuse to stop until tests pass
- Observe: log every tool call for audit
- Keep hooks fast and fail-safe; in Cursor, failClosed makes a crashing hook block instead of allow
Lesson 2 of 6
Permissions and Sandboxes
| Control | Codex | Claude Code |
|---|---|---|
| Filesystem / network scope | sandbox_mode: read-only | workspace-write | danger-full-access | Sandboxing + permission rules |
| When to ask | approval_policy: untrusted | on-request | never | Permission modes + allow / ask / deny rules |
| Hard blocks | Sandbox + config | Deny rules and PreToolUse hooks (apply even in bypass mode) |
- Default to workspace-scoped writes and no network; widen per task
- Allowlist safe commands (test, lint, build) to reduce approval fatigue
- Never run full-access modes on a machine with production credentials
- Cloud agents run in isolated VMs — still scope their tokens and repo access
Lesson 3 of 6
Reviewing Agent Output
- Read the diff, not the summary — summaries routinely overstate what was done
- Run the tests yourself (or in CI); check that new tests actually fail without the change
- Use a clean-context reviewer (subagent, Bugbot, Copilot code review) to hunt for bugs
- Look for the classic agent tells: deleted or skipped tests, broad try/catch, invented APIs, unused code, config weakened to make things pass
- Keep PRs small — ask the agent to split large changes
Test tampering
When an agent cannot make a test pass, it may edit the test. Protect test files with a hook or require explicit approval for test changes on bug-fix tasks.
Lesson 4 of 6
Evals for Your Agent Setup
Your AGENTS.md, skills, and MCP configuration are prompts that change behavior. Changing them without an eval is the same as changing code without tests.
- Golden tasks: a small set of real tasks with checkable outcomes (tests pass, file created, convention followed)
- Run each task several times — agents are nondeterministic; compare pass rates, not single runs
- Track cost and latency alongside quality
- Trace runs (LangSmith, harness logs) so you can see where a regression came from
Refresher
Offline vs online evals and LLM-as-judge are covered in the AI & FDE track's Production AI chapter.
Lesson 5 of 6
CI, Cloud Agents, and Team Rollout
- Headless runs in CI: Claude Code and Codex can run non-interactively for triage, fixes, and reviews
- Cloud agents from tickets: Copilot cloud agent on assigned issues, Cursor and Codex cloud tasks — write issues like specs
- Standardize with a team plugin: shared skills, hooks, and MCP config installed in one step
- Govern: approved marketplaces, pinned plugin versions, org-level deny rules
- Measure: cycle time, review rework, escaped defects — not lines of code generated
Staff-level framing
The goal is not maximum autonomy. It is maximum throughput at a quality bar the team trusts — which means investing in context, guardrails, and review as much as in the agent itself.
Lesson 6 of 6
Worked Example: The Commit That Skipped Tests
The memory file says 'always run tests before committing.' Late in a long session the agent commits anyway.
AGENTS.md: "Always run tests before committing." agent (turn 63): git commit -m "fix: handle empty cart" CI: 4 failing tests
.claude/settings.json → PreToolUse hook on Bash .cursor/hooks.json → beforeShellExecution hook guard: if command matches "git commit" and .agent/dirty exists → exit 2 "Tests have not passed since your last edit. Run pnpm test:unit." agent: Running pnpm test:unit... 2 failing → fixes → 212 passed → commit
- The rule moved from a suggestion into a deterministic hook
- The block message told the agent exactly what to do next
- The same guard works in both tools with thin wrappers
Takeaway
Prompts suggest; hooks decide. Put your must-always rules where they cannot be forgotten.