Lab · Advanced · 90 min
Golden-Task Eval for a Skill Change
Build a small harness that runs golden tasks against a skill several times, scores activation and outcome, and gates skill changes in CI.
Claude Code · Cursor · OpenAI Codex · LangChain / LangGraph
Show all 3 stepsHide list
Step 1 of 3
Lab brief
Goal
A repeatable eval showing pass rate, activation rate, and cost for the old vs new version of a skill.
Failure mode
Tweaking a skill because one run looked better, shipping it, and silently breaking the three other cases it used to handle.
Staff move
Treat agent configuration as code under test: fixed task set, multiple trials, checked outcomes, and a regression threshold in CI.
Step 2 of 3
1. Define golden tasks
| Task | Check |
|---|---|
| Release notes for a fixture repo with 12 commits | Output has feat/fix sections; no chore commits |
| Unrelated prompt: write a README intro | Skill did not activate |
| Repo with no tags | Skill handles gracefully (asks or uses first commit) |
| Breaking change commit | Appears under a Breaking section |
Step 3 of 3
2. Build the harness
- Run each task N=5 times headless (claude -p, codex exec, or a LangGraph runner)
- Parse transcripts for skill activation; run checks on outputs
- Record pass rate, activation precision/recall, tokens, and latency
- Compare baseline vs candidate; fail CI if pass rate drops beyond a threshold
Stretch
Add an LLM-as-judge for tone with a rubric, and calibrate it against 20 human-labeled outputs before trusting it.