When capable agents fail, check the harness first: unclear tasks, missing context, broken environment, weak verification, or lost state.
Notes from Learn Harness Engineering - Lecture 01
As of late 2025, the strongest coding agents on SWE-bench Verified reach roughly 50-60% pass rates on carefully selected issues. Everyday work is messier: vague specs, implicit rules, missing tests, and state spread across sessions.
Same Opus 4.5 prompt: 20 minutes, $9, core 2D retro game editor features did not work.
Planner + generator + evaluator: 6 hours, $200, the game was fully playable. The model did not change.
These are harness failures before they are model failures.
"Add search" forces guesses about scope, pagination, highlighting, and behavior.
Rules in Slack or someone's head are invisible to the agent.
The agent burns context on dependency and version conflicts.
Without executable checks, the agent declares done when it feels done.
Long tasks fail when each new session rediscovers the project from scratch.
Use these labels to debug failures without jumping straight to "try a better model."
Map it to task spec, context, environment, verification, or state management.
Make completion executable, not subjective.
AGENTS.md.Document stack, architecture conventions, and verification commands in the repo root.
After a few rounds, the recurring bottleneck becomes obvious.
Completion criteria:
- New endpoint: GET /api/search?q=xxx
- Supports pagination, default 20 items
- Results include highlighted snippets
- All new code passes pytest
- Type checking passes: mypy --strictSWE-bench Verified pass rate for strongest agents in late 2025.
Approximate lines of agent-generated code in OpenAI's five-month experiment.
PRs opened by three engineers using Codex.
Better context efficiency after adding repo rules and verification in the FastAPI example.
Three engineers wrote no code themselves; Codex generated application logic, infrastructure, tooling, and documentation. The pattern was to break large goals into design, code, review, and test blocks, then strengthen the harness whenever something failed.
One-sentence prompting made the agent spend 40% of context exploring, use old SQLAlchemy syntax, and ship runtime errors. Adding AGENTS.md, verification commands, and ADRs made the same model succeed in all three independent runs.