Lecture 09

Preventing agents from declaring victory too early

The agent's confidence is not completion evidence. A harness has to make done an external, layered runtime judgment.

Notes from Learn Harness Engineering - Lecture 09

termination_checkworker cannot self-pass

L1Static checkslint, typecheck, syntax, migration parsepass

L2Runtime checkstests, app startup, health probespass

L3User flowcomplete scenario with real side effectsblocked

The slippery slope starts when "looks correct" becomes "done."

The lecture's password-reset example looks productive: schema changes, endpoint, email template, and passing unit tests. The hidden reality is worse: email config is missing, migration can fail halfway, and the complete reset flow never ran.

Code looks fine

Syntax is valid and local logic seems plausible.

Partial tests pass

Unit tests are green, often with mocks and isolated state.

Runtime is skipped

Startup, config, migrations, and real services stay unobserved.

Agent says done

Confidence replaces evidence, so defects move to the next session.

Unit tests only

Unit tests deliberately isolate dependencies. That design makes them poor at detecting cross-component failures.

Renderer sends relative paths; preload expects absolute paths.
ORM cache retains old schema state after a migration.
Mocked config hides missing email service settings.

vs.

Execution evidence

Runtime verification observes the system after transformations from task specification to code to actual behavior.

Feature complete = end-to-end verification passed, not "code is written".

Three layers turn completion into a gate.

The agent can request a verdict, but the harness owns the verdict. Each layer increases cost and information value, so failures should stop the pipeline early.

Layer 1

Syntax and static analysis

Lint, typecheck, parse, and other low-cost checks. Necessary, but never sufficient.

Layer 2

Runtime behavior verification

Test execution, application startup, health checks, and critical-path validation.

Layer 3

System-level confirmation

End-to-end testing, integration validation, and user scenario simulation.

PASS ALL THREE LAYERS BEFORE "DONE"

Agent says donelint / typechecktests / startupcomplete user flowdone

Core concepts

The key move is to separate subjective confidence from executable termination criteria.

Premature completionThe agent asserts completion while correctness specifications remain unmet.

Calibration biasModel-reported confidence exceeds actual quality; Guo et al. showed modern neural networks are systematically overconfident.

Termination criteriaExecutable judgment conditions defined by the harness, not by the worker agent.

Verification-validation gateVerification checks specified behavior; validation checks system-level user requirements.

Runtime signalsLogs, process state, health checks, side effects, and cleanup status from actual execution.

Completion priorityFunctional correctness first, performance second, style last. No refactor before core behavior passes.

Refactoring before verification corrupts the verdict.

"While we're at it" work changes the boundary between verified and unverified code. The harness should block cleanup, optimization, and style passes until the core feature has passed its functional checks.

Interface mismatch

Mocks let each side pass alone while the real boundary fails.

State propagation

Fresh test fixtures miss stale caches and migration side effects.

Environment dependency

Mock services hide configuration, latency, and availability defects.

Separate the worker from the checker.

Anthropic's 2026 research found agents give overly positive assessments of their own work. The fix is not asking the same agent to be more objective; it is using an independent checker with real runtime tests.

Plannerexpands requirements

Generatorimplements feature by feature

Evaluatorclick-tests and rejects weak work

## Definition of Done
- Feature complete = end-to-end verification passed
- Required verification levels:
  1. Unit tests pass
  2. Integration tests pass
  3. End-to-end flow verification passes
- Do not proceed to level 2 if level 1 fails
- Do not proceed to level 3 if level 2 fails

Harness comparison

20 min

single agent

bare run

single agent cost

core game broken

6 hrs

three agents

planner + generator + evaluator

$200

three-agent cost

game fully playable

How to prevent early victory

Externalize termination judgment.

The harness independently runs validation and uses runtime signals, not agent confidence.

Build the three-layer gate.

Static checks, runtime checks, then complete user flow. Stop at the first failed layer.

Write repairable errors.

Tell the agent what failed and where to look, for example missing email config or template path.

Capture side effects.

Observe startup, feature paths, database writes, file operations, and temporary-resource cleanup.

Bad: Test failed

Better: POST /api/reset-password returned 500.
Check that email service config exists in env vars.
Template should be at templates/reset-email.html.

Password reset case

The harness intervention catches the defects inside the session instead of paying post-hoc repair cost later.

Premature hand-in

Schema edited, endpoint written, template added, unit tests green, completion declared.

Actual omissions

No end-to-end reset link flow, failed partial migration, missing email service config.

Harness checks

Start the app, exercise the reset flow, verify database consistency and side effects.

Result

All defects discovered within the session, saving 5-10x the cost of post-hoc fixes.

TAKEAWAY