Lecture 11

Making the agent's runtime observable

Reliability is an evidence problem: the harness must expose what the system did and why the work should be accepted.

Notes from Learn Harness Engineering - Lecture 11

task_traceruntime + process

Runtime

logs, traces, process events, health checks

Process

plans, sprint contracts, rubrics, acceptance criteria

ContractGeneratorSignalsReviewVerdict

Without observability, every retry is a guess.

The agent can touch many files, run for 20 minutes, and still be unable to explain why tests fail or which critical paths changed. The harness, not the agent's memory, must preserve the evidence.

Looks correct

Code review shows what was written, but runtime traces show what actually ran.

Mystical evaluation

Without rubrics, acceptance depends on implicit assumptions and inconsistent judgment.

Blind retries

The agent fixes nearby code instead of the root cause because failure context is missing.

Handoff cliff

Missing state forces the next session to rediagnose, wasting 30-50% of session time.

Same dark-mode task, different observability.

Vague loop

Planner gives a fuzzy description. Generator guesses. Evaluator rejects with "it doesn't feel right." The cycle repeats 3-4 times.

About 45 minutes, barely acceptable output.

vs.

Observable loop

Sprint contract defines scope, standards, and exclusions. Runtime records style application. Rubric cites concrete evidence.

About 15 minutes, high-quality output: 3x efficiency.

Observability has two layers.

Adding logs is not enough. Runtime signals explain behavior; process artifacts explain intent and acceptance.

Runtime observability

What did the system do?

Logs, traces, process states, health checks, data flow, resources, errors, and side effects.

Process observability

Why should this pass?

Plans, sprint contracts, scoring rubrics, exclusions, acceptance criteria, and evaluator decisions.

CONTRACT + SIGNALS -> REVIEW -> SPECIFIC VERDICT -> NEXT FIX

Core concepts

A good harness records enough decision context to replay what happened and why.

Task traceComplete decision-path record from task start to completion, like request tracing for agent work.

Sprint contractShort-term agreement defining scope, verification standards, and exclusions before coding starts.

Evaluator rubricStructured scoring that turns "is it good?" into evidence-based dimensions.

Runtime signalsObjective system behavior: logs, traces, health, data flow, resources, exceptions.

Layered observabilityRuntime and process evidence designed together so each reinforces the other.

The agent cannot solve this by printing logs.

Unknown unknowns

Agents only log what they think matters. That is usually not enough.

Inconsistent formats

Different sessions produce different logs, making systematic analysis brittle.

Process gaps

Sprint contracts and rubrics are structured harness artifacts, not print statements.

How to build it

Collect runtime signals automatically.

Capture lifecycle, critical paths, data flow, resources, exceptions, and full error context.

Negotiate sprint contracts.

Define scope, verification standards, and exclusions before the generator writes code.

Use evaluator rubrics.

Score dimensions such as correctness, architecture, test coverage, visual design, and code quality.

Standardize traces.

Use OpenTelemetry: session trace, task span, verification sub-spans, and standard attributes.

# Sprint Contract: Dark Mode Support
Scope:
- Modify theme toggle component
- Update global CSS variables
- Add dark mode tests

Verification:
- Visual regression tests pass
- Main E2E flow passes
- No flash of unstyled content

Exclusions:
- No print styles
- No third-party dark mode

Rubrics make evaluation reproducible.

The evaluator should cite specific evidence, not vibes. For example: "Button color contrast is insufficient: WCAG AA requires 4.5:1, measured 2.1:1."

Dimension	A	B	C	D
Code correctness	All tests pass	Main flow passes	Partial pass	Build fails
Architecture	Fully compliant	Minor deviations	Obvious deviations	Serious violations
Test coverage	Main + edge cases	Main flow only	Only skeleton	No tests

Observable harness data

45m

vague loop

dark-mode scenario

15m

observable loop

3x faster

30-50%

handoff waste

redundant diagnosis

$124.70

Anthropic run

3 hr 50 min

Anthropic's three-agent experiment

The DAW build made observability explicit: planner expands product context, generator works sprint by sprint, evaluator uses Playwright and scores hard thresholds.

Phase	Duration	Cost
Planner	4.7 min	$0.46
Build round 1	2 hr 7 min	$71.08
QA round 1	8.8 min	$3.24
Build round 2	1 hr 2 min	$36.89
QA round 2	6.8 min	$3.09
Build round 3	10.9 min	$5.88
QA round 3	9.6 min	$4.06
Total	3 hr 50 min	$124.70

Planner

Expands a short requirement into product context and high-level design.

Generator

Negotiates a sprint contract, implements, self-evaluates, and hands off.

Evaluator

Uses Playwright, checks UI/API/database behavior, and sends evidence-backed feedback.

TAKEAWAY