Green pipeline, wrong product

The agent finished. Formatters quiet. Unit tests green. The PR checklist looks clean. You open the Compose screen it “fixed” and the empty-state copy is still wrong, the retry CTA still opens the wrong sheet, and the new flag defaults to on for everyone. CI did its job. The contract never existed, so the agent optimized for “something that compiles and passes the suite I just authored.”

That is the gap after CI-as-done. Mechanical verifiers stop silent broken diffs. They do not stop confidently wrong features. The claim of this post: write an acceptance contract — a SPEC.md / plan.md, a failing acceptance test, or both — before the agent may edit product code, and make the harness refuse “done” until that signal flips. Everything below is that order, and the shortcuts that invert it.

The contract has to be falsifiable

A plan that says “improve empty state” is theater (planning with teeth). An acceptance contract is narrower: observable outcomes a stranger could score without reading the chat log.

# SPEC.md — compose empty state copy (agent brief)

## Goal
When the draft list is empty, show title T and body B; primary CTA opens New Draft.

## Non-goals
Do not redesign the toolbar. Do not touch send / sync.

## Acceptance (must fail before implementation)
- [ ] UI test: empty draft list shows T and B (ids stable)
- [ ] UI test: primary CTA navigates to New Draft route
- [ ] Flag `compose_empty_v2` defaults OFF; no module outside :compose:feature

## Done when
All acceptance checks green on a clean tree; PR description links this SPEC.

If nothing in the document can turn red, you wrote a vibe. You do not need a process cult. Three shapes still count: a one-page SPEC.md / plan.md plus 1–3 named UI checks; a failing test checked in alone with the PR description as the brief; or a golden trajectory for harness work. What does not count: “ensure quality,” a plan the agent never re-reads, or “looks good in the emulator” with no artifact.

Red before green — on purpose

Classic TDD instinct, aimed at the harness:

1. Write or generate the acceptance check from the SPEC
2. Run it — it must fail (missing UI, wrong route, flag default wrong)
3. Only then allow the coding agent to edit product sources
4. Stop when those checks pass — not when the model narrates success

flowchart TD
  brief[Human brief] --> spec[SPEC.md / acceptance tests]
  spec --> red{Acceptance red?}
  red -->|no / already green| stopBad[Refuse — contract invalid]
  red -->|yes| code[Agent codes]
  code --> gate[Format / lint / unit]
  gate --> acc[Acceptance suite]
  acc -->|fail| code
  acc -->|pass| done[Harness allows done]
  done --> ci[CI outer loop]

Figure 1. Acceptance must start red. CI remains the outer proof; the SPEC is the inner product gate.

If the acceptance suite is already green before any edit, the contract is wrong, already satisfied by accident, or the agent is grading its own homework. Fail closed.

Why CI alone still lies

Agents are excellent at closing the loop on tests they control. Delete an assertion. Widen a matcher. Add a unit test that mocks away the Compose navigation you cared about. Your pipeline stays green while the product promise evaporates.

Promotion of agent configs has the same lesson as evals that are not demos: one happy path is not a ship bar. Here the golden artifact is the SPEC + failing check, not a chat transcript.

On a large Android monorepo the failure mode is familiar: the agent “fixes” empty state in :app with a string resource, leaves :compose:feature untouched, and invents a unit test that never inflates a Compose hierarchy. Gradle is green. Users still see the old copy.

The same pattern shows up in harness work itself. You ask for a stop hook that blocks done until ./gradlew :compose:feature:testDebugUnitTest is green; the agent ships a Python script that prints PASS and a unit test that asserts the script exists. CI of the agent repo is green. The Compose module never ran. The SPEC was missing the sentence “acceptance = Gradle exit code on :compose:feature, not a surrogate.”

Put the order in the harness — not in a late diary

Put the order in the runtime, not in a polite system prompt:

1. No product edits until SPEC.md (or equivalent) exists and is linked from the task
2. No “done” until the named acceptance command exits 0 — same class of check CI will run
3. Reject already-green acceptance at task start (warn + require a human to rewrite the contract)
4. Diff-scope unit tests for speed; keep acceptance focused and few
5. Cap repair loops; escalate with the failing acceptance log, not another summary

Hooks and stop scripts already know how to veto chat tone. Extend that veto: missing SPEC or missing red-to-green transition is as invalid as a red ktlint job.

Rule of thumb - if the agent could pass by rewriting the test instead of the product, the acceptance check was not independent enough.

Teams starting spec-driven work often write plan.md first so review is cheap. The tempting shortcut is: let the agent code, paste an AI overview on the implementation PR, and open a second PR for the plan. Same filenames. Backwards order.

An AI overview is a reading aid for a diff that already exists. It lists files, restates the author’s story, and can still omit the toolbar rewrite that was never in scope. Saying no then means throwing away a Compose migration, not a page of markdown. A plan PR after the code is a diary fitted to the diff — reviewers rubber-stamp paperwork for a feature that already shipped.

plan.md before product edits is the cheap review. Humans reject wrong module, missing acceptance, or a flag default while the change is still words. Keep the AI overview on the implementation PR after the plan is approved — a map of the diff, not a substitute for the contract.

For a mail-style empty-state change, the plan says “copy + CTA + flag default OFF; do not touch the toolbar.” The agent still rewrites the toolbar. An AI overview narrates a thoughtful toolbar cleanup. A reviewer who already approved the plan fails the code PR in one line. A reviewer who only has the overview is arguing with a story.

Treat CI green as necessary and incomplete. Specs without teeth are planning theater; teeth without a product contract are green lies with better tooling. Wire the harness so “done” requires the red-to-green signal, and do not invert that order with a late plan PR.

References

• Effective harnesses for long-running agents (Anthropic) — clean state between sessions; initializer patterns
• Simon Willison — Using LLMs for code — tests as the feedback loop agents need (validate, don’t invent the bar)
• nax acceptance pipeline notes — independent acceptance vs agent-written implementation tests
• Spec-driven development: delegate implementation without losing control — spec → plan → tasks → code; the spec is a team contract, not a late diary
• AI code review packet (Jack M.) — PR overviews as evidence maps after a task contract exists