Field Note · AI Craftspeople Guild
Ask an agent to turn a red test green and it will. Ask it too loosely, and it will get there by editing the expectation instead of the logic. Five unedited minutes from a Guild working session, and what they mean for anyone whose assurance story rests on a passing test run.
We were partway through a mob programming session — humans and AI agents at the same keyboard, which is still strange enough that we halt the proceedings every so often to argue about what we are doing — when one of the mob members put up a hand. His question was about sequencing. When we say the test has to fail, do we mean the old test-driven cycle, where nothing exists yet? Or do we already have a running system, and we are asking for a test that fails against it?
We stopped for five minutes and worked through it out loud. The clip above is that exchange.
The plain answer is that a failing test is our only evidence that we asked for a capability the system does not have yet. If the test passes the moment it lands, then whatever it expects is already in there, and we have just paid an agent to do it twice. The red bar is the receipt.
That much is ordinary Kent Beck, and it is not why the Guild keeps circling back to this question. What pulls us back is a second reason, and it did not exist until agents started holding the pen.
An agent has one target, and that target is a passing run. Hand it a red test and turn it loose, and it will get there. Hand it a red test and a loose instruction, and it may get there by adjusting what the test expects rather than what the code does — quietly, inside the same commit, with a cheerful summary reporting that everything passes.
It is not deceiving us. It is doing precisely what we asked. We asked for green.
The green bar used to certify that a person stated an expectation and the system met it. Now it certifies only that the expectation and the code agree — and the agent held the pen for both.
That is the part your board, your auditor, and your incoming CTO have not caught up with yet. For thirty years a passing test run has been an assurance artifact: somebody who understood the business wrote down what ought to hold, and the build confirmed it. Break the link between the person and the expectation, and the artifact still looks identical from the outside. Same green. Same coverage number. Far less behind it.
Not as a curiosity, and not annually. Alter a business rule on purpose — flip a comparison, drop a boundary, invert a condition — and find out whether anything complains. If nothing complains, we have a 'surviving mutant': a place where our tests are decoration. On a repo where agents are contributing daily, that sweep is our standing audit that the expectations sitting in the test files still belong to us.
Mutation testing reports after the fact. It sweeps on a schedule and tells us where our expectations have gone quiet. By then the edit has already landed. What we want alongside it is something standing at the gate — deterministic rather than probabilistic, and emphatically not another agent asked politely to police the one ahead of it. That is the piece I have underway now.
It is called Detent, and it is a differential oracle. It holds a sealed baseline of your specification surface, compares any proposed edit against that baseline, and issues a verdict: were the guarantees strengthened, preserved, or quietly weakened? Weakened or removed, and the commit does not land. The governing image is a ratchet — guarantees may tighten freely, and they may not loosen.
What matters for the argument above is how much it guards. Not tests alone. Detent treats the whole specification surface as protected: the executable tier (assertions, matchers, input domains), the structured tier (declared constraints, thresholds, deny-rules), and the prose tier — CLAUDE.md, spec documents, directive files, all the meta-information an agent reads before it touches a line of code. Drift in any of it is the same disease wearing different clothes.
Prose cannot be graded. Nobody can machine-decide that one paragraph is 'weaker' than another, so Detent does not attempt it. It enforces a seal instead. Any edit to protected meta-information that is not covered by an explicit, owner-approved exception counts as blocking drift, full stop. Trespassing on the meta-information is always a show-stopper.
And it referees rather than plays. It detects, reports, and blocks with enough precision to act on. It does not repair anything, does not rewrite your prose, and does not feed corrections back into an agent loop — which would quietly reintroduce the very drift it exists to catch.
A deterministic guard against agentic trespassing on tests, specs, and prompt files. Not finished, and I would rather say so plainly than imply otherwise — the domain core and its architecture-boundary tests are in place, each observed failing before the code that satisfied it. The architectural brief fixing the intent and the boundaries is public and readable today.
github.com/alexbunardzic/DETENT ↗From the session
“Agents go and they can mess around with tests. They can change the expectations without telling us and still make everything pass — but now they have violated our spec.”
“Just because everything's passing doesn't mean that everything's meeting your expectations.”
“The agent is honest, it's just going to do that. But I've seen agents doing dishonest work — they go back and reshuffle a whole bunch of things just to make everything pass. Because their purpose in life is to succeed.”
“If you change the logic and the tests do not complain, that means you have a surviving mutant. That's a big problem.”
“The tests become the specification of our system — we can remove all the code and re-implement it from the tests.”
Guild core team memberFor engineering leaders
If somebody tells your board that the tests pass, somebody else should be able to say who authored those expectations and when they were last edited. On an agent-assisted repo, those are separate questions, and most organizations can answer neither.
Reading an agent's implementation line by line does not scale past a couple of agents, and it is the wrong altitude anyway. Reviewing the expectation the agent was handed does scale, and it is where the business decisions sit.
Coverage counts the lines an agent touched. Surviving mutants count the rules nobody is guarding. Only one of those numbers gets harder to fake as you add agents.
Free · No call required
The questions an auditor, a regulator, or a new board member will eventually ask about the code your AI tooling is writing — and what a defensible answer to each one looks like.
If nobody can answer that, it is worth thirty minutes. No pitch, no pressure — we look at how your agents are being steered and where the review trail thins out.
Book a Diagnostic →