The Agent Must Refuse, Not Guess

I audited one form that an agent had filled and submitted on my behalf, and found five answers on it that were not true.

Not lies, exactly. The agent had not decided to deceive anyone. It had been asked, by a dropdown, which state I lived in, and its code said: if no answer is configured for this field, pick the first option. The first option was Alabama. It was asked how many years of experience I had, and picked the first range, which was zero to five. It was asked whether I would need visa sponsorship, and the first option was yes.

Every one of those was a fallback I had written, months earlier, to stop the driver from stalling on a field it did not recognise. I had solved the crash and created something much worse, because a crash is visible and a wrong answer is not. The form went through. The confirmation page came up. The log said success.

This essay is about what it took to make the agent stop doing that, and why the first fix did not work.

The blocklist did not work

My first instinct was to make a list of things the agent must never guess. Location. Years. Sponsorship. Citizenship. Anything with legal or factual weight. If the field matches the list and there is no configured answer, stop and ask.

This held for about a week. Then a form asked the same question with different wording, the matcher missed it, and the fallback fired. Then a form asked something that was not on the list at all, because I had not imagined it, and the fallback fired. A blocklist is a list of the mistakes you have already made. The next mistake is, by construction, not on it.

The fix that actually worked was to delete the fallbacks. All of them. If the agent reaches a field it cannot answer from what it has been given, it does not answer. It writes the field, the question, and the URL to a file I read, marks the form as needing a person, and moves on to the next one. No form is submitted with a guess in it, because there is no code path that produces a guess.

The cost is that some forms sit unfinished until I answer one question. That turned out to be a small cost. It is usually one question, it usually takes me ten seconds, and every time it happens the answer goes into the established facts so that question never comes back. Over a few months the needs-a-person file got shorter, and it got shorter because the agent was learning what I would say, from me, rather than inventing it.

The rule generalises past forms

Once I had seen the fabrication in the form driver, I started seeing it everywhere.

A ledger that counted a prepared application as a submitted one, because the code that wrote the row ran before the code that submitted. An answer generator that, when it could not find a fact in the reference material, produced the most plausible-sounding fact. A summariser that filled a gap in a transcript with what the speaker probably said. None of these were crashes. All of them were confident output where the honest output was “I do not know.”

So the general form of the rule is: when the agent lacks the information to produce a correct output, the correct output is a refusal, routed to a person, and never a plausible substitute. The refusal has to be cheap for the agent to produce and cheap for me to resolve, or the agent will drift back to guessing because guessing is what makes the run look finished.

Verify the artifact, not the exit code

The second rule came from a different failure and turned out to be the same lesson.

A tool that generates a document ran, exited zero, and produced a file. The file was empty. A publishing step ran, reported success in green, and had published nothing, because the upstream build had silently dropped every asset it was supposed to include. A deploy script said the site was up, and it was, at the previous version.

In each case the agent had checked that the step ran, not that the step did the thing. The rule now is that every step that produces an artifact is gated on the artifact. A generated PDF is opened and its text is read back. A deployed page is fetched and its content is checked for the change. A submitted form is followed to its confirmation page, and the platform’s own “submitted” state is read, and only then is the ledger row written. Exit codes are for finding out why something failed, not for deciding whether it succeeded.

This is more code, and slower, and the runs look less clean. It is also the only reason I trust the runs at all.

Prior reports are claims

There is a corollary I did not expect, which is that the agent has to be sceptical of its own predecessors.

Every session hands off to the next with a written report, and the reports are good, but they are written by a system that is rewarded for reporting completion. A report that says “the sending pipeline is working” was true when it was written and may not be true now. A report that blames a vendor bug for a failure is, in my experience, wrong more often than it is right, and “vendor bug” is the phrase I now treat as a signal to go and look.

So the handoff rule is: a report is a claim, and a claim about the live system gets re-verified in the live system before it is acted on. Cheap to check, expensive to skip, and it has caught a stale claim in roughly one session out of four.

Two dry batches means the channel is closed

The last rule in this family is about knowing when to stop, and it is the one I would tattoo somewhere if I were the type.

An agent working a rate-limited resource, a contact-reveal endpoint, a search that throttles, an API with a quota, will see its yield drop as the limit engages. The natural thing for an agent to do is retry, on a timer, hoping the limit has cleared. This is exactly wrong. On one endpoint I watched the yield go from six in twelve, to one in fourteen, to one in five, to zero, zero, zero, zero, across hourly retries. The retries were not waiting out the throttle. They were hardening it.

The rule now is that two consecutive empty batches means the channel is closed for the session, and the agent stops. It does not schedule a retry. Instead it builds a small standalone page on my desk with every remaining item, the link, enough context to triage, and a place to paste what I find, and it tells me the page exists. Ten links take me two minutes to click by hand. No amount of clever retry logic beats that, and every retry the agent would have made costs tokens, time, and a little more of the endpoint’s patience.

What these have in common

All four rules are the same rule from different sides. The agent’s job is not to make the run look complete. It is to make the run be complete, and to say clearly, cheaply, and early where it is not. A refusal, a verified artifact, a re-checked claim, a closed channel handed to a person: each is the agent declining to paper over a gap with something plausible.

The paradox is that the runs that follow these rules look worse in the log. More items marked needs-a-person. More steps that took longer because they read back what they produced. Fewer green checkmarks. And they are the only runs whose green checkmarks mean anything.

The driver with the fallbacks deleted, the needs-a-person routing, the artifact gates, and the handoff page generator are the pieces I install rather than post. The rule is in the title.