How Reliable Is Your AI Agent?

For about three weeks I thought my server host was robbing me.

The agent I run, Ghost, is my own instance of Hermes, an open-source framework from Nous Research. It works on a rented box all day with me nowhere near it: research, drafts, scheduled jobs, the unattended autonomy everyone is being sold. In May it started slowing down. A job that took a minute took five. The server’s own dashboard blamed “steal time,” the polite name for a noisy neighbour on shared hardware eating the processor. So I did the normal thing. I complained to support, read forum threads about oversold hosts, priced a migration.

The neighbour was me. The failure had spent three weeks disguised as someone else’s fault, which is exactly what the dangerous ones do.

Ghost runs each of its tools inside a throwaway container and, by default, never deletes the dead ones. They piled up. When I finally ran the one command that would have told me on day one, the list of dead containers filled the screen and kept scrolling: 748 of them, all but five long dead, and the machine underneath had seized trying to keep track. The host throttled everything. Nothing was broken in the way broken usually looks: no crash, no error, no alert, only a number climbing by one, over and over, for weeks, with nobody reading it.

That counter is what running an agent on your own actually looks like. The demo writes you a poem. The real thing is a number climbing in the dark while the disk fills.

FIG·02 — Nothing broke; a number climbed: 748 containers, all but five long dead, unread for weeks, into the cliff.

Grade it twice

The counter was one way Ghost had fooled me. There was another, and this one you can run on your own agent this week.

I ran a proper audit of its research against primary sources. Twenty-two factual claims about companies and markets, checked one at a time. Twenty came back directionally right, the right company and the right direction and the right thesis. Ninety-one percent. You could sell that number.

Then I graded the same twenty-two strictly. Was every specific right too, the exact figure, the exact date, the exact quarter. Seventeen. Seventy-seven percent.

That gap is the entire problem. The claims Ghost got strictly wrong were not inventions. They were the boring kind of miss: a price that was right last week, a date that had drifted, a version that had moved on. It had the shape of the world right and its current state wrong, and current state is the part you act on.

So an agent that is confidently, directionally right is more dangerous than one that is obviously wrong. The obviously wrong one you check. The directionally right one earns your trust and then spends it on a stale figure you paste into a memo for someone who acts on it. If you have ever pulled a number from an AI and dropped it into a deck, you have shipped one of these without knowing.

A 91% that hides a 77% is the exact accuracy at which people stop checking.

The audit takes an hour and tells you more about your own agent than any benchmark can. Take twenty of your agent’s claims and grade them twice, once for the shape and once for every specific, current as of today. The spread between the two scores is your blast radius, and it is always wider than the single number you have been quoting.

FIG·03 — Grade it twice. The 91% that hides a 77%, and the three claims you would have shipped.

It was never the model

Once I had that shape in my eye I saw it everywhere, and never in the model. The scheduler reported success whenever Ghost replied at all, so a job could fail outright, write back that it could not fetch the data, and still get logged green because something had come back. A config change I made was silently overruled by a second file that loaded later and won, pointing the agent’s storage at the wrong place; the system did exactly what the files told it, and nothing reconciled the two. The search index ballooned overnight to a size with no relation to the data inside it, until the disk hit 100 percent, the agent started returning “no space left,” and the work stopped, with nothing watching it grow.

None of this was the model’s fault. Resource leaks, configs that override each other, success signals that lie: these are the oldest problems in running software, and intelligence buys no exemption from them. The model underneath was cheap and fast; a frontier one would have hit all of them the same. A smarter model slips less often, and it still cannot see the slip it makes, because the evidence sits outside it. Every failure started in the same place: the agent did something, and nothing outside it looked at the result. A better model changes the odds. It does not remove the need for something outside the agent to look.

That was the turn for me. I had spent weeks grading Ghost on how clever it was. What decides whether an agent is safe to leave alone is duller and far harder to fake: how much of what it does gets checked by something that is not the agent.

An agent cannot be the thing that confirms its own work.

It reasons from inside its own process, where a sandbox that cannot start a container and a whole host that is down look identical. It will hand you a confident account of which one it is, and the account is worthless, because the fact that would settle it sits on the other side of a wall it cannot see over. An agent that grades its own work can always move the grade, which is why the only gate a self-rewriting agent cannot game is time: the checks that hold are the ones it has no hands on.

Different operators, same answer

For a while I assumed this was specific to my setup. Then I read the operators who run Hermes hardest, and kept finding my own containers in their notes. None of us had compared notes; we were solving different problems, with different tools, in different words. One, tired of trusting the agent’s own edits, makes every self-change a diff a human signs off before it goes live.¹ The setup guide everyone passes around spends its length on the dull perimeter: what each surface may touch, what runs sandboxed, what a human approves.² Others come at it from other directions, building evals to close the loop on output³ and pruning the skills the agent writes for itself,⁴ and the instinct is identical every time.

Everyone who runs one of these for real comes back having built the same thing. Not a smarter model. A verifier.

This finds you whether or not you run a server. It bites anyone who lets an AI do something they then act on: the draft you send without rereading, the figure you quote because it sounded sure, the report you stopped opening because it is always fine. An agent does not have to be autonomous to fool you. It only has to produce something you have stopped checking.

You do not need your own month of this to learn what it teaches.

Where the check has to live

Fixing all of it was the same move every time. Put something outside the agent that can see what it cannot, and have it watch what is actually running, not the agent’s account of it. The form that takes depends on the failure.

The silent pile-ups, the containers and the disk, each had a leading number, one that moves long before the crash: the count of dead containers, the size of the index, the free space left. Those you watch directly. Set the alarm well below the cliff and read it far more often than it can break, every half hour rather than every twelve hours, so the warning fires while the problem is still a number and not yet a wall.

The lying scheduler was harder, because there the agent was the one producing the success signal. If a check can be passed by the thing it is meant to be checking, it proves nothing; what you need is a record the agent cannot paint green just by replying. So the failure log gets written from the real error, by the harness itself, where it fires whether or not the model chooses to cooperate.⁵

The stale figures got the narrowest fix of all. Ghost re-fetches a price or a date the moment it writes one, and never quotes from memory, because memory is where staleness hides. You pay that cost only on the things that decay, the price, the date, the version, the quarter. What does not move, it is allowed to remember.

All of that catches a failure after it happens, which is enough when the damage can be undone. A full disk clears. A stale figure gets corrected. It is not enough for money that has already moved, a post published under your name, or a file deleted. So the real sorting key is reversibility. If an action can be undone, a watcher behind it will do. If it cannot, the check has to sit in front of it, and the check has to be a person.

Ghost is barred from the irreversible ones. When it decides one is needed it writes a short request and stops. The operators who have run Hermes longest come to the same instinct from the other side: least privilege per surface, so the session you are sitting in front of can touch everything while the job that runs at four in the morning gets web and files and nothing else.² Where you cannot gate an action, make it reversible instead, snapshotting before a change and archiving instead of deleting, so there is always a state to roll back to. The boundary is dumb and absolute on purpose.

A safety rule an agent can argue its way around is not a safety rule.

Look back at those fixes and they share one property. The counter it cannot fake, the log the harness writes instead of it, the date it has to re-fetch, the person standing in front of the irreversible action: the agent has no hands on any of them. That is the whole requirement. A check an agent can reach, it learns to play to, which is how most verification quietly fails: the check stops testing the work and starts testing the performance the agent puts on for the check. The watcher that survives is the one the agent cannot forge, whether it can see the watcher or not.

None of this made Ghost smarter. It made Ghost watched, and watched turned out to be what mattered. The harness around a model now drives more of the real-world difference than the choice of model does, and a month of cleaning up after Ghost is that argument with scorch marks on it. The watchers and the gates and the re-fetches are where an autonomous system’s real competence lives. You will swap the model next month. The instruments stay.

An afternoon with a pen

So the work that matters has nothing to do with grading the output, the one part that was never going to break. It is an afternoon with a pen. Write down everything your agent does while you are not watching: every scheduled job, every file it writes, every figure it fetches, every change it makes to itself. That list is its real reach, and it is always longer than you expect. Then put each line through three questions. Can you undo it, and if not, does a person stand in front of it? What looks at the result, and is it anything other than the agent itself? And the question that catches what the first two miss: could the agent make that check pass without doing the work?

FIG·04 — The verification pass: Reach, Reversibility, Witness, Forgery.

Take the dullest case you like, an assistant that drafts your weekly update and posts it to the team channel on a standing job. A posted message is read before you can take it back, so a person should see it first. Then ask what actually confirms the figures in it, and whether that is anything more than the assistant rereading its own draft. Ask whether it could report “posted, all good” with last week’s numbers inside. Two of those checks are usually empty, and the empty ones are where it bites.

Every action checked by nothing but the agent is one of my 748 containers.

It is not failing yet.

It is adding one to a counter nobody is reading.

Run it on your own setup this week. Which of your agent’s actions is checked by nothing but the agent itself?

The Durability Curve is where I write up what outlasts the model: the harnesses, the checks, the structure that is still standing after you swap the engine underneath. Subscribe for the rest, or start with what survives.

Footnotes

1. Tony Simons (@tonysimons_), announcing Hermes Dreaming (X, May 2026). A plugin that stages an agent’s proposed self-edits as create, diff, validate, then apply or discard, so a human reads the change before it touches live state. ↩

2. zaimiri (@zaimiri), “8 Hermes Agent Settings You Need Before Building Anything” (X, June 2026). Argues the settings that matter most are the perimeter ones: per-surface tool access, sandboxed execution, manual approvals, and scheduled jobs set to deny by default. ↩ ↩²

3. Machina (@exm7777), “How to Fix AI Slop” (X, May 2026). Frames inconsistent output as a quality-control gap rather than a prompt problem, and builds the fix as an eval loop: generate, score against a written benchmark, gate, and feed every failure back as a permanent test case. ↩

4. The Hermes Curator, Nous Research, as documented by mem0 (@mem0ai) (X, May 2026). A background pass that ages, archives, and reviews the skills an agent writes for itself, with critical skills pinned out of its reach. ↩

5. Aparna Dhinakaran (@aparnadhinak), a code-level review of the Hermes harness (X, May 2026). Notes that lifecycle hooks can block or rewrite an action at the harness layer, enforcing policy independent of the model’s cooperation. ↩

---