write-ups

Your Agent Is a Distributed System

article·2026

The Loop Everyone Writes

Every agent I have seen starts as the same handful of lines.

Call the model. If it asks for a tool, run the tool, append the result, call the model again. Stop when it says it is done.

It works on the first afternoon. It works in the demo. It reads like a program: a loop, a few branches, a return value.

It is not a program. It is a distributed system that happens to fit on one screen.

I spent two summers on data pipelines, streaming and ingestion work on Kafka, Flink and Spark, where a message that is lost or counted twice is a real problem and not a rounding error. When I started building agent tooling, I kept having the same uncomfortable feeling. I had seen these bugs before. Just not in a loop that talks.

This piece is about that feeling, and about what happens when you take it seriously.

The Model Is a Network Call

From the system's point of view, a model call is a remote request. It can be slow, time out, fail halfway through a streamed response or return something that does not parse, and you pay for it either way. All of that is ordinary.

But a model call has one property most network calls do not, and it is the one that matters: it is not repeatable. Ask the same question twice and you may get a different answer. Not a different encoding of the same answer. A different decision.

Setting the temperature to zero does not fix this. In a real serving stack, the numbers your request produces depend on the batch it was packed into, and the batch depends on everyone else's traffic. When Thinking Machines ran the same prompt a thousand times at temperature zero, they got 80 different completions. Your model's answer is partly a function of strangers.

It can be engineered away, at a real cost in throughput, and almost no one serving you a model has done it. And even a perfectly deterministic model only repeats itself if every byte of its context repeats too. A tool result with a timestamp in it is enough to send the run somewhere new.

When an agent retries, it does not resend a request. It regenerates one, using a nondeterministic function.

In 1994 Peter Deutsch and his colleagues at Sun wrote down the fallacies of distributed computing, the assumptions engineers make that sooner or later turn out false. The network is reliable. Latency is zero. Bandwidth is infinite. Agents have their own list, and most agent frameworks assume all of it.

The model is deterministic. Tools are safe to retry. The context is free. The run finishes.

None of those are true. The rest of this piece is what follows from admitting it.

The Retry That Charged Twice

Here is the bug that teaches you the most.

An agent is handling a support ticket. It decides to issue a refund and calls the refund tool. The call goes out. The response never comes back. Maybe the payment service was slow, maybe the connection dropped, maybe the process running the agent was redeployed mid-request.

Did the refund happen?

You do not know, and that is the oldest problem in distributed systems. When a request times out, you cannot tell "it failed" from "it succeeded and the reply was lost." Retry, and you risk doing it twice. Do not retry, and you risk not doing it at all. At-least-once or at-most-once. Pick one.

Streaming systems learned to live with this a long time ago, and the answer is worth stating carefully because it is widely misunderstood. Exactly-once is not something the network gives you. It is something you build: at-least-once delivery, plus effects that are either safe to apply twice or committed atomically with your progress. Kafka's transactions do the second, but only for writes back into Kafka. The moment your pipeline touches the outside world, you are back to making the outside world idempotent.

The standard tool is an idempotency key. Every request with a side effect carries a unique key, and the service remembers the keys it has already processed. Retry as often as you like. The refund happens once.

Here is where agents make it worse.

In an ordinary service, the retry sends the same request. In an agent, the retry often goes back through the model, and the model, asked again, may not ask for the same thing. It might refund a different amount. It might decide to apologize instead. If the key is derived from the arguments, a slightly different call gets a different key, and the second refund goes straight through.

So the key has to come from the step, not from the model's output: this run, this step, this effect. Never the attempt. The attempt is the thing you are trying to make not matter.

Something useful follows. Stripe, for example, compares a retry's parameters against the original request and returns an error if they differ, so a re-asked model that picks a different amount gets an error instead of a second refund. That is exactly the behavior you want.

The key also helps with a second problem. The process you thought had died may only have been slow. Now there are two of it, both acting on the same run. Kafka handles this with producer epochs: when a new instance claims a transactional ID, the old instance's writes are rejected. It is what Martin Kleppmann calls a fencing token, and the point of a fencing token is that it is checked by whatever applies the effect.

So an agent runtime needs a lease on the run and an epoch on every effect, checked at the boundary. Where that boundary is a tool gateway you own, the zombie's call bounces at the gateway. Where it is someone else's API that knows nothing about your epochs, the step-derived key does the fencing: the zombie and its replacement send the same key, and only one refund happens.

But that still leaves the deeper problem. It is not enough to make the effect idempotent. You have to make the decision durable.

Write Down What You Are About to Do

That idea is older than any agent framework. Databases call it a write-ahead log: write down what you are about to do, then do it.

Durable execution engines apply the same idea to code. Temporal is the best known. Your workflow code must be deterministic, and anything that is not goes through the engine. Network calls and other side effects become activities. Timers and randomness come from the engine's own APIs. Every result is recorded in an append-only event history. If the process crashes, the engine re-runs your code from the top, and whenever the code reaches a step that already completed, the engine hands back the recorded result instead of running it again.

Map that onto an agent. The loop is the workflow. The model call is the nondeterministic step. So the model's output is exactly what must be recorded. Not as a log line for debugging later, but as the event the system replays from: what context the model was given, what it decided, which tools ran, what they returned.

Do this and crash recovery stops being frightening. The process dies at step 37. A new one starts, replays 36 recorded decisions in milliseconds without calling the model once, and picks up at step 37.

But be precise about what that buys. Temporal guarantees that an activity is observed as completed exactly once. It does not guarantee that it executes once. An activity that was in flight when the worker died is retried, because the engine cannot know whether it finished either. Durable execution does not abolish the timeout problem. It shrinks it to a single step, and the idempotency key covers that step.

You need both. The history makes the decision durable. The key makes the effect safe.

Skip the history, and recovery means asking the model again, and the model takes a different path. You did not recover the run. You started a different run that shares a prefix with the first, and some of the first run's side effects are already out in the world.

One more consequence is easy to miss, and it is about versioning. Replay only works if the code that replays a history behaves like the code that wrote it. In an agent, the prompt, the tool set and the model are all part of that code.

Change the tool set so the loop takes a different branch, and Temporal will catch it: the replayed run no longer matches its history, which is a nondeterminism error. Change the prompt or the model, and nothing catches it. Those are just inputs to the model call, and replay checks which steps ran, not, in general, what was passed to them. The run carries on, its first thirty-six decisions made under one prompt and the rest under another, and nothing tells you so. Editing the prompt is a code deploy that your deploy tooling cannot see.

The Context Is Not Free

Of the four, this one is the most agent-specific.

An agent's context window is its working state. It is also a bounded buffer. As a run gets long, something has to give, and what usually gives is history: older turns are summarized or dropped to make room.

From a systems point of view, that is truncating the log you replay from. A summary is lossy. It keeps what the summarizer thought mattered and throws away the rest, and once the original turns are gone, no future step can recover the decision that was made in them. The agent does not remember deciding not to touch the payments module. It remembers a sentence saying the work went well.

The fix is to separate the two things the context is being asked to be. The durable history, every decision and every result, lives outside the model, in a store that is never compacted. The context is a view built from that history for this particular step, and it can be as aggressively summarized as you like, because nothing durable is lost by summarizing a view.

This is why I built Agent Relay the way I did. It hands coding sessions between agents, and it hands over state: the decisions, and the validation that already happened. A transcript is a log of what was said. The next agent has to re-derive the decisions from it, and re-deriving is exactly the nondeterministic step. A record of decisions only has to be read.

Checkpoints, and the One Thing You Cannot Roll Back

Replaying from the start works, but long runs get expensive to replay, so systems take checkpoints.

Flink's approach is one of the most elegant ideas I worked with. It is inspired by the Chandy–Lamport snapshot algorithm. Marker messages called barriers flow through the pipeline alongside the data, and each operator snapshots its state once the barrier has arrived on all of its inputs. The result is a consistent snapshot of a running distributed system, taken without stopping it. After a failure, every operator rewinds to the last snapshot, and replayable sources like Kafka rewind to the matching offsets.

Agent frameworks are arriving at the same place. LangGraph saves a checkpoint of the graph's state after each step, so a run can be paused, inspected, resumed or forked.

And even there, the old bug is waiting. When a LangGraph run resumes after an interrupt, the interrupted node runs again from the beginning. A side effect placed before the interrupt happens twice. The checkpoint is per step, and so is the guarantee.

There is also a limit that streaming systems handle with care and agent systems mostly ignore. You can checkpoint your own state. You cannot checkpoint the world.

When an agent rewinds to step 20, the email it sent at step 24 is still sent. The branch it pushed is still pushed. The refund is still refunded. Your checkpoint rolled back the agent's memory of doing those things, not the things themselves.

The Pivot, and the Human Who Signs It

In 1987, Hector Garcia-Molina and Kenneth Salem described a way to handle long transactions that touch many systems and cannot hold locks across all of them. They called it a saga. Break the work into steps, and give each step a compensating action that undoes it. Book the flight, book the hotel, charge the card. If the charge fails, cancel the hotel, then cancel the flight. The charge is the step everything hinges on.

The saga literature has a name for the step after which there is no going back: the pivot. Everything before it must be compensatable. Everything after it must be retriable until it succeeds.

Designing an agent's plan is mostly deciding where the pivot is. In practice that changes how tools are designed. Every tool with a side effect is classified. Compensatable, with a known undo: create a draft, open a branch, stage a change. Irreversible: send the email, merge to main, move the money. The plan is ordered so the compensatable work comes first and the pivot comes as late as it can. If a plan has two irreversible steps, only one can be the pivot. The other has to come after it and be made retriable, like an email send with its own idempotency key.

One caveat matters a lot for agents: compensation is not rollback. Sagas give up isolation, so other systems may already have seen the intermediate state. A deleted branch was still pushed, and CI may already have run on it. "Reversible" means there is a known way to compensate, not that it never happened.

Flink already has the right shape for the pivot. Its exactly-once sinks write output inside a transaction and leave it uncommitted. Only when the checkpoint completes does the sink commit, and everything before that moment can be thrown away. An agent should work the same way. Stage every effect it can, and commit when someone says the checkpoint is good.

That someone is often a person, and a person is a very good commit point. The agent does all the work that can be undone and presents a single decision: here is what I am about to do. The person signs the pivot, which is where judgment is worth the most.

The difference is that Flink's checkpoint coordinator answers in milliseconds, and a person might take a day. That matters because two-phase commit has a famous weakness. If the coordinator disappears after asking everyone to prepare, the participants hold their prepared state indefinitely. A human is the slowest, least available coordinator you can choose. So every approval needs a timeout with a default outcome, and the default should be to compensate, not commit. The agent can do what a two-phase commit participant cannot, which is give up, because nothing it has staged has been promised to anyone yet. Human latency also outlives other systems' guarantees. Stripe can prune an idempotency key after 24 hours. If the approved refund call times out on Friday night and the run is parked for a person until Monday, the retry carries a key Stripe may no longer remember.

Flink has the same failure in a faster form. Its Kafka sink holds output in an open Kafka transaction until the checkpoint completes, and Flink's documentation warns that if the checkpoint and restart take longer than Kafka is willing to hold that transaction open, Kafka aborts it and the staged output is lost. Every staged effect has a shelf life, and a person is the slowest checkpoint there is.

I wrote in an earlier piece that every action should be reversible or it should be hard. This is that principle as an architecture.

The Run Has to Finish

The last fallacy is that the run finishes. In the basic loop, nothing says stop. Each step can create more steps.

Systems people answer this in two ways, and agents need both.

The first is admission control. Every run gets a budget, in tokens, tool calls and wall-clock time, enforced by the harness rather than requested politely in the prompt. Deadlines propagate the way they do in well-built RPC systems: a sub-agent inherits what is left of its parent's time, not a fresh allowance.

The second is backpressure, and agents mostly have none. In a pipeline, a slow stage pushes back on the stage feeding it, and the whole system slows to the pace of its slowest part. Agents produce work for other things: tools, other services, and very often a human review queue. When that queue is saturated, a well-built agent system should slow down or stop producing, and today almost nothing gives an agent that signal. It just keeps opening pull requests. A circuit breaker on a failing tool is a crude stand-in for it, and it should be shared across runs, or a hundred agents will each discover the same outage by themselves.

And then there is the poison message: one that crashes its consumer on every delivery and blocks everything behind it. The fix is a dead-letter queue that sets it aside for a person.

Agents have poison tasks. The task that cannot be done with the tools available, so the agent tries, fails, rephrases, tries again, and burns its budget producing increasingly creative versions of the same failure. A good agent system has a dead-letter path for work. After a bounded number of attempts, the run stops, packages up what it tried and why it thinks it failed, and hands that to a person.

A queue that cannot give up on a message eventually gives up on everything behind it.

You Cannot Debug What You Did Not Record

A distributed system is debugged from traces that show each request the way each component saw it.

For an agent, the request each step sees is its context: everything the model was given at that moment, in that order. That is the input to the nondeterministic step. If you did not record it exactly, you cannot explain the decision that came out. You can reread the agent's reply as many times as you like. Without the exact context, you are guessing.

So every model call is a span, and the span refers to the full context as it was sent. That has a cost. Contexts are large and often full of personal data, which is why the OpenTelemetry conventions for generative AI make capturing message content opt-in. Contexts also strain the limits durable execution engines put on what they record. Each turn re-records a context that is bigger than the last, so the history grows far faster than the run. The answer to both is the same: store each context once, outside the history, addressed by its content and kept under the same retention rules as the data inside it, and record a reference to it in the history and in the span. It is the only way to answer the question that matters after something goes wrong: given what it could see, was this a reasonable decision?

The Coordinator Is the Byzantine Node

I have been treating the model as an unreliable network call. Here is where it gets stranger.

In most distributed systems, the unreliable parts are the network and the machines. Machines fail by stopping, and networks by losing or delaying messages. The coordinator, the logic deciding what happens next, is the part you trust. You wrote it. It does the same thing every time.

In an agent, the coordinator is the unreliable part, and it does not fail by stopping. It fails by being confidently wrong.

Distributed systems does have a name for that. A component that is confidently wrong is a Byzantine fault, and the classical answer, going back to Lamport, Shostak and Pease in 1982, is to replicate it and vote. With 3f+1 replicas you can survive f that lie.

That answer assumes the replicas fail independently. Software reliability learned the hard way how fragile that assumption is. In 1986 Knight and Leveson had separate programmers write the same program and found the versions failed on the same inputs far more often than chance allows. Three samples from one model are not even separate programmers. They share the same blind spots, so they tend to be wrong together. I wrote about that gap in Your Model Ensemble Is One Replica. The short version is that "ask it three times" does not buy what a quorum buys.

And there is a deeper gap. A crashed node at least goes quiet. A timeout fires, and even if you cannot tell slow from dead, you know something is wrong. A wrong decision never goes quiet. It returns quickly, parses cleanly and looks exactly like a right one. The oldest problem in distributed systems is that you cannot tell slow from dead. The new one is that you cannot tell wrong from right.

So the harness has to treat the model's output the way a server treats a request from an untrusted client. Not as an instruction. As a proposal to be checked.

Which is why the answer is not a better model. It is a demotion. The harness is the coordinator. The model is a participant whose vote is checked.

The model proposes. The system disposes.

Tools are typed contracts, and arguments are validated at the boundary. Permissions follow least privilege: an agent that reads tickets does not hold a credential that can issue refunds, and an agent that can issue refunds cannot issue unlimited ones. The durable history, not the model's memory, is the source of truth about what has happened. When the model and the log disagree, the log wins.

What I Would Tell Someone Writing the Loop Today

Write the loop. It is the right way to start. Then, before it touches anything real, ask it the questions you would ask any distributed system.

What happens if this step runs twice? Where is the decision recorded, and can a new process replay it without asking again? Which actions can be undone, and who signs the ones that cannot? What stops this from running forever? When it goes wrong, will I be able to see exactly what it saw?

If you have answers, you have an agent system. If you do not, you have a demo that has not met its first timeout yet.

Every reliable system we have was built from parts that fail. We have never built one where the part most likely to fail is the part deciding what to do next.

Most of the engineering is old. That is the new part.