What a Token Actually Costs
The Bill Says Tokens
Every model API prices the same unit. Tokens in, tokens out, so much per million.
It is a sensible thing to bill. It is also a bad mental model to build on, because it suggests every token is the same: a uniform, fungible piece of work, and the way to make a system cheaper or faster is to use fewer of them.
That is not what is happening on the GPU. On the hardware, some tokens are nearly free and some are expensive, and what decides which is mostly not arithmetic. It is memory: the bytes a token makes the chip hold, and the bytes it makes the chip move.
This piece is the explanation I wish I had been given first.
Two Very Different Jobs
Generating text with a transformer is two jobs that happen to share a model.
The first is prefill. The model reads your entire prompt and builds its internal representation of it. Every prompt token can be processed at once, so prefill turns the model's matrix-vector products into matrix-matrix products. Give it more than a few hundred tokens and it is compute-bound: it goes as fast as the chip can do arithmetic.
The second is decode. The model produces the output one token at a time, because each new token depends on the one before. For every output token, the model runs a full forward pass, and to do that the GPU has to read every weight in the model from memory. All of them, once per step.
Put numbers on it. Take an 8-billion-parameter model, Llama 3.1 8B, stored in 16-bit precision. That is about 16 gigabytes of weights. An H100 SXM reads its memory at about 3.35 terabytes per second. Reading the weights once takes about 4.8 milliseconds, which puts a theoretical ceiling of about 210 tokens per second on a single stream, however fast the chip can multiply. Real kernels get to perhaps 80 or 90 percent of peak bandwidth, so the practical number is lower.
And the chip can multiply very fast: about 990 trillion dense 16-bit operations per second. Producing one token from an 8B model needs about two operations per parameter, around 16 billion. So at a batch size of one, decode asks for roughly one operation per byte it reads, on a chip that can do about 300 operations in the time it takes to read that byte. For most of decode, the arithmetic units are waiting for memory.
This is a dense model on one GPU, the simplest case. Split it across eight GPUs and each reads an eighth of the weights with its own bandwidth. Make it a mixture-of-experts model and each step reads only the experts the batch actually uses, which is the same idea as the rest of this essay, built into the architecture. Store the weights in 8 bits instead of 16 and the floor halves. Most of why FP8 and 4-bit serving exist is in that sentence.
So the two jobs have opposite shapes. Prefill is wide and limited by compute. Decode is long, narrow and limited by memory. Time to first token is mostly prefill, plus however long you waited in the queue. Tokens per second after that is decode.
They are different enough that putting them on the same GPU causes trouble. A long prefill landing in the middle of a batch stalls every sequence that is decoding. That interference, plus queueing, is why your time to first token can change with other people's traffic. Serving systems now break prefills into chunks and interleave them with decode steps, as in Sarathi-Serve, or run prefill and decode on separate GPUs or machines altogether, as in DistServe and Splitwise.
Why Batching Helps, and Where It Stops
If decode spends its time reading weights, the obvious move is to make each read do more work.
That is batching. If 64 requests are decoding at once, the GPU reads the weights once per step and uses them for all 64. Arithmetic per byte of weights goes up roughly in proportion to the batch size, until the weight multiplications finally become compute-bound at a few hundred sequences.
This is the main reason serving at scale is cheaper per token than running a model for one person. It is not that big providers have faster chips. It is that they have enough concurrent requests to fill the batch.
Naive batching has a problem, though. Requests finish at different times. One answer is ten tokens long and another is two thousand. Batch them together and wait for the batch to finish, and the short requests hold their slots until the longest one is done.
The Orca paper (Yu et al., OSDI 2022) fixed this with iteration-level scheduling, now usually called continuous batching. Instead of scheduling whole requests, the server schedules every decoding step. A request that finishes leaves the batch immediately, and a new one joins at the next step. Orca also batches requests that are at different positions in their sequences into the same matrix multiplications, and runs attention for each request separately. The batch stays full, and nobody waits on a stranger's essay.
But batching only shares the weights. It does nothing for the other thing decode has to read, and that turns out to be the real limit.
The KV Cache
When the model generates a token, it attends to every token before it. Recomputing the whole sequence's internal state at every step would be wasteful, so the model saves two vectors, a key and a value, for every token at every layer. That saved state is the KV cache.
The size per token is easy to work out: two (key and value), times the number of layers, times the number of key-value heads, times the size of each head, times the bytes per number.
For Llama 2 7B, with 32 layers, 32 heads of 128 dimensions and 16-bit numbers, that is 512 kilobytes per token. A single 4,000-token conversation holds two gigabytes of cache. Llama 3.1 8B uses grouped-query attention, which shares each key-value head across several query heads. It keeps 8 key-value heads instead of 32, so its cache is 128 kilobytes per token, a quarter of the size. Grouped-query attention is a modeling decision made for memory reasons. Multi-query attention, which keeps a single key-value head, went further.
Now do the budget. An 80-gigabyte GPU holding 16 gigabytes of weights has about 60 gigabytes left for cache. At 128 kilobytes per token, that is roughly 450,000 tokens of context, shared by every request on the GPU. It is a budget you divide however you like: a hundred 4,000-token conversations, or fourteen at 32,000.
Long context is not a bigger number on a spec sheet. It is concurrency taken from someone else.
The cache also costs bandwidth, and this is where the batching story turns. During decode, every step reads the cache of every sequence in the batch. The weights are shared across the batch. The cache is not. Each sequence reads its own, alone, so batching does nothing for it.
Fill that H100 with about 110 conversations of 4,000 tokens each, and a single decode step reads around 60 gigabytes of cache and 16 gigabytes of weights. Nearly 80 percent of the traffic is cache. That step takes about 23 milliseconds at best, which works out to about 44 tokens per second for each conversation and about 4,900 across the GPU, as a ceiling. And the weight multiplications are still not compute-bound: at that batch size they do about 110 operations per byte, well short of the 300 the chip could sustain. You run out of memory for the cache long before you run out of arithmetic.
At that point the weights are no longer the main cost of a token. Remembering is.
Paging, Borrowed From Operating Systems
Early serving systems handled the cache naively. Each request was given one contiguous block of memory big enough for the longest output it might produce. Most requests never used most of it, and the gaps between blocks could not be reused.
The vLLM paper (Kwon et al., SOSP 2023) measured how bad this was: in existing systems, only 20 to 38 percent of the memory reserved for the cache held actual tokens. Its fix, PagedAttention, is an old idea in a new place: virtual memory. The cache is split into small fixed-size blocks, a table maps each sequence's positions to physical blocks, and blocks are allocated only as tokens are generated. The waste drops to near zero, under 4 percent by the vLLM team's own count, and the paper reports two to four times the throughput of the previous state of the art at the same latency.
PagedAttention does not make a step read fewer bytes. It wastes fewer of the bytes you hold, which is what lets more sequences share the GPU and the batch grow.
Paging also makes sharing possible. If several sequences begin with the same tokens, they can point at the same physical blocks and copy them only when they diverge. That is copy-on-write, also borrowed from operating systems, and it makes the next idea practical.
The Prompt Is a Cache Key
If two requests begin with the same tokens, their cache for that shared beginning is identical, so there is no reason to compute it twice.
Prefix caching keeps the cache of recently seen prefixes and reuses it when a new request starts the same way. vLLM now does this by default. SGLang's RadixAttention organizes cached prefixes in a tree so that many requests branching from a shared beginning all reuse the common part. On a hit, that part of prefill is skipped entirely.
Notice what that trade is. Prefill is arithmetic. A cached prefix saves the arithmetic by holding bytes, in the same scarce GPU memory the last two sections were about. So it does not last forever. vLLM evicts the least recently used blocks. Providers keep cached prefixes for minutes to hours, and longer retention usually costs more or lives in slower memory. The discount providers give for cached input, at the time of writing often a tenth of the normal price, is really them renting you that memory, or the slower memory behind it, for a while. Writing to the cache can cost more than ordinary input.
The beginning of a prompt is a cache key, and it only matches if it is identical, token for token, from the very first token. Systems cache in whole blocks and often require a minimum length, but the rule that matters is that one changed token invalidates everything after it.
So order matters. Stable content goes first: system instructions, tool definitions, reference documents. Content that changes goes last: the user's message, the latest tool result. A timestamp, a request ID or a per-user greeting near the top silently cancels the cache for everything below it.
Spending Idle Arithmetic
Decode leaves most of the GPU's arithmetic idle. Speculative decoding (Leviathan et al. and Chen et al., both 2023) is a way to spend it.
A small, fast draft model guesses the next several tokens. The large model then checks all the guesses in a single forward pass, which behaves like a short prefill: parallel, and cheap compared with producing the tokens one at a time. The guesses are accepted in order, each with a probability that depends on how much the large model agrees with it, until one is rejected. The rejected one is replaced by a token drawn from a corrected distribution. If every guess is accepted, the large model adds one more of its own. So each pass yields between one token and one more than the number of guesses.
The acceptance rule is a form of rejection sampling, and it guarantees that the output follows exactly the large model's distribution. You get the big model's answers, typically two to three times faster in the original papers, using arithmetic that would otherwise have sat idle.
That last part is the catch, and it is subtler than it looks. Checking four guesses for every sequence multiplies the arithmetic a step needs by five. With short contexts and a full batch, that pushes the step past what the chip can do, and speculation buys little. With long contexts it is the other way round. When most of a step is reading cache, checking five tokens reads each sequence's cache once instead of five times, and speculation helps even on a busy server. MagicDec (2024), which pairs this with a draft model that keeps only a small slice of the cache, reported speed-ups of up to two and a half times on Llama 3.1 8B at batch sizes from 32 to 256, once sequences were long enough.
Either way, it does best when the text is predictable: code, structured output, and anything that copies from its input. That last case has a particularly cheap version. Prompt lookup decoding drafts tokens by finding matching sequences in the prompt itself, with no draft model at all, which suits edits to a file that is already in context.
The shape of your output is a performance decision.
What This Means for Agents
An agent is the workload that makes all of this matter at once. It is a long conversation the model has with itself and its tools, with the context growing at every step.
Start with how that growth adds up. Say an agent adds 2,000 tokens to its context at each step, for 50 steps. Its final context is 100,000 tokens. But every step re-sends the whole context so far, so across the run the model is handed about 2.5 million input tokens to produce that 100,000-token history.
Prefix caching rescues most of that, if the context only ever grows at the end. Then each step recomputes only what is new, and the rest is read from cache. But cached tokens are cheaper, not free. At a tenth of the price, the nearly 2.5 million cached tokens alone cost about two and a half times as much as the 100,000 tokens that were ever new. Without caching, you pay full price for all 2.5 million: twenty-five times the history itself, and about seven times the cached run. Either way, the total grows with the square of the number of steps, not in proportion to it. Appending is cheap per step. Over a long run, it is quadratic.
The same square shows up inside the GPU. Every decode step reads the whole cache, and the cache grows every step. A 100,000-token context on Llama 3.1 8B is about 13 gigabytes of cache, almost as large as the model. On its own, with nothing else on the GPU, every output token then reads 29 gigabytes instead of 16, and the ceiling drops from about 210 tokens per second to about 115. The agent's memory has become nearly as expensive as the model.
That is the real argument for compaction, and it is a trade with a price on both sides. Summarizing the history shrinks every future step and resets the square. But a rewritten context no longer matches its own cached prefix. The next step pays full prefill on everything after the first token that changed. The instructions and tools at the top stay warm. Everything below them is cold.
So compaction should happen rarely and deliberately, at a point where the saving on all the steps still to come clearly outweighs one cold start. It should not happen continuously or by accident. The accidental version is common and invisible. A framework that re-renders the conversation slightly differently each turn breaks the prefix just as surely as a summary does. So does a chat template that drops earlier reasoning, or generated text that tokenizes differently when it is sent back. All of these make an append-only context look append-only while silently paying for a cold one.
Fan-out has a trap of its own. Five sub-agents that start from the same long context should share one prefill. But on at least one major provider, a cached prefix only becomes available once the first response has started, and caches can live on particular machines. Launch all five at the same instant and each may pay the full prefill. Send one request first to warm the cache, then fan out. Even when the prefix blocks are physically shared, each sequence still reads them during attention. Kernels like Hydragen and FlashInfer's cascade attention exist to read a shared prefix once for the whole group.
Output is the other half. Each output token is a sequential step that reads the model and the whole cache. So the cheapest output is the output the model does not write. An edit instead of a rewritten file. A reference to a document instead of a restatement of it, expanded afterwards by ordinary code. Structured output that is short and predictable, which is also the kind speculation accelerates best, if you serve the model yourself or your provider speculates for you.
And small models are fast because each step reads fewer bytes: fewer weights, and a smaller cache per token. That is why the narrow steps of an agent, like routing, classification and extraction, are where a small model changes the latency budget.
The Unit That Matters
The token is a unit of accounting. It is a perfectly good way to send an invoice.
But the GPU does not experience tokens. It experiences bytes, held and moved, and arithmetic done or left idle while it waits. Almost every serving technique of the last few years is a trade in memory. Continuous batching and speculative decoding make each byte read do more work. Grouped-query attention and quantization shrink the bytes. PagedAttention stops wasting the bytes you hold. Prefix caching spends held bytes to avoid redoing arithmetic.
Systems programmers went through this shift when the memory wall arrived, and they stopped counting instructions and started counting cache misses. Agent builders are about to go through the same one. The question stops being how many tokens something uses. It becomes what it makes the hardware hold, what it makes the hardware read, and how much of that can be shared.
Count your tokens, by all means. Then price your design in bytes: the ones it holds, and the ones it moves.