Deploying LLMs on AWS without the bill surprise

The surprise in an LLM bill is rarely a mystery line item. It is a GPU that stayed warm while nobody called it, a prompt that was sent again unchanged, or a batch job that shared a path with user traffic and forced you to keep the large shape running. Deploying a model on AWS without that surprise means designing serving around the traffic you have, and making token use and failures visible. This note is about the shape of that design. It is not a claim about a particular invoice.

Online and batch are different systems

A person waiting on a chat turn and a nightly job that rewrites a catalogue do not want the same machine. Put them on different paths. The online path scales with concurrent requests and cares about tail latency. The batch path can queue, use a different instance family, and stop when the queue is empty. If you mix them, the batch job either starves the chat path or forces you to keep capacity you only needed for the night.

Autoscaling has to be allowed to reach zero when the product allows it

Some products are quiet for long stretches. If the serving stack cannot scale in, you pay for idle accelerators. Scaling in has a cold start. You decide, with the product owner, whether a cold start is acceptable for that surface. A support agent that must answer immediately may keep a minimum warm capacity. A back-office tool that people open a few times a day may not. Write that decision down. It is a product requirement, not a default from the instance type.

Cache what does not change

Many prompts are stable: a system instruction, a retrieved policy that was asked an hour ago, a classification of a repeated ticket. A cache in front of the model, keyed by the exact prompt or by a hash of the inputs that matter, avoids a second inference. Cache only when a repeat is safe. Do not cache a tool result that depends on a record that just changed. Set a lifetime that matches how fast the underlying text changes. The cache is part of correctness, not only a discount.

Right-size the serving shape

A larger model needs more memory and often more than one GPU. A smaller model, or a model you adapted for a narrow task, may serve the same route on a smaller shape. Quantisation and batching of concurrent requests are tools, and each has a quality or latency tradeoff you should see in your eval set, not assume. Managed endpoints and self-hosted inference are both legitimate. Pick the one whose operational surface you can actually run: patching, scaling limits, and where logs go.

online  -> cache -> autoscale inference (min set by the product)
batch   -> queue  -> separate workers, scale to empty
observe -> errors, token counts, queue depth, cold starts
alarm   -> when token use or error rate leaves the band you chose

Observability is how the bill stays unsurprising

You want traces that tie a user turn to the model call, the token counts, the cache hit or miss, and the tool calls. You want the error rate and the queue depth. You want to know when a deploy changed the shape of traffic. None of this requires publishing a number in a blog post. It requires that an engineer can open a day and see why inference ran. Without that, cost control is a meeting. With it, cost control is a change you can make: a tighter prompt, a cache, a smaller route for an easy task, batch moved off the hot path.

Guardrails that also save money

• Reject or shorten inputs that are far outside the task. A novel pasted into a classifier is a bug and a bill.
• Cap output length in the API call when the product only needs a short answer.
• Do not retry a failed call in a tight loop. Retry with a bound, and surface the failure.
• Keep evaluation traffic off the production endpoint, or tag it so it is not mistaken for user load.

A deployment that does not surprise you is boring on purpose. Separate paths, scaling that matches the product, a cache with a real lifetime, and traces that show tokens and failures. The model weights are not the bill. The way you serve them is.

What we decide before the first instance

Before a GPU is requested we write down the traffic shape in words: interactive or batch, steady or spiky, allowed to go cold or not. We write down which prompts are safe to cache and which depend on a live record. We write down where logs and traces land, and who looks at token counts when they move. We pick a serving shape that fits the model we actually evaluated, not the largest shape on the menu. Managed and self-hosted are both acceptable if the team can operate the one we pick: scaling limits, how a bad deploy is rolled back, and how a stuck worker is noticed. Cost control is those decisions made early. A later surprise usually means one of them was left implicit: batch sharing the online fleet, a cache with no lifetime, retries with no bound, or a minimum capacity nobody agreed to. None of this requires a published saving. It requires that the serving design is as reviewable as the prompt. If you cannot point at the path a request takes, you cannot explain the bill, and you should not be surprised when it arrives.