How to Budget an LLM Context Window
An LLM context window is a fixed number of tokens, and everything you send — the prompt, the tools, the retrieved context, the history — plus everything the model writes back has to fit inside it. This guide shows you how to budget that window deliberately: allocate the token budget across each line-item, reserve room for the output, and keep the growing parts from overflowing and truncating your run.
In plain words
Budgeting an LLM context window means treating the model's token limit as a fixed budget and deciding, on purpose, how much of it each part of the request may use — and PlayCISO has a free, browser-based visualizer that lays it out for you. You split the window across eight line-items: the system prompt, tool schemas, RAG retrieval, conversation history, scratchpad, few-shot examples, guardrails, and a reserved slice for the output.
Reserve the output tokens first, cap the parts that grow — RAG and history — and compress them before they overflow. Window sizes vary by model and change over time, so look up your own model's limit rather than assuming a number. When the total crosses the ceiling you get overflow: a rejected request, or silent truncation that quietly drops something load-bearing.
Step by step
- 1
Know your model's window — don't assume it
A context window is the hard ceiling on how many tokens the model can process in one call — everything you send in plus everything it generates back. The size varies by model and even by model version, and vendors change it over time, so never hard-code a number from memory. Look up the exact figure for the specific model you are calling and treat it as the fixed budget you are dividing up. If you support more than one model, budget against the smallest window you might route to.
- 2
Inventory everything that competes for the window
Eight things draw down the same budget on every call: the system prompt, the tool and function schemas, retrieved context (RAG), the conversation history, any scratchpad or chain-of-thought the model writes, few-shot examples, guardrail and formatting instructions, and the space reserved for the output. List each one and estimate its token cost. The ones people forget are tool schemas (large JSON definitions that ship on every request) and scratchpad, which grows silently as the model reasons.
- 3
Reserve output tokens first
Decide how long the response can be and carve those tokens out before you allocate anything else — the window has to hold the input and the output together, and the model cannot finish a reply it has no room to write. If you need a 2,000-token answer, 2,000 tokens are gone before the prompt is built. Setting an explicit max-output cap and subtracting it up front is what keeps a long prompt from leaving no room to respond, which shows up as a truncated or empty completion.
- 4
Budget RAG and history deliberately, not greedily
RAG retrieval and conversation history are the two line-items that expand without limit if you let them, so give each a fixed cap rather than "whatever fits." Cap the number and size of retrieved chunks and rank hard so the few most relevant passages win — more retrieved text is not more accuracy, and padding the window with marginal chunks both costs tokens and dilutes the signal. Give history its own ceiling too, so a long conversation cannot crowd out the retrieval and instructions the current turn actually needs.
- 5
Watch history growth before it overflows
The system prompt, tool schemas and few-shot examples are roughly fixed per call, but conversation history grows with every turn. On a long session it is the history that quietly pushes the total past the ceiling. When that happens you get overflow: either the request is rejected, or the harness silently drops the oldest messages to make room — and if the part that gets dropped was load-bearing (the original instruction, a key fact), the model appears to "forget" or go off the rails. Track the running total per turn so you see the ceiling coming instead of hitting it.
- 6
Trim and compress to stay under the ceiling
When a line-item is growing, shrink it on purpose rather than letting truncation decide for you. Summarize older conversation history into a compact running summary and keep only the recent turns verbatim. Cap retrieved chunks and drop low-ranked ones. Prune tool schemas to the tools the current step can actually use, and trim few-shot examples to the few that earn their tokens. Controlled compression keeps the information you chose; blind truncation discards whatever happens to be at the edge.
- 7
Monitor cost as tokens scale
Tokens are billed, so the budget is a cost lever, not just a capacity limit. Every token in the window is paid for on every call, and because history and retrieved context ride along on each turn, cost grows super-linearly over a long session unless you cap them. Monitor tokens-per-call and tokens-per-session, watch the trend as usage scales, and revisit your allocation when the bill or the latency climbs — a window that fits but costs too much per turn is still a budget problem.
Enter your model’s window size and see the system prompt, tool schemas, RAG, history, scratchpad, few-shot, guardrails and output reserve laid out against it — so you can spot overflow before it truncates your run.
Related
The interactive allocation tool this guide walks you through — size each line-item against your window.
Recursion budget simulatorModel how recursive and multi-step agent loops consume tokens and depth before they blow the budget.
Multi-model agent cost calculatorEstimate token cost across models and steps so your context budget maps to a real bill.
Harness designerDesign the agent harness — prompts, tools and memory — that decides what lands in the window each turn.
Frequently asked questions
What actually eats up the context window?
Eight things share the same budget on every call: the system prompt, the tool and function schemas, retrieved context from RAG, the conversation history, any scratchpad or reasoning the model writes, few-shot examples, guardrail and formatting instructions, and the tokens reserved for the output. People usually account for the prompt and the history but forget the tool schemas, which can be large JSON that ships on every request, and the scratchpad, which grows as the model reasons.
What happens when the context window overflows?
One of two things, neither good. The request is rejected outright because it exceeds the model’s limit, or the harness silently truncates it — usually by dropping the oldest messages or clipping retrieved text to make room. Silent truncation is the dangerous case: if the dropped content was load-bearing, the model appears to forget instructions or facts and its answers degrade without any error. Reserving output tokens up front and capping the growing line-items is how you avoid both.
How much of the window should RAG get?
Give it a fixed cap rather than "as much as fits," and rank retrieved passages hard so only the most relevant ones make it in. More retrieved text is not more accuracy — padding the window with marginally relevant chunks spends tokens and dilutes the signal the model needs. A good default is to cap both the number of chunks and the total tokens they can occupy, then tune that cap against answer quality, not against how much you can cram in.
Does a bigger context window just fix the problem?
Not really. A larger window buys headroom, but every token in it is still billed on every call and still adds latency, so a bigger window often means a bigger bill and a slower response rather than a solved problem. Models can also attend less reliably to information buried in the middle of a very long context. Treat a large window as room to be deliberate with, not a license to stop budgeting — cap the growing line-items either way.
How does conversation history cause overflow?
The system prompt, tool schemas and few-shot examples are roughly fixed per call, but history accumulates with every turn. On a long session the history keeps growing until the total crosses the ceiling, at which point you overflow — the request is rejected or the oldest turns get dropped. Because the earliest messages often carry the original instruction, dropping them is what makes a model seem to lose the plot late in a conversation. Summarize older turns into a compact running summary and cap history so it cannot crowd out the current turn.
How do I keep a long agent run from truncating?
Budget before you run, and compress on purpose instead of letting truncation decide. Reserve the output tokens first, cap RAG and history with explicit ceilings, prune tool schemas to the tools the current step needs, trim few-shot examples to the few that earn their place, and roll older history into a summary. Track the running token total per turn so you see the ceiling approaching and compress ahead of it, rather than discovering the overflow after the model has already dropped something important.