A large context window makes more input possible. It does not tell an application which input deserves to be there. A prompt can fit within the limit and still bury the evidence that matters beneath stale history, repeated passages, and verbose tool output.
Context design begins with deciding what the next step needs to know.
A research result worth testing against
Lost in the Middle, first submitted on July 6, 2023, studied multi-document question answering and key-value retrieval. The authors found that the position of relevant information could substantially affect model performance: the tested models often handled evidence at the beginning or end better than evidence in the middle. The paper also found degradation as contexts grew longer. These results concern the models and tasks studied; they are not a measured claim about every current model. Read the paper.
The engineering takeaway is a test, not a universal prompt-ordering rule: move the same evidence to different positions and check whether your own task remains reliable.
Make the budget explicit
Divide a request into instructions, user input, conversation state, retrieved evidence, tool results, and reserved output. Count tokens with the tokenizer appropriate to the deployed model. Character counts are useful for rough diagnostics but should not enforce a hard token limit.
A budget worksheet can express the decision without assuming a vendor's current pricing or context limit:
available evidence budget = total context allowance - instruction tokens - user and conversation tokens - tool schema and result tokens - reserved output tokens - safety margin
Check the selected API's counting rules: some models also account for reasoning tokens or other internal output categories. Keep the configured limit separate from the model's theoretical maximum.
Remove repetition before removing evidence
A useful packing pipeline groups chunks by source, removes exact duplicates, and preserves source IDs. Avoid sending the same paragraph three times because overlapping chunks all ranked highly. Keep document dates and section headings when they are needed to interpret the text.
For tool results, choose fields the next step actually uses. A model comparing order status may need an order ID, state, and timestamp; it probably does not need an entire internal audit payload. Preserve authoritative records in storage rather than squeezing every detail into the conversation.
Summarization is another tradeoff. A summary can omit a qualification that matters later. When a decision depends on precise wording, retrieve the original passage and cite it instead of treating the summary as the source of truth.
Compare compact and expanded contexts
Run the same questions with a compact evidence set and a larger one. Measure task success, unsupported statements, latency, and token consumption. Include cases with contradictory versions and evidence placed in the middle.
If the expanded context improves the difficult cases enough to justify its cost, keep it. If it only adds repetition, prefer the compact version. The goal is sufficient, traceable evidence for the next decision—not filling the window because space is available.