What is it?
Each time you call a language model, everything it can use must be placed in that single request: the system instructions, your messages, earlier turns of the conversation, any pasted documents or tool results, and room for the answer. The context window is the upper limit on that total, measured in What are Tokens in AI?.
If the content exceeds the limit, something has to give: the request is rejected, the oldest messages are dropped, or you must shorten the input. The model cannot see anything outside the window.
Why does it matter?
It explains many everyday frustrations:
- "The AI forgot what I said earlier" — in a long chat the early messages may have been trimmed or summarised away.
- "It ignored part of my document" — the part may not have fit, or it was buried among too much text.
- "It's slow and expensive on long inputs" — more tokens means more processing.
It also shapes system design. Instead of stuffing everything into the window, developers retrieve only the relevant parts (What is RAG (Retrieval-Augmented Generation)?), store durable facts outside the model (Agent Memory: Short-Term, Long-Term and Beyond), and summarise old turns.
How does it work?
Chat products create the illusion of memory by re-sending the conversation with every new message. The model itself is stateless between requests.
Window sizes differ widely between models and change as new models are released, so check the current documentation for the model you use rather than relying on a remembered number.
Bigger is not automatically better. Research on long-context use (for example the paper "Lost in the Middle") has found that models can use information at the start and end of a long prompt more reliably than information in the middle, and long inputs add cost and latency. Quality depends on the model and the task, so measure it on your own material (How to Evaluate AI Systems and Outputs).
Example
A support assistant has a 100-page policy manual. Options:
- Paste the whole manual every time. Simple, but costly, slow, and it may not fit.
- Retrieve first. Index the manual (What are Embeddings in AI?, What is a Vector Database?), fetch the three most relevant sections for each question, and put only those in the prompt. This is What is RAG (Retrieval-Augmented Generation)?.
- Summarise history. Keep recent turns verbatim, compress older turns into a short summary.
Most production systems combine 2 and 3.
When should I use it?
Design around the window whenever inputs can grow: long chats, document analysis, code repositories, agents that accumulate tool outputs. Put the most important instructions and facts where they are easy to find (clearly labelled, not buried), remove irrelevant text, and measure token usage during testing.
Common mistakes
- Thinking the context window is memory that persists. It resets with each request.
- Filling it "because it fits". Extra noise can reduce accuracy and raises cost.
- Not reserving room for the reply. Long prompts can leave too little space for the answer.
- Assuming a large window removes the need for retrieval. Often it does not.