Your Model Isn't Confused, It Can't See: Ollama Context Window Management

You paste a long document into your local model, ask about the last paragraph, and the model answers about the first one. Or you have a productive twenty-turn session, and around turn fifteen the model starts forgetting rules you set at turn one — apologizing for things it already did right, ignoring the format you agreed on. The diagnosis feels obvious: small model, gets confused.

Wrong diagnosis. In a local setup, the most common cause is much dumber and much more fixable: the model literally cannot see part of your conversation. The context window — the working memory the model reads on every message — has a hard size limit, and Ollama's default is smaller than most people realize. When your chat overflows it, the overflow doesn't error out. It silently falls off the edge, and the model keeps chatting, confidently, about the part it can still see.

Understanding your context window is the single highest-leverage piece of local-LLM knowledge: it explains most "the model got confused" moments, and managing it turns a flaky assistant back into a sharp one.

The Whiteboard Model

Picture the model's context as a whiteboard. Every message you send, the model walks up to the whiteboard, reads everything on it, and writes a reply. Nothing is remembered between messages — there is only the board.

• Small window = small board. Fit more on it than it holds, and the oldest writing disappears — but nobody tells you it disappeared.
• Every token competes. System prompt, persona, pasted documents, the last fifteen exchanges, today's question: all of it shares one board.
• Small models have small boards and degrade earlier. A 7B model with a long board starts losing the thread before the board is technically full — attention thins out long before the hard limit.

This is why the same prompt can work beautifully in a five-turn chat and fail in a thirty-turn one. The prompt didn't change; the board did.

The Trap: Ollama's Default Window

Here is the gotcha that bites almost everyone. Ollama's default context length is 2048 tokens — roughly 1,500 words of everything combined: system prompt, conversation, pasted text. Meanwhile the model you downloaded may support 8k, 32k, or 128k. The model can hold it; the server never sends it.

Symptoms of silent truncation:

• The model answers questions about the start of a long document (that's what fit in the window) while you ask about the end.
• Rules set early in a session quietly stop applying.
• Pasting a big file "works" — the model responds — but its answers only cover the first chunk.

The fix is one setting:

# per-session
ollama run llama3.2
>>> /set parameter num_ctx 8192

# or permanently in your Modelfile

FROM llama3.2
PARAMETER num_ctx 8192
SYSTEM """..."""

How big can you go? Check the model's card — and be honest about hardware, because bigger windows cost real memory:

Raising the ceiling is step one. Filling it responsibly is step two — because a big window used badly is just a big bill with the same confusion.

Session Hygiene: The Rolling Summary

Cloud guides give you the "50-turn rule": long threads go stale, start fresh. On local models the same rule arrives much earlier — call it the 15–20 turn rule — and the fix gets better than "start over." The local version is a rolling summary: before resetting, have the model compress the session, then carry the summary into the new one.

# end of a long session — ask for the handoff
"Summarize this session as a handoff note: decisions made, formats agreed,
open questions, key facts. Max 150 words."

# new session
"Session context from last time: {{summary}}. Continue from there."

You lose the noise — the dead ends, the repeated apologies, the full quotes — and keep the signal. The whiteboard gets erased, but the important notes get rewritten onto the fresh one in small writing. This pattern is also exactly what your n8n workflows should do for long-running jobs: when a conversation approaches its budget, summarize, reset, continue.

Knowledge Doesn't Belong on the Board

The second management move: stop pasting reference material into chats. Brand guidelines, SOPs, product specs, past reports — that's knowledge, and knowledge belongs in files the model reads on demand, not in window space it pays for on every single message.

The practical split:

For folders that grew past a handful of files, retrieval does the budgeting for you: embed the documents once, and at question time only the relevant chunks enter the window.

ollama pull nomic-embed-text   # embeddings, runs locally
# chunk docs -> embed -> store vectors -> retrieve top-k per question

A question about pricing pulls the pricing paragraphs, not the whole policy binder. The whiteboard stays clean, the answers get sharper, and a 500-document library works on hardware that could never hold 500 documents in context.

Budget the Board Like Money

The habit that ties it all together: before a long task, count roughly what will share the window.

Budget check for a 8k window:
- system persona + rules     ~400 tokens
- pasted source document     ~3,000 tokens
- conversation so far        ~2,000 tokens
- room for the answer        ~1,500 tokens
- slack                      ~1,100 tokens
= fits. If the document were 6k, it wouldn't — retrieve instead of paste.

Thirty seconds of arithmetic prevents the classic failure where you paste a huge file, ask a question, and get an answer about the wrong half of it. If it doesn't fit, don't paste — retrieve, summarize first, or split the task.

Make It Practical Tonight

• Set num_ctx explicitly everywhere — one session check (/show info displays it) beats every mystery "confusion" you've blamed on the model.
• Start one long-running task over with the rolling-summary pattern; feel the difference at turn five of the new session.
• Move your most-pasted document into a file that gets read on demand; retire the paste habit.
• If a task genuinely needs 32k of live context, let it — deliberately, with the memory cost visible, rather than by accident at 2048.

The model was never confused. It was reading a board with half the writing erased. Manage the board, and the assistant you bought the hardware for shows up.

Frequently Asked Questions

How do I check what context size my Ollama session is actually using? Run /show info inside a session — it reports the effective context length — or watch the server logs at startup, which print the loaded num_ctx. Both are worth checking once, because the default surprises people: the model card may advertise 128k while the server silently serves 2048.

Can I just always set num_ctx to the model's maximum? You can, but you'll pay for it twice: memory usage grows with the window even when the conversation is short, and very long contexts make small models measurably worse at following instructions, not better. Size the window to the task — 8k covers most daily work; go bigger when a real document, not a vague fear, demands it.

Does a rolling summary lose important details? Some, yes — that's the trade. The details it loses are usually the ones hurting you anyway: repeated corrections, abandoned approaches, verbose quotes. If a specific fact is load-bearing, name it in the summary prompt ("always keep the agreed output schema verbatim") or write it into the Modelfile SYSTEM where it never needs summarizing.

How is this different from just using RAG for everything? RAG solves the knowledge problem — stable material, retrieved on demand. It doesn't solve the conversation problem: decisions, agreements, and thread state that only exist in the chat. Healthy local setups use both: knowledge folders with retrieval for the stable stuff, session hygiene and rolling summaries for the living conversation, and the Modelfile for what should never leave.