Every Word Costs a Second: Token Optimization for Local LLM Workflows

Ask your assistant to fix a null-check bug and it answers with a paragraph of empathy, a plan, the code, a summary of the code, an apology for the previous inconvenience, and an offer to help with anything else. The fix was eleven characters. The packaging was two hundred tokens — and on a local model, you paid for every one of them twice: once in electricity, once in seconds of your life watching text scroll.

Cloud users noticed this first and coined "Caveman mode" — instructing the assistant to drop all conversational packaging and answer like a telegram: L42: added null guard. Done. Same logic, same code, no wrapping paper. On hosted APIs the payoff is a line item. On a local stack the payoff is physical: tokens per second are capped by your hardware, not a vendor's data center, so every generated filler word is latency you feel. Brevity isn't rudeness on a local machine — it's the difference between a snappy tool and a sluggish one.

And it goes further than style. Terse output is smaller in your context window, easier for scripts to parse, and — because long explanations give a model more surface area to wander — often more accurate. Here's the full economics, and how to wire it into a local stack so humans and automations both get exactly the amount of words they need.

Why Local Changes the Math

Every token has three prices. On the cloud, you mostly notice one:

The local column is why terse mode is a bigger deal at home. A 7B model on modest hardware might generate 20–40 tokens per second. A 200-token polite wrapper is five to ten seconds of pure ceremony, per reply, all day. Cut the wrapper and the same hardware feels twice as fast — no upgrade required.

Then the second price: as we covered in the context-window piece, everything the model generates stays on the board for the rest of the session. A chatty assistant burns its own memory on its own manners. Terse replies keep the window free for code, files, and the actual conversation.

Three Levels of Brief

Brevity isn't binary — match the compression to the task:

• Lite — full sentences, zero filler. For exploring unfamiliar code or learning something new. You still want a readable explanation; you don't want an essay. "Line 42 misses a null check because the API can return an empty object. Guard added."
• Full — fragments, articles dropped. The daily workhorse. "L42: added null guard. Also fixed same pattern L88."
• Ultra — notes only. For deep-flow sessions in a codebase you know cold, and for mechanical work: refactors, test scaffolding, renames. "L42 +guard. L88 +guard. tests pass."

One rule holds at every level: the reasoning doesn't change, only the narration. The model still analyzes the same logic. You're not dumbing it down; you're declining to have the analysis read aloud.

Wiring Terse Mode Into a Local Stack

On cloud tools this is a plugin. Locally it's a few lines you own. For interactive work, bake the voice into the Modelfile so every session starts terse:

FROM llama3.2
SYSTEM """
Answer in the fewest words that fully solve the task.
- No greetings, preambles, apologies, or offers of further help.
- No summary of what you just did, unless asked.
- Code and exact changes first. Explanation only on request, max 3 sentences.
- If uncertain, state the uncertainty in one line, not a paragraph.
"""

For finer control, keep three Modelfiles — editor-lite, editor, editor-ultra — and switch like gears:

ollama create editor-ultra -f Modfile.ultra
ollama run editor-ultra "add input validation to all form handlers"

Input Tokens Count Too

The same economics apply to what the model reads. That 4,000-word rules file riding in every request is input-token latency on every single call. Compress it the same way you compress output:

Before (4,100 tokens):  "When you are writing code for this project,
please make sure that you always remember to use the established
error handling patterns which are described in..."
After (900 tokens):     "Error handling: wrap fetch calls in try/catch,
map known codes to user messages, rethrow unknown. Never silent-pass."

Same rules, quarter of the cost, quarter of the read-time before every generation. Instruction files are machine food, not marketing copy — feed them compressed.

Machine-to-Machine Wants Telegraphese

The strongest case for terse mode isn't the human side at all. When an n8n workflow asks a local model to classify an email or draft a status line, conversational packaging isn't just waste — it's a parsing hazard. "I'd be happy to help! Here is the classification: ORDER" breaks the naive parser that expected ORDER.

Give automations an explicit output budget and format:

workflow: inbox-triage
nodes:
  - name: classify
    type: http-request
    url: "http://localhost:11434/api/generate"
    body:
      model: "llama3.2"
      stream: false
      prompt: |
        Classify: order|question|complaint|spam.
        Reply with the single label only. No punctuation,
        no sentence, no explanation.

{"model_reply": "order"}

One word in, one word out — fastest possible generation, trivially parseable, and the terseness that would feel rude in conversation is precisely the contract machines want from each other. Any automation you build after reading this should default to an output budget; verbose default voices are for humans who didn't configure anything.

The Hidden Bonus: Accuracy

The counterintuitive part of terse mode: constraints toward brevity often improve correctness. A long explanation gives the model more room to hedge, drift, and contradict itself mid-paragraph; a one-line answer commits. In eval terms, this is the "test the invariants, not the prose" principle — the label order is checkable, the paragraph containing it is arguable. When you strip narration, what's left has to be right.

This doesn't mean brief always wins. Use Lite mode when the goal is understanding — an explanation you'll learn from deserves sentences. Switch to Full and Ultra when the goal is progress. The mature habit is changing gears deliberately rather than living at one setting.

Make It Practical Tonight

• Add the terse SYSTEM block to your most-used Modelfile; feel the speed difference for one evening.
• Write the three-level gear set (lite/full/ultra) and label which tasks belong in which.
• Take your biggest standing instruction file and rewrite it at quarter length — same rules, machine food.
• Audit one automation's prompt: add an explicit output format and watch both the latency and the parser failures drop.

The polite wrapper was a solution to a human problem — making strangers comfortable with a new tool. You're not a stranger anymore, and half your model's colleagues are scripts. Talk to the machine the way the work needs: short, exact, and only when there's something to say.

Frequently Asked Questions

Does terse mode make the model less capable? No — the reasoning is identical; only the narration changes. The model still analyzes the same problem and writes the same code. If anything, brief answers are often more accurate, because there's less prose surface for hedging and drift. What you lose is hand-holding, which matters in Lite contexts (learning, exploring) and nowhere else — that's why brevity is a gear, not a permanent setting.

How much latency does this actually save locally? Roughly the fraction of each reply that was packaging. If polite wrappers are 60–70% of tokens and your hardware does 30 tokens/second, a 200-token reply drops from ~7 seconds to ~2–3. Across a working day of dozens of replies, terse mode returns real minutes — the same upgrade as a faster GPU, achieved with a prompt.

Won't ultra-terse output hurt collaboration or documentation? Not if the record lives somewhere better than chat. The fix is separation: terse in the conversation, durable in files — commit messages, changelog entries, decision logs written as files (by the model, on request, at normal length). The conversation is a workbench, not an archive; keep the archive out of the workbench's token budget.

What about vision or multimodal outputs — same rules? Same economics, sharper trade-off: images and long documents are token-heavy inputs, so the compression instinct matters even more there. Summarize long documents once into terse notes, then work from the notes instead of re-reading the full document every turn — the rolling-summary pattern from context management, applied for speed instead of memory.