Prompting Small Local LLMs: The Rules That Actually Matter

You spent a year getting good at prompting the big cloud models. Your prompts have a persona, a step-by-step reasoning instruction, seven formatting rules, three examples, and a sign-off flourish. Then you paste the whole thing into llama3.2 running on your own machine — and the output follows two of the seven rules, invents a third, and ignores your examples entirely.

The prompt didn't get worse. The model got smaller. Small local models don't fail loudly like a server erroring out; they fail politely, producing confident, plausible text that quietly missed half the brief. And almost every prompting habit you built for frontier models makes it worse, because those habits assume an attention budget a 3B–8B model simply doesn't have.

The good news: prompting small models well is easier, not harder — you just need different rules. Five of them cover nearly everything.

Why the Model Size Changes the Prompting Math

Three things shrink with the model:

• Attention budget. Every rule in your prompt competes for a smaller model's limited capacity to track instructions. Frontier models juggle seven constraints; a 7B model starts dropping them around three.
• Sycophancy gets louder. Small models mirror your framing even harder than big ones. Ask a leading question and you haven't asked a question — you've placed an order.
• Hallucination rate rises. Less world knowledge means more confident gap-filling. The smaller the brain, the more creative the fill.

None of these make small models useless — they make small models honest about prompt quality. A prompt that survives a 7B model is a genuinely clear prompt.

Rule 1: The Constraint Budget Is Two, Not Seven

The famous "rule of three" — models degrade past three simultaneous constraints — is optimistic for local models. Budget two per pass, then split the job:

PASS 1 — the core job, minimal rules
"Rewrite this customer reply in a warm, plain tone. Keep every fact."

PASS 2 — format pass on the draft
"Take the previous draft. Cut it under 60 words. Remove all exclamation marks."

Each pass has one job and one or two constraints. The model can't forget rule five if there is no rule five. This costs a second generation — which, on local hardware, costs you nothing but a few seconds.

The anti-pattern it replaces:

# the frontier-model habit that kills small models
"Act as a senior editor. Rewrite warmly, under 60 words, no exclamation
marks, keep all facts, 5th-grade vocabulary, add a call to action,
no jargon, mirror my voice, and format as three short paragraphs."

A 7B model reading that produces something confident that satisfies maybe four of those nine instructions — and you can't tell which four without checking all nine.

Rule 2: Neutral Phrasing, Because Small Models Parrot Harder

Ask "buying is better than renting, right?" and a big model hesitates; a small one agrees enthusiastically. Your framing doesn't just bias small models — it frequently becomes their answer.

The fix is mechanical: strip every adjective that carries your hoped-for conclusion out of the question. On a small model, neutral phrasing isn't politeness — it's calibration.

Rule 3: The Persona Lives in the Modelfile, Not the Prompt

Roles work on small models — often better than elaborate reasoning instructions, which smaller models skim. But re-typing "act as a reliability engineer" into every chat is the cloud habit. Locally, the persona is a file you write once:

FROM llama3.2

SYSTEM """
You are a plain-spoken editor for a technical blog.
Voice: short sentences, concrete nouns, no hype words.
Always: keep every technical fact, cut filler, prefer examples over adjectives.
Never: apologize, hedge, or invent sources.
"""

ollama create editor -f Modelfile
ollama run editor "revise this paragraph"

Now every generation — including the ones your n8n workflows fire at 3 a.m. — starts pre-persona'd, pre-toned, pre-constrained. The prompt you type shrinks to the actual task, and the model's remaining attention budget goes to the work instead of the framing.

Rule 4: Describe the Goal, Don't Paste Examples

Few-shot examples are a frontier-model luxury. On a small local model, three examples eat context the model needed for the task itself, and the output tends to blur the examples together instead of extracting the pattern. Describe the destination instead:

# instead of three example tweets, the goal:
"Write a tweet announcing our new local-AI report.
Goal: make a technical reader curious enough to click.
Constraints: one concrete number, no hashtags, under 200 characters."

What the goal description loses in imitation, it gains in clarity — and clarity is the currency small models trade in. If you truly need style consistency across many outputs, that's what the Modelfile SYSTEM is for (Rule 3), not pasted examples.

Rule 5: Confidence Is Noise — Verify Structurally

Small models hallucinate more, and they hallucinate fluently. "Never trust a confident fact" doubles as "never trust a confident small-model fact, twice." The local stack's answer is structural: you don't have to eyeball everything, because you can assert:

# a five-minute safety net (promptfoo against your local model)
prompts:
  - "Summarize this week's client metrics: {{metrics}}"
providers:
  - openai:gpt-4o-mini:
      apiBaseUrl: http://localhost:11434/v1
      apiKey: ollama
      model: llama3.2
tests:
  - vars: { metrics: "calls: 42, emails: 118, meetings: 6" }
    assert:
      - type: contains, value: "42"        # numbers must survive
      - type: max-length, value: 400
      - type: llm-rubric
        value: "mentions no numbers that weren't provided"

Numbers must come from the input; length must hold; invented statistics get flagged by the rubric judge. The principle from the eval world applies double here: smaller model, more assertions. What you can check mechanically, you stop worrying about.

The Rules Side by Side

Prove the Rules on Your Own Machine

The quiet advantage of learning prompting on local models: experiments are free and private. Take any rule above, write the bad version and the good version, and run both ten times through the same model. The difference that cloud guides describe abstractly — "models degrade past three constraints" — shows up in your own outputs, in your own ten samples, with your own task. Twenty minutes of that beats a hundred prompting articles, and the habit you build (test the prompt, don't trust the tip) transfers straight back to every cloud model you'll ever use.

Make It Practical Tonight

• Take your longest working prompt and split it into two passes: core job, then format. Run both on your local model.
• Write one Modelfile for your most-repeated task's persona; retire the "act as..." line from your prompts forever.
• Delete the examples from one prompt and replace them with a three-line goal description. Compare.
• If a local output states a number, build the one assertion that catches it being invented.

Small models don't reward clever prompting — they reward clear thinking, then they hold up a mirror to whatever's left. That mirror is the training tool frontier models never gave you.

Frequently Asked Questions

Do these rules also apply to bigger local models like 14B or 30B? They relax rather than reverse. A 14B model handles three or four constraints reliably and benefits less from persona-baking — but neutral phrasing, goal-over-examples, and mechanical verification hold at every size, because they address how all models are built, not how big they are. The constraint budget is the only rule that scales with parameters.

Isn't splitting into passes slower than one big prompt? On cloud APIs, maybe — you're paying per call. Locally, a second pass costs seconds of compute you already own, and the first-pass quality jump usually saves a regeneration cycle anyway. Splitting feels slower and finishes sooner.

What if I genuinely need few-shot examples for style? Put the style in the Modelfile SYSTEM (Rule 3) as described voice traits, not pasted samples. If you absolutely must show examples, use exactly one — the shortest that carries the pattern — and keep the goal description alongside it. One example plus a goal reads as illustration; three examples reads as a blur template for a small model.

How do I know if my model is even following my constraints? Count them mechanically: take your prompt's constraints, check the output for each, and score pass/fail. Do that across five generations and you have a constraint-following rate — the number that tells you whether you're over budget. When the rate drops below ~80%, you've found your model's real limit, which is more useful than any published rule of thumb.