Build an AI Test-Prep Question Generator With Ollama

Most practice-question banks die the same death: after three weeks of drilling, you stop solving the problems and start recognizing them. Question 47 "feels" like answer C. That is not math skill — that is pattern memory, and it collapses the moment the real exam shuffles the wording.

A large language model flips the economics. Instead of drawing from a fixed list of 200 questions, you can generate a fresh set every night, tuned to the exact topic a student is weak on. Run the model locally with Ollama and the whole thing costs nothing per question, works offline, and never sends a child's learning data to anyone's cloud. The catch — and it is a real one — is that language models are probability machines, not graders. They will happily write a brilliant question where the correct answer is not among the choices. The pipeline below is about generating freely and verifying ruthlessly.

Why question banks go stale

A printed workbook is a map with a fixed number of roads. Study it long enough and you memorize the map instead of learning to drive. Research on practice testing is consistent on the mechanism that makes drilling work: the act of retrieving an answer strengthens memory. But retrieval practice stops working the moment retrieval becomes recognition — when the question, not the concept, triggers the answer.

Finite banks also cap difficulty targeting. A book gives everyone the same chapter test. A generator can produce twenty questions on nothing but quadratic equations, at exactly the difficulty a student failed at yesterday.

That is the promise. Now the engineering reality.

The local stack

Everything here runs on one ordinary machine — the same laptop a student already owns:

The privacy point is not decoration. Study behavior is sensitive data — it reveals learning difficulties, pace, and gaps. Local-first means the only copy of that record is on the family's own disk.

Teach by example: in-context learning

The single biggest quality jump comes from showing the model what a good question looks like instead of describing it. Put three or four real, high-quality exam questions — with their correct answers — directly into the prompt. The model imitates tone, difficulty, and formatting far more faithfully than it follows adjectives like "SAT-style."

There is a trade-off. Feed it too many examples and the model gets lazy: it starts producing near-copies of the examples instead of new scenarios. Three to five examples is the sweet spot — enough to define quality, not enough to fence it in.

The three ways AI quiz generation fails

This is where most DIY attempts fall apart. A language model predicts likely text; it does not solve the math and then types up the solution. That single fact produces three characteristic failures:

The pipeline, step by step

1. Generate with a local model

Pull a model and start generating — no API keys, no per-token billing:

# One-time setup
ollama pull qwen2.5:7b

# Generate one question as strict JSON
curl http://localhost:11434/api/chat -d '{
  "model": "qwen2.5:7b",
  "stream": false,
  "messages": [
    { "role": "system", "content": "You write SAT-style algebra questions. First solve the problem privately, then build four choices that include your computed answer. Vary difficulty across questions." },
    { "role": "user", "content": "Write one new question about quadratic equations." }
  ]
}'

2. Enforce the shape with a schema

Free-form output invites formatting chaos. Ollama supports structured output — hand it a JSON schema and the model is constrained to fill it:

{
  "type": "object",
  "properties": {
    "question": { "type": "string" },
    "choices": { "type": "array", "items": { "type": "string" }, "minItems": 4, "maxItems": 4 },
    "answerIndex": { "type": "integer", "minimum": 0, "maximum": 3 },
    "explanation": { "type": "string" },
    "topic": { "type": "string" }
  },
  "required": ["question", "choices", "answerIndex", "explanation", "topic"]
}

Pass it by adding "format": { ...schema... } to the request body. The schema kills an entire class of bugs — arrays of the wrong length, missing fields, answer index 7 on a four-choice question.

3. Validate and retry

The schema cannot check meaning. That needs a few lines of code:

type Question = {
  question: string;
  choices: string[];
  answerIndex: number;
  explanation: string;
  topic: string;
};

function validate(q: Question): string | null {
  if (new Set(q.choices.map(c => c.trim().toLowerCase())).size !== 4)
    return "duplicate choices";
  if (!q.explanation.toLowerCase().includes(q.choices[q.answerIndex].toLowerCase().slice(0, 8)))
    return "explanation does not reference the keyed answer";
  return null; // valid
}

async function generateValidated(topic: string, tries = 3): Promise<Question> {
  for (let i = 0; i < tries; i++) {
    const q = await askModel(topic); // structured-output call from step 2
    const err = validate(q);
    if (!err) return q;
    console.warn(`attempt ${i + 1} rejected: ${err}`);
  }
  throw new Error(`no valid question for ${topic} after ${tries} tries`);
}

The retry loop is the heart of the system. A local model costs nothing per attempt, so rejecting the worst 10–20% of output is free quality control. You are not fixing bad questions — you are refusing to ship them.

Automate the nightly drill sheet

The pipeline earns its keep when nobody has to touch it. A crontab entry or an n8n scheduled workflow generates a fresh sheet while everyone sleeps:

# docker-compose for a always-on question service
services:
  ollama:
    image: ollama/ollama
    volumes: ["ollama:/root/.ollama"]
  generator:
    build: ./generator
    environment:
      - MODEL=qwen2.5:7b
      - TOPICS=quadratic-equations,linear-functions,word-problems
    depends_on: [ollama]
volumes:
  ollama:

Point a small script at it from cron (0 5 * * *), render the accepted questions to Markdown or PDF, and every morning there is a ten-question drill targeting yesterday's weak topics. In n8n, the same flow is a Schedule Trigger → HTTP Request to Ollama → a Function node running the validator → a file write — no code beyond the validation function.

Multilingual explanations, private by default

One genuinely underrated capability: the same local model that writes the question can rewrite just the explanation in Thai, Spanish, or Vietnamese at the student's request. The question stays in English — as the exam demands — while the reasoning behind the answer arrives in the language the student actually thinks in. Because everything runs locally, there is no vendor deciding what languages deserve support, and no record of a struggling student's requests sitting on a distant server.

Where this goes next

Three upgrades are worth the effort once the basic pipeline is boring:

• Topic restriction — constrain generation to one skill at a time so a student can drill a weakness to mastery instead of sampling everything.
• Multimodal geometry — pair text with generated diagrams so "find the area of the shaded region" actually shows a shaded region.
• Difficulty tracking — log which generated questions a student missed and feed that back as difficulty steering in the prompt.

Final thoughts

An AI question generator does not replace teaching; it removes the scarcity from practice. The workbook's two hundred questions become two thousand, aimed at exactly the wrong answers of yesterday. The reliable versions of these systems all share the same shape: a generous generator, a skeptical validator, and a loop between them that keeps every flawed question safely behind the curtain. Build it once, run it locally, and the drill sheet refreshes itself forever — no cloud, no subscription, no ceiling on practice.