Stop AI From Citing Bad Sources: The Source Rules Brief

You ask an AI for market statistics and get back a beautiful report with twelve citations. Solid, right? Then you click one. It's a blog post. That blog post cites another blog post. That one cites a press release. The press release says "according to industry research" with no link at all. Your "twelve sources" turn out to be one unnamed claim wearing twelve costumes.

This is the hit-rate problem with AI research: the model is rewarded for looking thorough, not for being right. If three listicles and one government report mention the same keyword, the model may treat all four as equal evidence. Present that to a client or a supervisor, and the one link they click will be the bad one. It's a statistical certainty — bad links outnumber good ones in these reports.

The fix isn't a better model. It's a rulebook. Before the AI searches for anything, you hand it a source rules brief: a ranked list of what counts as evidence, what's banned, and a required output format where every claim carries its tier, date, and deep link — or an honest UNSUPPORTED tag. This post shows the full brief, plus a five-pass audit for research you already have, runnable locally so your notes stay on your machine.

Why the Model Can't Judge Sources on Its Own

When a search tool fetches results, it matches your query against pages. Authority isn't part of that match. A 2019 statistic repeated on 400 SEO pages has more "presence" than a 2025 revision published once on an agency site — so the old number gets presented as consensus.

Understanding this changes what you blame. The model isn't lying; it's counting. Repetition masquerades as agreement, and marketing pages are optimized for exactly the keywords you're searching. Without rules from you, "popular" and "authoritative" look identical.

The Source Rules Brief

Send this before any research request. It has three parts: evidence tiers, exclusions, and an output contract.

Part 1: Evidence tiers

Define what you'll accept, best to worst:

Part 2: Exclusions

Name the junk explicitly, because it will otherwise dominate:

• SEO blog posts and listicles
• Content marketing pages and press releases styled as "findings"
• AI-generated summaries that don't link an original
• Anything without a named author or publication date

Part 3: The output contract

Require a fixed record per claim:

{
  "claim": "Global widget market grew 12% in 2024",
  "tier": 2,
  "source_name": "Official trade statistics agency",
  "deep_link": "https://example.gov/2024-widget-report#growth",
  "published": "2025-03-01",
  "confidence": "high"
}

The deep link matters most. A link to a homepage where you still have to hunt is not a citation; it's a bookmark. Require the URL to land on the exact page — ideally the exact section — containing the claim.

And the rule that does the real work:

If no source meeting Tier 1-3 criteria exists for a claim,
output the claim with status UNSUPPORTED and no citation.
Do NOT substitute a blog post, estimate, or "industry consensus".

That single instruction is what stops the model from pleasing you with junk. UNSUPPORTED in a draft is annoying. A fake citation in a client deck is a fire.

Running Research With the Brief

In practice the whole thing is one paste before your question:

SOURCE RULES:
Tiers: 1 peer-reviewed/raw data; 2 gov/regulator/central bank;
3 named-author established news; 4 company docs (self-claims only).
Banned: SEO blogs, listicles, press releases, unlinked AI summaries.
Every claim must include: tier, source, deep link, publish date.
No qualifying source = mark UNSUPPORTED. Never guess a citation.
QUESTION: <your research question>

Now compare what comes back. Before the brief: twelve citations, half dead ends. After: six claims, each with a working deep link, plus two honest UNSUPPORTED flags telling you where the primary data simply doesn't exist.

The Five-Pass Audit for Existing Research

Already sitting on a pile of AI-generated citations? Audit it with five targeted passes — each pass hunts one failure mode:

1. Citation grading — re-check every source against the tiers; discard anything below your floor.
2. Primary-source tracing — follow each claim back through its chain. Blog → blog → press release → nothing is a broken chain; find where it ends.
3. Disagreement hunting — ask for credible sources that contradict your conclusion, not support it.
4. Stale-data checks — for every figure: has it been revised, superseded, or withdrawn since publication?
5. Conflict-of-interest flags — who funded the source? A vendor-funded "study" is an ad with a graph.

The audit prompts are plain text, so they run anywhere — including locally:

ollama run llama3.1 "You are a research auditor. For each claim below,
trace the citation chain to its primary source. Report: final link in
chain, whether it is Tier 1-3, and whether the chain breaks.
CLAIMS: $(cat research-notes.md)"

Running the audit on your own machine matters when the notes include client material or unpublished strategy docs — nothing sensitive leaves the building.

Confidence Ratings for the Final Draft

Before anything goes into a presentation, force one last labeling pass:

Rewrite my findings with a confidence rating per claim:
HIGH = Tier 1-2 source, dated within 24 months
MEDIUM = Tier 3, well-reported but no primary data
LOW = poorly sourced, dated, or potentially superseded

Now you know exactly which numbers you can say out loud in a meeting and which need a hedge — before someone else finds out for you.

Frequently Asked Questions

Why does AI keep citing outdated statistics?

If many sites repeat an old figure, the model reads that repetition as consensus, even if the original study has been revised or withdrawn. Date requirements in the source rules brief, plus a stale-data audit pass, catch most of these.

What counts as a deep link?

A URL that lands directly on the page — ideally the section — containing the claim. A link to an outlet's homepage, or to a PDF index you must search yourself, fails the test. If you can't click it and see the number in two seconds, it isn't a citation.

How do I stop the model from inventing sources entirely?

The UNSUPPORTED rule. Models fabricate citations to satisfy a request that demands citations; giving them an explicit, permitted exit — "no source exists" — removes the pressure to please. You'll get more honest gaps and zero fake links.

Do smaller local models handle the audit passes?

Yes. Tracing chains and grading against a fixed tier list are comprehension tasks, not creative ones. Mid-size local models do them well, which is convenient since research notes are exactly the material you want off third-party servers.

Wrap-Up

AI research fails on sources, not on effort — the model will happily fetch forty links of which four matter. Hand it a source rules brief first: four evidence tiers, an explicit ban list, and a per-claim format with tier, date, and deep link. Demand UNSUPPORTED over filler, audit existing piles with the five passes, and label the final draft by confidence. The model stays fast; the evidence becomes something you can actually defend when someone clicks.