Honest AI Research: Hunt Conflicts, Keep the Disagreements

Ask an AI to research a topic and it hands you a beautiful summary in ninety seconds: confident prose, tidy structure, a conclusion at the end. The problem is that the polish is free — the model would produce it whether or not the underlying evidence was any good. A clean summary of a muddy evidence base isn't research; it's a rhetorical upgrade applied to guesses.

Real research has three habits the default prompt never exercises: it decides what counts as evidence before searching, it deliberately hunts for the other side, and it refuses to dissolve disagreement into a smooth average. Here's how to build those habits into an AI-assisted process — and into the automated research pipelines that increasingly write our first drafts for us.

Habit one: define evidence before you search

The most common failure in AI-assisted research is silent category confusion: an opinion presented with the confidence of a fact. Ask "what are the benefits of microservices?" and you'll get a fluent list — but much of it is practitioners' preference wearing the costume of evidence.

The fix is to classify the question before researching it:

Run the classification pass first, as an explicit prompt:

Pre-research pass:
"Classify each research sub-question below as EMPIRICAL (settle by
data), JUDGMENT (expert opinion, report attributed), or VALUES
(no right answer, flag as preference). For EMIRICAL questions,
state what evidence would settle them. Do not research yet."

Sub-questions:
1. Does adoption of X reduce cost?
2. Do teams prefer X?
3. Is X the right direction for our company?

That last line of output — what evidence would settle this — is the standard the rest of the research gets measured against. Anything the sources can't meet gets reported as unresolved, not smoothed over.

Habit two: search against yourself

A prompt is an argument for one side. "Why is Strategy A better than B?" will return a well-organized case for A, because you asked the model to build one. Balanced research requires running the question both ways — and, more importantly, hunting for the material that never makes it into success stories.

Failures are systematically under-published. The teams that tried an approach and quietly abandoned it don't write blog posts; the survivors do. So adversarial search isn't paranoia, it's correcting a sampling bias:

Adversarial pass:
"Search for evidence AGAINST [approach]:
1. Case studies where it failed or was abandoned — and WHY.
2. Critics' strongest specific objections (not strawmen).
3. What conditions made the successes work that a copycat
   might not have.
Return each with its source and what would falsify it."

The failure-case harvest is the highest-value step. Knowing that a method fails badly under condition Z — and that three companies discovered this silently — is worth more than ten glowing case studies, because you can actually check whether you have condition Z.

Habit three: keep the disagreement, name its type

Left alone, a model merges conflicting sources into one moderate sentence — "results are mixed" — which is the research equivalent of a shrug. Two very different situations get flattened into the same blur:

The distinction matters because each type has a different reader action: methodological conflicts mean "recompute with consistent definitions"; fundamental conflicts mean "this is genuinely contested — judgment required"; data-quality conflicts mean "one of these is just wrong." Blending all three into "mixed results" robs the reader of the only actionable part.

The extraction prompt keeps the structure honest:

{
  "task": "For each major claim, return",
  "fields": {
    "claim": "the specific assertion",
    "sources_for": ["who says it, on what basis"],
    "sources_against": ["who disputes it, on what basis"],
    "conflict_type": "methodological | fundamental | data_quality | none",
    "resolution": "what would settle it, or 'unresolved'"
  },
  "rule": "Never average conflicting sources into one sentence."
}

Structured output does the guarding automatically — the schema simply has no field for a smooth blend.

The assembled research loop

Put together, robust AI research runs as a loop, not a chat:

1. Scope — classify questions; define the evidence bar for each.
2. Multi-angle search — the case for, the case against, the failure cases.
3. Conflict extraction — claims table with the schema above; disagreements typed, not averaged.
4. Honest write-up — "known, debated, unresolved" as three explicit sections, with uncertainty surfaced rather than edited out.

And because the loop is just prompts and structure, it automates cleanly — a local model makes the whole pipeline private for sensitive topics:

# research agent: adversarial loop on local models
topic: "{{ input.question }}"
steps:
  - classify: { model: qwen25:7b, prompt: classify.md }      # fact/judgment/values
  - pro:      { model: qwen2.5:7b, prompt: case_for.md, web: true }
  - con:      { model: qwen2.5:7b, prompt: case_against.md, web: true }
  - failures: { model: qwen2.5:7b, prompt: failure_cases.md, web: true }
  - synthesize:
      model: qwen2.5:7b
      prompt: claims_table.md                                  # conflict schema above
      guard: "reject any output that averages conflicting sources"
  - report:   { render: known_debated_unresolved.md }

Every automated research digest should carry that guard line. A pipeline left to its defaults will produce exactly the confident smooth summary this whole method exists to prevent — and it will produce it every morning, on schedule, to your whole team.

The review habit before anything ships

Before research reaches a reader, run one last audit against the write-up: Where does the evidence actually fall short, and does the document say so? If a reader with domain expertise would object to any sentence, that sentence needs a source, a qualifier, or deletion. A research deliverable that never says "we don't know" on anything is telling you something — and it isn't that the researchers knew everything.

Final thoughts

AI made the finding part of research nearly free, which means the value moved entirely to judgment: what counts as evidence, where the counter-case hides, and which disagreements are real. Define the evidence bar first. Search against yourself, and dig for the failures nobody publishes. And when two sources clash, say so precisely — method, judgment, or bad data — instead of averaging them into mush. That's the difference between a summary that reads well and research you can actually act on.