Building the Scientific Question Compiler: A Map of What We Do Not Know

Building the Scientific Question Compiler: A Map of What We Do Not Know

By Daniel Kliewer · 2026-07-23

Yesterday I wrote the roadmap: the four "research directions" for the Scientific State Compiler are not four projects, they are five passes in one architecture — Evidence → Questions → Mechanisms → State → Discovery. The roadmap's central question was which pass is the missing bottleneck? My answer was concrete: Pass 2, Questions — the lowest-risk, highest-leverage increment, the one that turns a corpus into a map of what we do not know.

Today I built it. The Scientific Question Compiler takes a folder of papers and compiles them into a structured map of scientific uncertainty: typed, ranked, dependency-linked unanswered questions, surrounded by the known / contested / silence landscape they emerge from. It runs entirely on our own hardware against a 35-billion-parameter model. It is live here, the code is on GitHub, and this post is the honest account — including the bug that taught me something.

What it does

A corpus of markdown papers flows through a pass chain. Each pass writes a typed intermediate representation:

ingest → claims → contradictions → gaps (Unknown Detector) → QUESTION → assemble

• claims — pulls typed scientific claims (population / intervention / outcome / confidence) from each paper.
• contradictions — finds where claims genuinely disagree.
• gaps — the Unknown Detector: what the corpus implies but does not state, plus a silence map of topics conspicuously absent.
• QUESTION — the core pass. It promotes gaps + contradictions + silences into structured Question IRs, classifies each (mechanistic / causal / boundary / optimization / translational), scores it on five axes (importance, feasibility, knowledge gap, scientific leverage, evidence density), ranks by a weighted composite, and links questions by dependency. The output is an uncertainty graph, not a summary.

I compiled a small corpus on Alzheimer's disease and metabolic dysfunction — the example domain from the roadmap. The compiler found, among others, this highest- ranked question:

q1 (high): Is BBB insulin-transport failure a cause or a consequence of neuronal insulin resistance in early Alzheimer's disease?

It traced the dependency chain beneath it: answer q1 before you can ask whether restoring BBB transport reverses decline (q3), or what the molecular cascade actually is (q2). That chain — what must be known before this becomes answerable — is the part a search engine never shows you.

What broke — and what it taught

The first run produced nothing. Not an error — just empty passes, silently. I had to actually watch it to find out why, and the cause is worth recording because it will bite anyone building on a reasoning model.

Our 35B model does not answer immediately. It emits a Chain-of-Thought — roughly fourteen thousand characters of reasoning — before the actual answer. That is a feature; it is why the model reasons well. But it has a consequence I had not fully internalized: if you cap max_tokens too low, the CoT eats the entire budget and the real JSON answer gets truncated to nothing. I had set the claims pass to 2000 tokens. The model spent those on thinking and returned an empty object. My JSON extractor correctly found no {...} and the pass returned empty. No crash, no warning — just a quiet void where the claims should be.

The fix is mundane and general: give reasoning models headroom. I set a global `KC_MAX_TOKENS=*** and sized each pass's budget above its CoT length (claims 6000, contradictions 5000, gaps 6000, question 9000). The compile then produced a full, valid artifact. Two principles fall out of this:

1. Schema-first, repair-always, trust the artifact not the model. The model's output drifts from the schema (a string where an array was promised, a null where a note belonged). Every pass clamps and validates before the next consumes it. The artifact — the versioned JSON you can inspect and diff — is the product. The model call is a fallible subroutine.
2. Budget for the model's behavior, not your prompt's length. A reasoning model's "answer" can be ten percent of what it emits. Plan for the other ninety percent.

This is the same discipline the Discovery Compiler needed, and it is the reason both compilers are survivable: a compiler does not treat its model as an oracle. It validates, repairs, and versions. When the loop is the product, observability becomes the operating system — and you cannot observe what you do not checkpoint.

Why local-first is not optional here

Every pass ran against a model on our own machine. No cloud, no API key, no data leaving the box. This is not a constraint I am apologizing for — it is the point. A research infrastructure that phones home is not yours. If the map of scientific uncertainty lives on someone else's GPU, you do not have a map, you have a subscription. Sovereignty is a precondition for trust, and trust is a precondition for anything worth calling knowledge work. The same 35B model that powers this could run on a researcher's laptop; the explorer is a static site that costs nothing to host.

The live result

The compiled artifact renders as a navigable map:

• Uncertainty Map — what the corpus establishes (Known), where it disagrees (Controversial), what it implies but never states (Silence), and the typed gaps underneath.
• Questions — the ranked, classified, dependency-linked research questions, each with its score and the five axes that produced it.
• Known & Contested — the established base and the disputed frontier the questions target.

You are looking at Pass 2 of the Scientific State Compiler, running on hardware we own.

What this makes possible next

With the Question pass working, the roadmap's next move is clearer. The bottleneck analysis from yesterday named Pass 4 (State) as the weakest link, but Pass 2 just gave us the raw material State needs: a ranked uncertainty graph. The immediate next build is to compose those questions, contradictions, and silence maps into a versioned state of a field — accepted vs uncertain vs deprecated models, a conflict-preserving ledger, and a historical-evolution dimension that turns versioning into a timeline. Once State exists, the Discovery pass (hypotheses → discriminating experiments) closes into a true loop.

Concretely, the next increments, each independently shippable:

1. State object — fold Question IR + consensus/silence into a scientific-state-ir keyed by field, reusing the Scientific Git versioning engine from the Discovery Compiler.
2. Evolution pass — diff successive state versions into structured historical transitions (model accepted / deprecated / uncertainty shifted).
3. ScienceOS dashboard — a surface rendering Current State, Historical Timeline, Uncertainty Map, Conflict Map, Mechanism View.
4. Close the loop — point the recursive agent at the state artifact so each iteration re-ranks questions and commits a new version.

Today's compiler is the seed for all of it. It is small, local-first, and inspectable — the kind of thing you can actually run and trust.

The point

We are not building a better search engine. We are building the missing infrastructure for directing research: a versioned, observability-first map of where understanding is weakest, compiled on our own hardware, that you navigate instead of searching. The telescope extended vision; the microscope extended scale; the computer extended calculation. A scientific compiler extends scientific reasoning — by making the next most valuable question obvious, well-grounded, and traceable back to its sources.

Pass 2 is done. The live map is up. The loop is beginning to turn.

The Scientific Question Compiler is open source at github.com/kliewerdaniel/scientific-question-compiler and running at scientific-question-compiler.vercel.app, compiled entirely on local hardware.