Research Active ★ Featured Jan 2026

Aurora

A modular reasoning architecture that knows when to abstain: six stages with an explicit uncertainty channel, a deterministic DAG router, stateless experts and a verification anchor. GSM8K accuracy 47.2% to 55.6% while confidently wrong answers drop from 19.8% to 2.9%.

Live demo Source Docs ★ 3

researchtransformersuncertaintycalibrationpytorchnvfp4reasoning

gsm8k accuracy: 55.6%
confidently wrong: 2.9%
error slip: 14%
inference cost: 0.29x
memory: 1.6 GB
calibration ece: 0.0073
multi-art: 57.9%

One forward pass as a neural graph: the uncertainty channel U gates what executes and what gets accepted. Open the image for the full-size animated version.

Language models answer wrong questions in the same confident tone as right ones. In medicine, law or finance, a fluent guess is worse than a refusal. AURORA-Transformer (ART) is our attempt to make the refusal a first-class outcome: the model cannot reach the reader without passing a verification gate, and the uncertainty behind that decision is a learned output rather than a side effect of the final softmax.

The library is six small modules, each with one job, and every forward pass exposes the uncertainty signal to the caller. It ships the training pipeline, a benchmark harness over six evaluation suites, a FastAPI server, YAML configs for single and multi-instance deployments, and a pytest suite that CI runs on Python 3.10, 3.11 and 3.12.

The six stages

Binary Perception Layer (BPL). Text is pre-tokenised offline into fixed-width 32-bit records that are memory-mapped at training time, so the runtime loop reads integers instead of parsing strings.
Cognitive Transformer Core (CTC). A bounded, decoder-free transformer that projects its final states into three channels: reasoning R, control C and a sigmoid-bounded uncertainty U, trained with an auxiliary correctness objective.
Dynamic DAG Router (D³R). Five control primitives (EXECUTE, COMPOSE, BRANCH, AGGREGATE, VERIFY) are selected by deterministic argmax against projection vectors frozen after training. The router assembles acyclic graphs with hard caps on depth and width, and drops any sub-task whose uncertainty crosses a threshold.
Domain experts. Stateless modules behind an ExpertRegistry: math (with SymPy for exact symbolic evaluation), code, retrieval and a passthrough fallback. They share no memory and cannot call each other.
Context and Reality Anchor (CRA). Every result is checked against R and U and returns ACCEPT, REJECT or ABSTAIN before it counts for anything.
Synthesis Head (SYN). Accepted results are merged through confidence-weighted attention, producing the answer or an explicit abstention when too little survives verification.

The uncertainty channel feeds two places, not one: the router uses it to decide what runs, and the anchor uses it to decide what is accepted.

One forward pass, step by step

The question enters the BPL as bounded token records, which keeps input variance low and removes the CPU-bound parsing loop.
The CTC reasons over those embeddings and returns three separate signals instead of a hidden state soup: what it understands, what it wants to do, and how unsure it is.
The router turns the control signal into a task graph. Routing is deterministic, so the same input produces the same graph and the same run reproduces.
Sub-tasks above the uncertainty threshold are suppressed before execution. Nothing below the threshold is dispatched.
Surviving tasks run on stateless experts and return structured results.
The anchor verifies each result. Rejected results are dropped, and if too few survive, the synthesis head abstains in plain language.

What the numbers look like

	Dense baseline	ART (NVFP4)
GSM8K accuracy	47.2%	55.6%
Confidently wrong	19.8%	2.9%
Errors slipping verification	82%	14%
Relative inference cost	1.00x	0.29x
Memory	5.4 GB	1.6 GB

Three independent instances voting under uncertainty (Multi-ART) push accuracy to 57.9%, confident errors to 1.8% and slip-through to 5%, with disagreement triggering abstention instead of a forced consensus.

Reliability diagram for ART under NVFP4 with an expected calibration error of 0.0073: the accuracy bars track the diagonal, so an 80% confidence claim is right about 80% of the time.

Accuracy as the model is allowed to abstain: the answers it keeps become more accurate as coverage shrinks.

Why 4-bit helps instead of hurting

The architecture was designed for NVFP4 from the start. Activations stay bounded, the graph is non-recursive, and the routing is deterministic, so quantization noise shows up as measurable uncertainty rather than silent hallucination. Dense models at the same precision degrade sharply; ART barely moves. That property is also why the error-growth curve stays flat with depth while dense models compound small numerical errors layer after layer.

Native NVFP4 acceleration needs Blackwell-class hardware (compute capability 10.x). Everywhere else the library runs in a faithful simulation mode with the same numerics, just without the speed and memory win, which is how the test suite runs on CPU in seconds.

What ships in the repo

The six modules as importable PyTorch components, with the main ART model composing them
Training pipeline, dataset handling, metrics, calibration utilities and quantization-aware layers
A benchmark harness covering six standard suites, with failure injection and cost analysis
YAML configs for art_base, art_full and multi_art, plus export and inference scripts
A FastAPI inference server and a pytest suite that CI runs across three Python versions
The JOSS-style manuscript, its figures, and every result table as structured JSON

One honest limitation: the 8B instruction-tuned checkpoint was trained on a corpus we cannot redistribute, so the weights are not in the repository. Everything needed to train a checkpoint from public data is, and the export placeholders document the expected layout.

Stack

Part	Technology
Core	PyTorch, custom bounded transformer
Precision	NVFP4 4-bit, CUDA kernels
Perception	mmap binary token records
Experts	SymPy math, code, retrieval, passthrough
Serving	FastAPI
Configs	YAML, three presets
Tests	pytest, Python 3.10 to 3.12

What made it hard

Stopping the error avalanche

In a dense stack, a small numerical error in an early layer changes everything downstream: depth multiplies noise. Our first prototypes showed the same curve. The fix was structural rather than numerical. The reasoning core is non-recursive, activations are Lipschitz-bounded, and the router caps graph depth and width, so there is no path for an error to feed back into itself. The error-growth curve flattens, and that one property is what makes the rest of the design work.

Making uncertainty a real number

Early versions read confidence off the output layer, which is exactly the habit that produces confidently wrong answers. The uncertainty channel is now a separate, sigmoid-bounded output trained with an auxiliary correctness objective, and calibration is measured with a reliability diagram rather than assumed. Getting the expected calibration error under one percent took several rounds of threshold tuning and a hard look at which sub-tasks the gate was letting through.

Deterministic routing

Agent frameworks route with sampling, which makes a run impossible to reproduce and a bug impossible to isolate. We froze the projection vectors after training and route by argmax, so the same input builds the same task graph every time. Reproducibility is the reason the benchmark numbers mean anything.

4-bit numerics from the first commit

Designing for NVFP4 later would have meant rewriting the core. We kept activations bounded and avoided recursive state so quantization noise lands in the uncertainty channel as a detectable signal. Dense baselines under the same precision got dramatically worse, which was the moment the design choice paid off.

A gate that is neither paranoid nor careless

Set the threshold too high and the model abstains on questions it could answer; set it too low and errors slip through. The 14% slip-through rate against 82% for the dense baseline came from tuning against six suites rather than one, because a threshold that looks safe on GSM8K was leaking on the verification-heavy suites.

The checkpoint we cannot ship

The 8B instruction-tuned checkpoint was trained on a corpus we are not allowed to redistribute. Shipping the architecture without the weights keeps the repository honest, but it also doubled the documentation: training instructions, export placeholders, and a clear note in the README so nobody runs a benchmark against a missing file.

Outcome

What exists today

A PyTorch library with the six modules importable on their own, plus the composed ART model and a one-call forward pass
GSM8K accuracy up from 47.2% to 55.6%, confidently wrong answers down from 19.8% to 2.9%, and errors slipping past verification down from 82% to 14%
Inference cost at 0.29x and memory at 1.6 GB against 5.4 GB for the dense baseline, with Multi-ART at 57.9% accuracy and 1.8% confident errors
A reliability diagram with an expected calibration error of 0.0073, and abstention curves that trade coverage for accuracy
A benchmark harness over six suites with failure injection, cost analysis and stress tests
Three YAML presets, training and export scripts, and a FastAPI inference server
A pytest suite that CI runs on Python 3.10, 3.11 and 3.12, plus an end-to-end smoke test
A JOSS-style manuscript, sixteen result tables as JSON, and every figure regenerable from committed data

What I'd do differently

If we built it again

Put the verification gate in the first prototype. We built the router, experts and synthesis before the anchor, then rewrote each module to accept a verdict it did not expect to receive.
Train a small reference checkpoint on public data and ship it. Without weights, every reader who wants to reproduce a number has to train first, and that is a wall most of them will not climb.
Measure calibration from the first training run. Calibration turned out to move independently of accuracy, and a model that scores well can still be badly overconfident.
Keep the benchmark harness and the training loop in one repository from day one. They drifted for a while, and the numbers in the paper briefly disagreed with the numbers in the JSON.
Treat the uncertainty threshold as a tuned hyperparameter with a documented sweep. It ended up mattering more than most architecture choices, and the sweep that justified 14% slip-through happened late.