Phase −1 · offline validation
180 simulated forecasters, 6 correlated information clusters, 900 claims. Weights and the extremization parameter are fitted on the first 630 chronologically and evaluated out-of-sample on the remaining 270. The market reference is itself a skilled forecaster, so beating it is hard by construction.
Everything below is regenerated by npm run emit. Nothing on this page is typed by hand.
Risk engine evidence of the first onchain stock options exchange
The exchange prices margin and collateral off models. This page is the measured evidence that those models behave as specified, including the runs where they do not. A venue that will not publish its failing cases is asking for trust it has not earned. Open the terminal.
Reference run
Lower Brier is better. The axis is floored, not zero-based, every method sits in a narrow band and the floor is printed on the axis so the truncation is never silent.
| Method | n | Brier | Skill vs market |
|---|
The one that fails
Every other protocol site would drop this row. It stays, because a validation harness that cannot fail proves nothing, and because the honest reading of this evidence is narrower than the pitch.
| Seed | Market | Naive | Engine | vs naive | vs market | Gate |
|---|
✓ What this supports
Calibration weighting beats naive pooling on 6 of 6 seeds. The aggregate also beats the best single forecaster selected on training data, which is the whole point of aggregating rather than talent-scouting.
✗ What this does NOT support
Consistent alpha over a strong market reference. Seed 42 loses by , . Any claim of reliable outperformance versus the tape is unsupported by this evidence.
⚠ The limitation that matters most
These forecasters are synthetic, with known ground truth. This shows the engine behaves as specified, not that it works on real instruments. The real Phase −1 gate replays this same harness against public corpora: Metaculus histories, Good Judgment Open, historical prediction-market prints. The harness already accepts that input shape. It needs a loader and a dataset, and it is the single highest-value task on the roadmap.
Aggregate calibration
The dashed diagonal is perfect calibration. Point area encodes how many forecasts landed in that bin, the sparse tails are where apparent miscalibration is mostly just small n, which is exactly why Wilson intervals ship by default.
Display honesty
This curve is coarse-binned and explicitly approximate. The three-term Murphy identity is exact only on exact probability values, see the correction below.
| Stated | Observed | n |
|---|
Corrections
Each of these was written in prose, believed, and then falsified by the test suite. They are documented as corrections rather than quietly fixed, because with a technical audience that is a credibility asset, not an embarrassment.
01
We said: weight forecasters by skillScore (versus market).
What happened: all 180 simulated forecasters received zero weight. Almost nobody beats a strong market reference alone, so max(0, SS) zeroed everyone and pool() returned null for every claim.
Fix: weight on skillVsBaseline; rank and sell on skillScore. A forecaster individually worse than the market can still add information, because pooling exploits error diversification.
02
We said: bin forecasts into 10 bins, B = REL − RES + UNC.
What happened: the test failed, 0.18069 vs 0.17856.
Why: the identity is exact only when forecasts sharing a bin are identical. Expanding (p−o)² around a bin mean leaves both a within-bin variance term and a cross term. The first fix, adding only the variance term, still failed.
Fix: decompose() groups on exact probability values. reliabilityCurve() does coarse binning for display and is labelled approximate.
03
We said: extremization corrects pooled under-confidence and improves out-of-sample accuracy.
Measured: fitted a = 1.3 produced Brier , versus , with none. It overfit the training split. The redundancy penalty also cost slightly.
Status: both retained on theoretical grounds but must be re-validated per claim class, never assumed. The best measured configuration is plain "weighted, no extremization."
04
We said: stating 0.55 and being wrong costs 0.2025, more than 4× cheaper than a confident error.
It costs .... The ratio to a confident error is ..., not more than 4×.
Both figures on this page are now computed by brier() at build time, so the same error cannot recur silently.
The board
A board showing only winners is advertising, not evidence. Negative skill is shown, not hidden, and note how little weight a strong-looking record actually earns once shrinkage and redundancy are applied.
| Forecaster | n | Brier | Skill vs market | Skill vs baseline | ρ | Weight | Status |
|---|
Weighted on skill-vs-baseline, ranked on skill-vs-market. Conflating the two is what zeroed every forecaster in the first implementation, correction 01 above.
Reproduce it
Every number here comes out of these commands. Run them.
$ npm test 38 passing $ npm run validate reference run, seed 20260918 $ npm run validate -- --seed 42 VERDICT: FAIL , the row we did not delete $ node test/crosscheck.mjs browser build matches node build exactly $ npm run emit regenerates every figure on this site
,