PolyLedger × ForecastBench

PolyLedger × ForecastBench is a public forecasting-verification experiment.

This repository contains a reproducible pipeline for generating ForecastBench submissions and sealing them before resolution. Each submission is paired with a hashchain ledger entry, file SHA256 commitments, and a Merkle commitment to structured rationale leaves. The aim is simple: forecasts should be auditable before we know whether they were right.

Independent participant project; not affiliated with the Forecasting Research Institute.

Why This Exists

Forecasting benchmarks are vulnerable to three quiet failure modes: forecasts can be edited after the fact, rationales can be rewritten after resolution, and market-based questions can blur the line between independent judgment and crowd copying. PolyLedger treats those as verification problems. Every round is committed before resolution, and later review can check the cryptographic trail.

How To Verify A Rehearsal Entry

Verify committed artifacts without regenerating submissions or appending to the local ledger:

python3 scripts/verify.py --ledger-entry ledger/2026-07-05.entry1.json

This recomputes:

• the ledger entry_hash
• the generated submission file SHA256
• the structured rationale file SHA256
• the rationale Merkle root
• the ledger/chain.jsonl hashchain consistency

Forecast Sets

• polyledger-core: statistical priors + LLM overlay + market crowd blend
• polyledger-pure: independent version without market crowd blend
• polyledger-quant: statistical control and fail-safe submission

polyledger-quant is a market-anchored control. Its market forecasts intentionally shrink the 10-day-old freeze value toward 0.5, so high correlation with freeze-time market values is expected by design and is not an independence claim. Independence metrics belong to the future A/B entries (polyledger-core and polyledger-pure).

Status

Current status: rehearsal pipeline. The public append-only ledger discipline begins with the first live ForecastBench submission.

OpenTimestamps: stub during rehearsal; real anchors from the first live seal.

Implementation

Implemented:

• ingest/: download and validate ForecastBench question sets.
• quant/: deterministic quant forecaster with market freeze shrinkage, yfinance/FRED diffusion-prior support, and historical base-rate fallbacks for dataset sources.
• compose/: build and validate a polyledger-quant submission file.
• seal/: v2 rehearsal ledger entry with SHA256 hashchain plus OpenTimestamps stub.
• submit/: local naming and GCS upload helper stub.

Rationale note: ForecastBench submission files keep reasoning: null, but the ledger commits to separate structured rationale leaves for the quant control: method name, parameters, base-rate metadata, and monotonic/clipping adjustments.

Base-rate note: historical base rates use only prior rounds and resolutions with resolution_date <= target_forecast_due_date. This is an as-of cutoff by ForecastBench resolution date, not by source publication timestamp; source publication lag can theoretically affect a small boundary slice.

Reproduce The Rehearsal

This rewrites local outputs/ and ledger/ rehearsal artifacts. Use it in a clean checkout when you want to test byte-for-byte reproducibility:

python3 scripts/run_dryrun.py --question-set 2026-07-05-llm.json

Refresh the no-leakage historical base-rate cache for a target round:

python3 scripts/build_base_rates.py --target-due-date 2026-07-05

Operational rule: rebuild the base-rate cache on D-1 for each live round, then commit the cache and run the dry run. For the 2026-07-19 round:

python3 scripts/build_base_rates.py --target-due-date 2026-07-19
python3 scripts/run_dryrun.py --question-set 2026-07-19-llm.json

Run tests:

python3 -m unittest discover -s tests

Network history lookups are disabled by default so the dry run is fast and reproducible. Base-rate priors are read from the committed cache. To enable Yahoo/FRED history fetches for yfinance/FRED priors:

python3 scripts/run_dryrun.py --question-set 2026-07-05-llm.json --fetch-history