Repository navigation
Finding: vendor probabilities arrive on a 2-decimal grid, bounding achievable ECE resolution #9
Description
This is a recorded finding, not a defect. The vendor is entitled to round
what it puts on the wire, and two decimals is a reasonable place to round for
a decision API whose output is meant to be thresholded. This documents a
bound on what can be measured through that API.
What was measured
On a live run of 40 choice rows against jev-1.13.0 on 2026-09-21, all 165
probability values landed exactly on a multiple of 0.01, as did all 40
confidence values. The distribution is quantized to two decimals on the wire.
What it bounds
METHODOLOGY records this under "Probabilities arrive quantized, which bounds
the resolution of any figure here". The four consequences:
- Binning. The default ECE scheme is ten equal-width bins, which are wider
than the 0.01 grid, so quantization does not bias the binning. A scheme with
more than 100 bins would be measuring the grid rather than the model. - Thresholds. A cascade threshold cannot be tuned more finely than 0.01. A
sweep reporting a cut to four decimals is reporting three digits of noise. - Ties. At 0.01 resolution over three to six options, exact ties for the
maximum are ordinary rather than freak events. See the ties issue. - Recalibration. A fitted temperature is applied to quantized inputs, so
the corrected probabilities are smooth but carry no more information than
what arrived.
What remains open
- Is the achievable ECE floor higher than the sample-size floor already
computed? The nulls currently model binning noise and sample size. They do
not model input quantization. If quantization raises the floor, every ECE
figure through this transport is being read against a floor that is slightly
too low, which would make models look marginally worse calibrated than they
are. Probably a small effect at ten bins, but it is unquantified. - Should the report state the observed grid? It could detect the
quantization from the returned values and say so, rather than relying on a
reader having found this in METHODOLOGY. That is a small change and might be
the whole remedy. - Does it vary by model or by option count? One run, one model version.
Activity
fix(calibration): equal-count bins never split a tie (#82)
feat(report): state the grid probabilities arrived on, and show it co…
TMHSDigital commentedon Sep 25, 2026
Two of the three open questions are answered in #98:
scripts/quantization_floor.pyshows a calibrated model reported on a 0.01 or 0.05 grid clears the floor's 95th percentile at the nominal 5 percent, from 40 to 10,000 rows. METHODOLOGY has the table.Still open: whether the grid varies by model or option count. That needs live runs against more than one model version.
docs(methodology): the tie-break and the grid, measured live on 2026-…
TMHSDigital commentedon Oct 4, 2026
Checked again on 2026-10-04 with a second 40-row run against
jev-1.13.0: 124 of 124 probabilities and 28 of 28 reported confidences are on the 0.01 grid, across yes/no, choice and score rows (two to five options), and ten further calls on one row were on it too. So the grid does not vary by option count in what we can reach. Whether it varies by model is still untested: the endpoint only reportedjev-1.13.0. Recorded in METHODOLOGY. Leaving this open for a second model version.