You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Finding: vendor ties mean the reported choice can differ from argmax, so adapters must read the choice #10
This is a recorded finding, not a defect. plumbline already does the
right thing. This documents why, so nobody "simplifies" it back.
What was measured
On a live run of 40 choice rows against jev-1.13.0, one row
(hard-opus-c-temporal_numeric-04) returned two options tied for the maximum
at identical probability, 0.23 each.
The vendor selected one of them. An argmax over the returned distribution
picked the other, because which one an argmax returns is decided by dictionary
iteration order and nothing about the answer.
The gold label was the option the vendor did not select. So on that row,
two implementations disagreed about whether the system was right, and neither
logged anything unusual. One in 40 rows.
Why ties are ordinary here
Probabilities arrive on a two-decimal grid over three to six options. At that
resolution a tie for the maximum is an expected event, not a freak one. See the
quantization issue for the mechanism.
The rule
Read the vendor's selected label. Never recompute it.
A distribution is evidence about the selection; it is not the selection.
plumbline reads answer.choice and looks up probabilities[answer.choice] for
the calibratable column, which is correct even on a tie. The adapter docstring
and METHODOLOGY both state the rule, and METHODOLOGY frames it as guidance for
anyone writing an adapter rather than as a note about one vendor.
Note that the value is always a maximum: zero of 40 rows had prob_selected
differing from max(distribution.values()). It is the key that can differ.
What remains open
Should a tie be recorded in the artifact? Currently a tied row looks like
any other. A reader auditing a close result cannot tell that a row was
decided by the vendor's tie-break rather than by a margin. A flag would cost
nothing and would make the audit possible.
Should the report surface the tie count? If a meaningful fraction of rows
were decided by tie-break, that is a fact about the measurement's precision
and arguably belongs beside the accuracy figure.
Is the vendor's tie-break deterministic? Unknown. If the same row can
resolve differently between calls, cached and live results can disagree for a
reason that is nobody's fault, and the cache documentation should say so.
Two of the three open questions are answered in #97:
Should a tie be recorded in the artifact? Yes. Each record carries tied_for_top.
Should the report surface the tie count? Yes, beside accuracy, including how many tied rows went against gold.
Still open: whether the vendor's tie-break is deterministic. That needs repeated live calls on a tied row, and tied_for_top is what makes those rows findable.
Ran the last open question live on 2026-10-04: the row that tied on the first run (hard-opus-c-temporal_numeric-04), asked ten times against jev-1.13.0, each from an empty cache.
The selected label was the same on all ten, and none tied.
The probabilities are not fixed: the selected option ranged 0.37 to 0.43 and the runner-up 0.31 to 0.36 across the ten calls.
So the distribution is a draw per call, a tie is a draw too, and the tie-break cannot be tested apart from the draw it breaks. Recorded in METHODOLOGY (00d27b8 and the commit after it). Leaving this open: tied_for_top marks where the question can arise, and a tie would have to be caught live to see how it resolves.
Activity
feat: record a tie for the top option and count ties in the report (#97)
TMHSDigital commentedon Sep 25, 2026
Two of the three open questions are answered in #97:
tied_for_top.Still open: whether the vendor's tie-break is deterministic. That needs repeated live calls on a tied row, and
tied_for_topis what makes those rows findable.TMHSDigital commentedon Oct 4, 2026
Ran the last open question live on 2026-10-04: the row that tied on the first run (
hard-opus-c-temporal_numeric-04), asked ten times againstjev-1.13.0, each from an empty cache.So the distribution is a draw per call, a tie is a draw too, and the tie-break cannot be tested apart from the draw it breaks. Recorded in METHODOLOGY (00d27b8 and the commit after it). Leaving this open:
tied_for_topmarks where the question can arise, and a tie would have to be caught live to see how it resolves.docs(methodology): the tie-break and the grid, measured live on 2026-…