Skip to content

Finding: vendor ties mean the reported choice can differ from argmax, so adapters must read the choice #10

Open

Description

@TMHSDigital

This is a recorded finding, not a defect. plumbline already does the
right thing. This documents why, so nobody "simplifies" it back.

What was measured

On a live run of 40 choice rows against jev-1.13.0, one row
(hard-opus-c-temporal_numeric-04) returned two options tied for the maximum
at identical probability
, 0.23 each.

The vendor selected one of them. An argmax over the returned distribution
picked the other, because which one an argmax returns is decided by dictionary
iteration order and nothing about the answer.

The gold label was the option the vendor did not select. So on that row,
two implementations disagreed about whether the system was right, and neither
logged anything unusual. One in 40 rows.

Why ties are ordinary here

Probabilities arrive on a two-decimal grid over three to six options. At that
resolution a tie for the maximum is an expected event, not a freak one. See the
quantization issue for the mechanism.

The rule

Read the vendor's selected label. Never recompute it.

A distribution is evidence about the selection; it is not the selection.
plumbline reads answer.choice and looks up probabilities[answer.choice] for
the calibratable column, which is correct even on a tie. The adapter docstring
and METHODOLOGY both state the rule, and METHODOLOGY frames it as guidance for
anyone writing an adapter rather than as a note about one vendor.

Note that the value is always a maximum: zero of 40 rows had prob_selected
differing from max(distribution.values()). It is the key that can differ.

What remains open

  • Should a tie be recorded in the artifact? Currently a tied row looks like
    any other. A reader auditing a close result cannot tell that a row was
    decided by the vendor's tie-break rather than by a margin. A flag would cost
    nothing and would make the audit possible.
  • Should the report surface the tie count? If a meaningful fraction of rows
    were decided by tie-break, that is a fact about the measurement's precision
    and arguably belongs beside the accuracy figure.
  • Is the vendor's tie-break deterministic? Unknown. If the same row can
    resolve differently between calls, cached and live results can disagree for a
    reason that is nobody's fault, and the cache documentation should say so.

Activity

  1. added
    adapterAdapter transports and vendor configuration
    methodologyHow a number is computed and what it does or does not mean
    open questionA measured finding with something still unresolved. Not a defect.
    on Sep 22, 2026
  2. TMHSDigital commented on Sep 25, 2026

    @TMHSDigital
    OwnerAuthor

    Two of the three open questions are answered in #97:

    • Should a tie be recorded in the artifact? Yes. Each record carries tied_for_top.
    • Should the report surface the tie count? Yes, beside accuracy, including how many tied rows went against gold.

    Still open: whether the vendor's tie-break is deterministic. That needs repeated live calls on a tied row, and tied_for_top is what makes those rows findable.

  3. TMHSDigital commented on Oct 4, 2026

    @TMHSDigital
    OwnerAuthor

    Ran the last open question live on 2026-10-04: the row that tied on the first run (hard-opus-c-temporal_numeric-04), asked ten times against jev-1.13.0, each from an empty cache.

    • The selected label was the same on all ten, and none tied.
    • The probabilities are not fixed: the selected option ranged 0.37 to 0.43 and the runner-up 0.31 to 0.36 across the ten calls.

    So the distribution is a draw per call, a tie is a draw too, and the tie-break cannot be tested apart from the draw it breaks. Recorded in METHODOLOGY (00d27b8 and the commit after it). Leaving this open: tied_for_top marks where the question can arise, and a tie would have to be caught live to see how it resolves.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    adapterAdapter transports and vendor configurationmethodologyHow a number is computed and what it does or does not meanopen questionA measured finding with something still unresolved. Not a defect.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions