Skip to content

fix(cloud_validation): match rehearsals by completion time, not a skewed created_at #321

Closed

Description

@Jammy2211

Overview

On 2026-11-01 the Heart board went STALE ("release validation incomplete: no rehearsal for current source") although integrate run 41291876649 passed for v2026.11.1.1.dev85301 and its matching TestPyPI rehearsal (PyAutoHands run 41290559836) finished before it. GitHub reported that rehearsal's created_at an hour late (09:50:55Z, after its own updated_at 09:09:27Z), and cloud_validation.collect() skips any candidate whose created_at is later than the integrate start — so a valid rehearsal was discarded before matching_rehearsal() ran.

Plan

  • Order rehearsal vs integrate by when the rehearsal finished, not by GitHub's start stamp.
  • Keep every identity check (version, mode, index, run_id, run_attempt, build_sha, completed status) unchanged.
  • Keep rejecting any rehearsal that genuinely finished after the integrate started.
  • Stop a skewed created_at from stamping the validation report's timestamp.
  • Regression tests modelled on run 41290559836.
  • Tier: glance — merge mode: auto-merge on green if the Witness passes, post-merge summary.
Detailed implementation plan

Affected Repositories

  • PyAutoHeart (primary)

Branch Survey

Repository Current Branch Dirty?
./PyAutoHeart main (ea6c1b4) clean

Suggested branch: feature/heart-rehearsal-clock-skew

Reproduction (clean main ea6c1b4)

Using the evidence fixture in tests/test_cloud_validation.py, set the producer's created_at after the integration's created_at and its updated_at before it: collect() returns rehearsal_matched: False (outcome incomplete).

Implementation Steps

  1. heart/checks/cloud_validation.py: add _effective_created(run) returning the earlier of created_at and updated_at (a run cannot finish before it was created, so created_at > updated_at means a skewed start stamp); None when either is missing, preserving today's skip.
  2. In collect(), use it for the candidate prefilter (created > observed → continue) and for observed = min(observed, created). matching_rehearsal() keeps produced (updated_at) <= integration start as the ordering guard, unchanged.
  3. tests/test_cloud_validation.py: skewed start + finished before integrate → matched, pass, report ts not later than the integrate start; skewed start + finished after integrate → incomplete; existing tests unchanged and green.
  4. Full Heart pytest; post-merge, the next Heart Health cloud tick re-judges the live chain. No evidence, threshold or scoring edits.

Key Files

  • heart/checks/cloud_validation.py — rehearsal/integrate pairing for the cloud board
  • tests/test_cloud_validation.py — provenance and freshness tests

Original Prompt

Click to expand starting prompt

Cloud validation discards a matching rehearsal when GitHub reports a skewed run created_at

Type: bug
Target: PyAutoHeart
Repos:

  • PyAutoHeart
    Themes:
  • release-validation
  • heart
    Difficulty: easy
    Autonomy: supervised
    Priority: high
    Status: draft
    Consequence: glance
    Witness: a heart/checks/cloud_validation.collect() test with injected fetch/get_artifact reproduces run 41290559836's timestamps (created_at 09:50:55Z, updated_at 09:09:27Z) against an integrate starting 09:09:36Z: it matches the rehearsal (rehearsal_matched: true) and the report is not stamped later than the integrate start; a rehearsal whose jobs genuinely completed after the integrate started is still rejected.
    Review-minutes: 10

Source: /health dashboard review, 2026-11-01 (claude.ai/code session).

Why

On 2026-11-01 (US DST-end day) the published board went STALE with
"release validation incomplete: no rehearsal for current source", and
validation_report:coverage read "incomplete — no rehearsal evidence —
v2026.11.1.1.dev85301", although the nightly chain had produced a valid pair:

  • PyAutoHands release.yml run 41290559836 (rehearsal, run 853 →
    v2026.11.1.1.dev85301): API reports created_at / run_started_at
    2026-11-01T09:50:55Z but updated_at 09:09:27Z. Its jobs report
    started_at 09:51:10Z and completed_at 09:00:08–09:09:20Z — every
    start stamp is one hour late; the run really finished at 09:09:27Z.
  • PyAutoHeart Release Integrate 41291876649 started 09:09:36Z and passed.

collect() prefilters candidates with created > observed → continue, so
the skewed created_at (09:50:55 > 09:09:36) discarded the rehearsal before
matching_rehearsal() (version, run_id, run_attempt, build_sha,
produced <= start) was ever consulted. The release verdict stayed STALE for a
genuinely validated source. Re-running the chain clears it only by luck of
timestamps; the gate should not depend on GitHub's start stamps being sane.

Re-read the API timestamps of runs 41290559836 and 41291876649 before acting —
GitHub may have corrected them.

What

  • Order candidates by when the producer finished, not by created_at: the
    existing produced = updated_at <= integration start check in
    matching_rehearsal() is the real ordering guard. Treat a run whose
    created_at is later than its updated_at as clock-skewed (use the earlier
    of the two, or drop the created > observed prefilter and rely on
    produced <= start).
  • Do not let observed = min(observed, created) stamp the report from a skewed
    created_at.
  • Keep every identity check (version, mode, index, run_id, run_attempt,
    build_sha, completed status) unchanged; never accept a rehearsal that really
    completed after the integrate started.
  • Regression tests via collect()'s injectable fetch / get_artifact.

Never clear the verdict by editing evidence or thresholds; the fix is to the
collector's time ordering only.


Generated by Claude Code

Activity

  1. Jammy2211 commented on Nov 1, 2026

    @Jammy2211
    ContributorAuthor

    Shipped

    • PR: fix(cloud_validation): match rehearsals by completion time, not a skewed created_at #322, merged as 9f3cbfe (glance tier, auto-merged on green).
    • cloud_validation.collect() now takes a producer run's start as the earlier of created_at and updated_at, for both the candidate prefilter and the report timestamp. A start stamp that GitHub reports late can no longer hide a rehearsal that finished before the integrate run. Identity checks and the produced <= integrate start guard are unchanged.
    • Witness: the regression test built from run 41290559836 fails on ea6c1b4 and passes with the fix. The guard test (a rehearsal that finished after integrate stays incomplete) passes. Full suite: 1313 passed locally; CI 2/2 legs green.
    • Revert: git revert -m 1 9f3cbfe.

    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions