Skip to content

CI: boot the release candidate against data fixtures from every published stable #990

Description

@frahlg

Why

Our own validation walks beta-by-beta on our sites, but users jump: the 2026-08-29 incident was a v1.15.0 box meeting v2.3.0 (#988). No automated check ever boots the candidate against the on-disk state a real older box brings — config.yaml written by an old version, state.db with old schema, driver overlays.

Proposal

An e2e job in the release pipeline (beta.yml, and as a make target locally):

  1. Keep a small library of data-directory fixtures, one per published stable (config.yaml + state.db + drivers dir), captured from the e2e stack right after that version's suite ran — not hand-written.
  2. For each fixture: start the candidate Core against a copy of it and require /api/health 200 and a fully started /api/status within a deadline.
  3. Assert zero ERROR-level config/load lines except known, listed repairs (e.g. Duplicate is_site_meter bricks boot: config rejection exits before the HTTP server binds #988's demotion warning).

This turns "an old config bricks the new version" from a field incident into a red check before a beta tag exists. Start with fixtures for the still-deployed stables (1.15.0, 2.0.0, 2.3.0) rather than every historical version.

🤖 Generated with Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions