Skip to content

Two REQUIRED contexts cannot fail, and the cross-arch self-test asserts nothing about the gate #389

Description

@avrabe

[fathom (gale) — gate-potency audit; the four below I re-verified myself]

1. CONFIRMED — Zephyr C Coverage is a REQUIRED context that cannot fail on test results

.github/required-contexts.txt:4   Zephyr C Coverage
.github/workflows/coverage.yml    twister ... \
                                    --outdir "${GITHUB_WORKSPACE}/twister-out" \
                                    || true

The || true discards the test runner's exit status, and no later step inspects
twister-out/twister.json for FAILED/ERROR — it is only uploaded as an artifact.
A total kernel-test build break, or a 100%-fail run, reports this required check
green.

2. CONFIRMED — FFI Feature Sizes is REQUIRED and never builds 13 of 43 features

ffi/Cargo.toml   all = [...]                 43 features
bazel-tests.yml  FEATURES="sem mutex ..."    30 features

NEVER built in isolation (13):
  atomic cbprintf condvar cpu_mask ipc ipi mmu net_buf pm
  spinlock spinlock_validate usage zms

A hardcoded list beside a generated one. #[cfg(feature = "condvar")] code that
only compiles because another feature happens to be enabled passes this required
check. spinlock, atomic and condvar are in the missing set.

3. CONFIRMED — check-cross-arch.py --self-test asserts nothing about the gate

if {"read32", "write32"} == {"read32", "write32", "synth_func_18"}: ...
if set() != set(): ...

Both controls compare literal set literals. The function never calls
imports(), undefined(), build(), or the real comparison in main(). It
verifies that Python's == works.

So if main()'s arm == imp were weakened to a superset-tolerant comparison —
reintroducing precisely the leaked-symbol bug gale#316 was opened to fix — the
self-test would still print ok and exit 0.

I have been treating this gate's control as meaningful in my own work on this
branch. It is not.

4. CONFIRMED — the four Renode bench workflows gate on a UART string, not on results

engine-bench-renode{,-flight,-lto,-synth}.yml pipe every analyze.py run to
> /tmp/....md || true and no step reads the exit code — while analyze.py is
what computes drops == 0, samples >= 95%, median_delta < 10%. The only
exit-code-bearing step is the robot, whose test is named "Engine Control Bench
Completes With Zero Drops"
and whose body only does
Wait For Line On Uart === END ===. The robot file's own comment admits it never
parses drops.

A regression producing nonzero drops merges green.


Reported by the audit, NOT verified by me

  • paths: filters that skip the gate when its own input changes:
    gust-syscall-seam.yml omits drivers/wit/** (every provider imports it);
    drv-components.yml omits the very directories its census covers
    (dma-own/, os-node/, spawn-provider/, timer-provider/);
    gustos-dissolve.yml omits check-data-overlap.py, which its build script
    invokes as a hard gate.
  • check-driver-components.py:113 turns an nm failure into WARN + return 0,
    while still incrementing the "N objects compared" counter — so the
    anti-vacuity count would lie.
  • build-dissolve-gustos.sh:169 — its own negative control prints SKIPPED and
    exits 0 when its fixture is missing.
  • tools/verify-published-component.sh is cited as method: automated-test for
    VER-OS-PUBLISH-001 (status verified) and is invoked by no workflow.
  • tools/check-v-closure.py:27 globs artifacts/*.yaml non-recursively,
    excluding artifacts/variants/ (zero live impact today).

The iso-core-fused-cm3.o census blind spot the audit also found is filed
separately in #386.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions