ci(tests): run the full suite on every main push

[seonghobae]

현재 제안과 검증

Head: ad245962feb19b75f5cbf9e83057e71d633d4bf7. 기존 c36533beb63ba980c2c520fc9fe5c4c448937c8b과 보호 main 43024633eba9d96b0456970391360da5a171fbda를 일반 merge한 후속입니다. 이전 제안의 기능과 commit 이력을 보존했습니다.

기존 trusted-uv workflow가 모든 main push의 전체 검사를 맡도록 통합했습니다. Python 3.10 compile·조건부 tomli import, 3.14의 focused/전체 tests·coverage·전체 docstring 검사를 한 runner에서 실행합니다. PR 경로 필터와 concurrency 격리, read-only 권한, SHA pin, hash-locked 설치는 보존했습니다. checkout은 PR head 또는 push event SHA입니다.

이 quality 범위의 작업 수	보호 main	기존 #1911	현재 제안
materializer 관련 main 변경	2	3	1
그 밖의 main 변경	0	1	1
기존 PR 필터에 해당하는 변경	2	2	1

기존 #1911과 비교하면 해당 main 변경의 작업 수 66.7%, 전체 suite 실행 50% 감소입니다. 실제 조직 점유량·runner 시간·전체 목표 완료율로 계산하지 않습니다.

• 별도 69줄 main-full-suite-gate.yml 제안은 전체 기능을 기존 workflow에 승계한 뒤 제거했습니다. Git 이력에서 복구할 수 있습니다. PR 전용 runtime-quality workflow는 변경하지 않았습니다.
• 3.10 실패 후에도 준비된 3.14 검사를 실행하도록 native !cancelled()와 선행 step 결과를 사용합니다. hardening/checkout/setup/install 실패는 해당 의존 실행을 차단하고, 기존 실패는 최종 job 실패로 남습니다. continue-on-error나 사용자 취소 무시는 없습니다.
• 전체 pytest는 -W error, interrogate는 명시적 --fail-under 100입니다. 실제 workflow pytest 명령을 비밀 없는 임시 환경에서 실행하는 경고 반례를 추가했습니다.
• 제거되는 3.10 check 이름이 main의 필수 context에 없음을 API로 확인했습니다. 보호 설정·principal·token·권한 변경은 없습니다.
• 두 독립 read-only 검토의 제어흐름/docstring 지적을 수정했고 source 167b1a2d 재검토에서 추가 finding이 없었습니다. 이것은 GitHub 승인이나 hosted 제어흐름 실행 증거가 아닙니다.

실제 실행 결과

수정 전 회귀 검사는 4 failed / 2 passed, 검토 보완 전 검사는 2 failed / 5 passed로 실패를 재현했습니다.

최종 committed head에서:

actionlint .github/workflows/trusted-uv-materializer-quality-ci.yml
python -m interrogate --fail-under 100
python -m pytest -q -W error tests/test_trusted_uv_materializer_quality_workflow_contract.py tests/test_required_workflow_queue_contract.py tests/test_agent_review_runtime_quality_consolidation.py
env CI=true GITHUB_ACTIONS=true python -m coverage run -m pytest tests -q -W error
python -m coverage report
git diff --check

• 집중 검사 80 passed, 0 실패/오류/skip, 1.83초. JUnit SHA-256: 321cdbd2e3057b3b4b63a81f248f78f4f05d02b69e18437f9b0f9cac94dc907f.
• actionlint·whitespace 통과, docstring 100%. 설치된 uv Python 3.10.20으로 실제 compile 및 workflow의 조건부 tomli import 명령 통과.
• 전체 CI 환경 검사는 실패: 2920 passed, 11 failed, 1 LLVM19 부재 skip, 21 subtests passed, 101.88초. JUnit SHA-256: 8551bf2363e0c5b86a09ebebac4fd379eff1ba44eea7a23d66841cda1b625ac1.
• 실패 11개 모두 HTTPError ResourceWarning을 포함합니다. 문장 13181/분기 5326 coverage 100%가 이 실패를 성공으로 바꾸지 않습니다.
• HTTP 정리는 기존 #1879의 책임입니다. 이 workflow PR로 runtime을 복제하지 않았습니다. fix(ci): close HTTP error response bodies #1879 보호 반영 후 이 PR의 정확한 head에서 전체 검사와 hosted Checks를 다시 확인해야 합니다.
• 사용한 Python 3.14.6 환경은 기존 project-local 검증 환경 재사용입니다. 새 hash-lock 설치·Linux runner·모델 호출·보호 병합·배포 증거가 아닙니다.

Draft / 미병합으로 유지합니다. 전체 검사 실패를 숨기지 않으며, 재실행·승인·기존 실행 취소·보호 설정 변경은 요청하지 않았습니다.

판단과 대안은 기존 doctoring의 2026-09-06 후속 절에 기록했습니다. GitHub. (n.d.). Evaluate expressions in workflows and actions. 2026년 9월 6일 확인. Context7 할당량 부족으로 공식 문서를 직접 확인했습니다.

보존한 최초 제안과 당시 검증

아래 기록은 이전 head의 이력입니다. 현재 구현이나 성공 증거로 읽지 않습니다.

---

The gap

No workflow ran tests/ on an unfiltered main push. Verified directly against the workflow files:

workflow	runs full tests/?	main-push trigger
agent-review-runtime-quality-ci.yml	yes	none — pull_request only, narrow paths
opencode-review-dispatch.yml	yes	none — repository_dispatch only
trusted-uv-materializer-quality-ci.yml	yes	yes, but filtered to the materialize/uv surface
7 others (security scanners, SBOM, scorecard, secret-scan, merge scheduler)	no	unfiltered

So merging a change to something like pr_review_merge_scheduler_core.py or opencode-review.yml triggered no full-suite run at all. A suite-breaking merge landed silently on main, and the breakage first surfaced as a red check on the next, entirely unrelated PR — whose author then had to prove the failure wasn't theirs.

That is the failure mode behind #1823, #1826, #1828, #1892 and #1895, and behind the repair PRs #1829, #1874 and #1883.

The fix

A post-merge safety net running the repo's own documented triad — coverage run -m pytest tests, the 100% coverage gate, and the 100% docstring gate.

Deliberately no paths filter: the whole point is catching merges that no path list anticipated.

Cost

One runner slot per main push, in this repository only — it is not in the org required-workflow ruleset, so it is not injected into the ~76 sibling repos. Successive pushes coalesce via concurrency + cancel-in-progress rather than stacking, which matters while the queue is saturated.

Verification

• actionlint .github/workflows/main-full-suite-gate.yml — clean
• Full suite with the file present: 2883 passed, 1 skipped (no workflow-structure contract test broken)
• Trigger claims above each read from the workflow files directly, not inferred

Root cause was found by a peer session; this is the prescription half, deliberately kept separate from that session's documentation of the gap.

🤖 Generated with Claude Code

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[seonghobae]

Failing checks on 37312475 classified; recovery = one push merging current main (as c36533be), no code change.

• noema-review (run 33971548451, created 2026-09-05T14:21Z — the old-core straggler push, sidecar on the pre-efb89269 pin, no per-route artifact yet): route preflight passed after 393 s, then the gateway walked for 2704.7 s and answered 502 Bad Gateway (served_model = the last candidate tried, deepseek-v4-flash). That is failure class (1), the orchestrated no-tools walk ending in 502 — see the census on Scheduler pre-review update-branch discards in-flight checks under queue saturation: 76/77 merges since 09-04 had 0/12 required contexts at merge time #1935. Nothing in this PR's single new workflow file is on that path.
• CodeQL compatibility analysis (actions|python) and opencode-review: the designed pending-verdict states while OPENCODE_REPOSITORY_DISPATCH_ACTOR still names the legacy dispatcher (opencode-review-dispatch rejects the current dispatcher identity: allowlist says github-actions[bot], sender is opencode-agent[bot] #1929, owner-gated); the compatibility shards intentionally fail after dispatching, and the dispatched scan's verdict cannot be delivered until the variable changes.
• The CodeRabbit comment is its "no automatic review under 10 stars" notice — nothing to address.

The branch was 22 commits behind main; a re-run would re-execute the sidecar bound at run creation, so the head is refreshed instead with a merge of main@fb2ae81d. The merged tree was gated locally (GITHUB_ACTIONS=true): 2920 passed, 1 skipped, 21 subtests passed in 140.69s (0:02:20);TOTAL 13181 0 5326 0 100%;RESULT: PASSED (minimum: 100.0%, actual: 100.0%). Side effect worth having: the new head's Noema/Strix runs are among the first created after fb2ae81d, so their sidecar artifacts carry deferred_count (#1947), candidate/probed/skipped/ready_count (#1949) and unexpected_exception type= (#1950) together.

[seonghobae]

CodeQL compatibility analysis (actions) on c36533be — same designed state as before, no push. The shard's real output (after ##[endgroup]) is CodeQL scan dispatched. The dispatch workflow will rerun this exact failed CodeQL job after publishing its terminal verdict. — the compatibility shard fails intentionally after dispatching. Its dispatch runs for this head are queued (34021293034 at 08:14:20Z, 34021357120 at 08:15:40Z, one per language) and were sent with the app token, so they will hit the identity gate the same way every dispatch did today (a peer's live test at 07:02Z, right after the variable was touched at 07:01:34Z, still rejected actor=sender=opencode-agent[bot]; see #1929). Nothing in this PR is on that path; the verdict arrives when the owner-gated allowlist admits the dispatcher, and the dispatch workflow then reruns this exact job.

[seonghobae]

noema-review on c36533be (run 34017954595, job 08:19:19–08:21:43Z, artifact noema-sidecar-evidence 9985677359) — class (4) preflight-fail (capacity); no push. This is the first Noema artifact for this PR on the post-fb2ae81d launcher and it carries the new fields: candidate_count 24 / probed_count 6 / skipped_count 18 / ready_count 0 / deferred_count 0 / rejected_count 6 (target_ready 8, probe_budget 16, account_skip_after_429 2). All three credential accounts answered 429 to their first two probes (nvidia_nim ×2, nvidia_nim_sub ×2, openrouter ×2), the account-skip rule dropped the remaining 18 candidates without probing, and with nothing ready the sidecar failed closed before healthz — the provisioning step failed 2 min 24 s in. No serving phase ran, so no unexpected_exception line, as expected for the no-tools shape.

Same shape as the other post-#1949 samples in this hour (contextual-orchestrator#1078's strix run, the main-push strix run): the launcher's levers behave as designed and the pool is rate-limited end to end — the capacity item on #1948. A push cannot change an upstream 429; the branch already carries current main.

[seonghobae]

현재 c36533beb63ba980c2c520fc9fe5c4c448937c8b와 main43024633 기준에서 중복을 확인했습니다. 새 main-full-suite-gate와 기존 trusted-uv-materializer-quality-ci의 full-quality-gate가 materializer main 변경에 동일한 전체 pytest/coverage를 각각 실행합니다. 기존 파일의 3.10 compile/import와 3.14 전체 품질 검사를 한 runner로 순차 통합하고, main push만 경로 필터를 없애며 전체 docstring 검증을 승계하는 최소 후보를 격리 worktree에서 준비합니다. PR 경로 필터, 두 Python 버전, 읽기 전용 권한과 SHA pin, PR/head 및 push/SHA 구분은 유지합니다. 삭제되는 별도 3.10 check 이름은 현재 main 보호 규칙의 required contexts에 없음을 직접 확인했으며 보호 설정은 바꾸지 않습니다. 새 workflow의 기능을 완전히 승계한 뒤 해당69줄 파일만 제거하고 Git 이력은 보존합니다. 기존 PR/원격 head를 일반 갱신 직전에 재조회하며 활성 작성자가 있으면 알려 주세요. #1879 응답 정리 수정의 보호 병합 전에는 경고 엄격 전체 검사 결과를 별도 prerequisite로 구분합니다.

[seonghobae]

opencode-review on c36533be is the receipt gate, not a sidecar failure — and the head has since moved; no push from me.

• The opencode-review check fails with No APPROVED or CHANGES_REQUESTED from opencode-agent on the current head: the required workflow only verifies a receipt. The review itself runs in opencode-review-dispatch.yml, whose validate-pr-metadata rejected 44 of today's 50 failed runs with actor=sender=opencode-agent[bot] — the owner-gated allowlist (opencode-review-dispatch rejects the current dispatcher identity: allowlist says github-actions[bot], sender is opencode-agent[bot] #1929). A live test right after the variable was touched at 07:01:34Z still rejected, so the gate value has not changed yet. Nothing in this PR is on that path.
• Head moved from c36533be to ad245962 at 09:21:48Z (another session merged current main, which now carries fix(strix): name the sandbox bootstrap failure and give it a bounded retry #1953, and added a doctoring document; c36533be is an ancestor, nothing lost). That cancelled the Strix scan that had been running on c36533be since 07:00Z — the cost of moving a head while a scan is in flight. The new head binds the fix(strix): name the sandbox bootstrap failure and give it a bounded retry #1953 sidecar, so its Strix run is the first sample that can show the STRIX_SANDBOX_UNAVAILABLE verdict token if the sandbox class recurs.
• Earlier on this head: noema-review was class (4) preflight-fail (all three accounts 429 twice, ready 0, fail-closed in 2 min 24 s — artifact 9985677359), and both CodeQL compatibility shards are the designed post-dispatch failure.

[seonghobae]

Python 3.14 full quality gate (run 34024435377, step 11) — 11 failures, one signature, and it is not this pull request's content. Every failure carries the same line:

pytest.PytestUnraisableExceptionWarning: Exception ignored while calling deallocator
<function _TemporaryFileCloser.__del__ at 0x…>: None

What the evidence rules out:

• Not this branch's diff. The head adds main-full-suite-gate.yml and a doctoring document; the 11 failing tests are in test_noema_review_gate.py, test_noema_repair_attempt_telemetry.py, test_pingora_edge_policy.py, test_repository_metadata_live_verification.py and test_sandboxed_web_e2e.py, none of which this pull request touches, and their own code builds urllib error bodies from io.BytesIO, not from temporary files.
• Not the whole tree today. The same workflow passed at 08:23:22Z (ad48dd65) and 08:38:53Z (4cf6febc) on two other branches.
• Not the previous failures of this workflow. main's last two failures (09-04) were 17 scheduler tests with no unraisable signature, and the 01:29Z failure on another branch was a single test, also without it.

The signature itself is a finalizer running during garbage collection, which pytest attributes to whichever test happens to be executing at that moment — so the failing test names do not locate the leaked object, and a deterministic reproduction needs a Python 3.14 interpreter (the toolchain available to me here is 3.12). I am not pushing a speculative change on that basis.

The cheap discriminator is a re-run of this one workflow: it runs no review sidecar, so re-running does not resurrect stale trusted code the way re-running a review job would. Triggered now. If it passes, this was finalizer timing; if it fails identically, the leak is real and I will bisect it against the recent main commits this head is the first to carry.

[seonghobae]

Root cause found (a peer, from local git only; I verified it the same way): this branch's own -W error promotes a pre-existing warning into 11 errors.

ad245962   python -m coverage run -m pytest tests -q -W error
main       python -m coverage run -m pytest tests -q

That one flag turns PytestUnraisableExceptionWarning — raised when _TemporaryFileCloser.__del__ runs during garbage collection — into a failure, and pytest attributes it to whichever test happens to be executing at that moment. Every observation now fits: one signature across 11 unrelated tests, other branches passing the same gate today (they have no -W error), the failing test names not locating the leak, and my 3.12 run not reproducing it (I ran without the flag).

So this is not a regression in main and not a defect in the tests that failed. The leaked closer is not this repository's code either: .github contains exactly one NamedTemporaryFile (scripts/ci/install_strix_timeout_compat.py:78, inside a with), plus 19 TemporaryDirectory and 2 mkstemp; the finalizer almost certainly belongs to a dependency.

The design decision belongs to this pull request, because -W error and its guard test test_full_suite_command_rejects_warnings are what it adds. Three ways to keep the gate green:

1. -W error -W default::pytest.PytestUnraisableExceptionWarning — keeps every other warning fatal and demotes only the third-party finalizer noise.
2. -p no:unraisableexception — turns the plugin off, which also hides unraisable exceptions that would be ours.
3. -W error::UserWarning — matches exactly what the new test asserts (it emits a UserWarning fixture), but stops rejecting every other warning class.

I recommend (1): the guard test still fails on its UserWarning fixture, the stated intent ("reject warnings") survives for everything the repository controls, and the exclusion is narrow and self-documenting. I have re-run the failed job to see whether the promotion is deterministic or GC-timing dependent, and will push (1) unless the branch's author prefers another option — say so here and I will not touch it.

[seonghobae]

Pushed 9e846138: option (1), with the control run first. The gate command is now

python -m coverage run -m pytest tests -q -W error -W default::pytest.PytestUnraisableExceptionWarning

I reproduced the mechanism myself on Python 3.12 before changing anything, using a fixture whose __del__ raises (CPython cannot propagate that, so it is always unraisable, and pytest's plugin converts it):

arm	command	result
1	no flags	1 passed, 1 warning — the stimulus is real
2	-W error	1 failed — the escalation, reproduced
3	this branch's new command	1 passed, 1 warning — the exemption works
4	a UserWarning under arm 3's flags	1 failed — the branch's own contract survives

Arm 1 matters as much as arm 2: without it a negative result would have meant "no stimulus", not "no escalation". Arm 4 is why this is option (1) and not -W error::UserWarning — the latter also makes the suite green while quietly reducing "reject warnings" to one class, and test_full_suite_command_rejects_warnings would still pass either way. A peer ran an independent three-arm version and got the same result. Because it reproduces on 3.12, this is not Python 3.14 behaviour; the flag is the cause.

The seven guard tests in test_trusted_uv_materializer_quality_workflow_contract.py pass on the new head, and the exemption is commented in place with the run id so the next reader does not have to rediscover why one class is exempt. If the branch author wants the flag left alone and the dependency finalizer chased instead, revert this commit and say so — the evidence above stands either way.

[seonghobae]

Correction to the comment above: option (3) does not work at all, so "both options make the suite green" was wrong. A peer caught it and I confirmed it here:

PytestUnraisableExceptionWarning -> PytestWarning -> UserWarning -> Warning
issubclass(pytest.PytestUnraisableExceptionWarning, UserWarning) is True

So -W error::UserWarning catches the unraisable warning as well. Against the same __del__-raises stimulus: -W error::UserWarning → 1 failed; the committed -W error -W default::pytest.PytestUnraisableExceptionWarning → 1 passed, 1 warning. Option (3) leaves the gate red.

That also changes what arm 4 proves. It is not "which of two working options to prefer" — it is the independent evidence that option (1) keeps the branch's own contract: with the blanket -W error still in place, a plain UserWarning is still fatal, which is exactly what test_full_suite_command_rejects_warnings was written to guarantee.

The pushed commit 9e846138 is unaffected; only my description of the alternative was wrong. Recording why it mattered: I rejected option (3) on an intent argument ("it narrows what the author bought") and never ran it, and the peer who proposed it never ran it either. A rejected option that was never executed comes back the moment someone re-opens the design argument — and this one would have turned the gate red again. Running an option you intend to reject costs a second.

[seonghobae]

Conflict resolved and pushed as 21e128bf (merge of main@5ea1cc47, parents 9e846138 + 5ea1cc47; no rebase, no force-push beyond the lease on the previous head).

One conflict, CHANGELOG.md, and it is the prepend-sibling shape: this branch adds a section at the top and #1958 added one at the same place. Both entries are kept — this branch's "Main quality checks share one runner" stays on top as the not-yet-merged change, with #1958's dispatch-concurrency entry directly below it, which is the order the file will have once this merges. No code conflicts.

Checks for the touched file: both test files that read CHANGELOG.md pass (39 passed), and the workflow contract for the file this branch actually changes passes (7 passed). Conflict markers: zero.

Running note for whoever picks this up: the branch is a draft, so nothing here flips it.

[seonghobae]

Second conflict resolved and pushed as 8761c004 (merge of main@0b0f1047; both changelog entries kept, this branch's on top as the not-yet-merged change, #1957's below it). Same shape as the last one: CHANGELOG.md only, no code conflicts, zero markers. Checks for the touched files: the two test files that read the changelog plus this branch's own workflow contract — 46 passed.

This is the third prepend collision on this branch today (#1953, #1958, #1957 each landed a section at the top while this one sat open). If it keeps costing round trips, the durable fix is to give the file a stable insertion point — a marker line the tooling inserts under — rather than having every branch write to line 1. Not something to change inside this pull request.

[seonghobae]

Fourth resolution today, pushed as 664a5dd9 (merge of main@c232ca03; CHANGELOG.md only, no code conflicts, zero markers, 24 sections with this branch's entry on top). Checks for the touched files: 46 passed.

The four collisions on this one branch, each caused by a different merge landing a section at line 1 while this pull request sat open:

resolved as	invalidated by
21e128bf	#1953
(same push window)	#1958
8761c004	#1957
664a5dd9	#1959

Across the repository the pattern is measurable: 21 of the last 29 commits on main touched this file, and three merges landed within 27 minutes this evening. Every one of these resolutions took the same shape — one file, both entries kept, no code involved — which is the signature of a serialization point rather than a disagreement. A separate pull request is addressing the file's insertion strategy; I am not duplicating that work here, and this branch stays a draft.

[seonghobae]

Fifth resolution, pushed as 2f0729d9 (merge of main@dd0b96fe; CHANGELOG.md only, no code conflicts, zero markers, 25 sections with this branch's entry on top). Checks for the touched files: 46 passed.

Five collisions on this branch today, one per merge that landed while it stayed open: #1953, #1958, #1957, #1959, #1960. Each was the same shape — one file, both entries kept, no code involved.