Repository navigation
Report Security run preflight failures without claiming the audit ran #740
Description
This was generated by AI during triage.
A Security run can fail in its GitHub gates before starting a Security audit, yet its Run notification calls that an audit failure and supplies no cause or diagnostic pointer.
Observed incident
The reported October 16, 2026, approximately 10:09 p.m. EDT notification from host stephens said:
Result: audit failed
Audit: audit failed
Built with codex · gpt-6.1-sol · high (configured)
Host: stephens
Took: 0s
At 2026-10-17 02:09:11 UTC the scheduler log records a Security run starting. At 02:09:12 it records gh issue list failed: GraphQL: API rate limit already exceeded for user ID <REDACTED>. The same pre-audit failure pattern recurs around this time. This is a GitHub dependency failure; the configured Harness/Model line does not establish that a model session ran or failed.
Evidence: ~/.thirdshift/logs/cron.log, lines 327710–327711 on stephens at inspection. Matching emails retrieved from the surrounding cluster include 02:08:12 and 02:10:12 UTC; the exact 02:09 email was not retrieved, so the scheduler record is the evidence for that minute.
Source verification
The Security run's Ready issue query runs before the audit. A query error is converted into an audited outcome; ending accounting labels a failure without a reproduction number audit failed. Notification formatting suppresses all Security failure causes except fixed safeguard-refusal categories. This privacy rule is intentional, but leaves this operational failure unexplained. Command logging starts only after the gates, explaining the absence of Command/Session log pointers.
The existing failed_security_audits_send_one_notification_with_a_safe_status scenario includes a failing pre-audit issue list and explicitly expects both misleading audit-failure wording and no Cause line. It provides an isolated reproduction of the reporting defect with fake GitHub and Resend, rather than exhausting a live account.
Scope and related work
Report the pre-audit stage and a safe operational cause, with a usable diagnostic route. Preserve private Security finding and provider-diagnostic filtering. GitHub cooldowns, idle polling, and repeated-alert policy belong to #739. Closed #603 separates reproduction failures after a completed audit; it does not cover this earlier stage. Tracker/source searches found no separate implementation or issue for pre-audit reporting, and no .out-of-scope/ rejection directory is present.
Activity
JacobStephens2 commentedon Oct 17, 2026
Agent Brief
Category: bug
Summary: Accurately report a Security run blocked before its Security audit starts, with a safe operational cause and a usable diagnostic route.
Current behavior:
A failing GitHub gate before the Security audit is accounted as
audit failed. The Run notification repeats that status for Audit, suppresses the cause, and has no Command or Session log pointer because work has not started. In the reported incident GitHub GraphQL quota exhaustion caused this result in about a second. Existing scenario coverage explicitly accepts this misleading output.Desired behavior:
Retain enough stage information to distinguish a Security run that could not start its audit from a failure during an audit or subsequent reproduction. For a failed pre-audit GitHub query, the subject and Result identify the pre-audit failure and Audit says
not started. Include a fixed or otherwise explicitly sanitized operational cause; confirmed GitHub quota exhaustion must be recognizable as such. Unknown GitHub failures use a generic safe dependency-failure category rather than raw diagnostics. Supply a usable diagnostic route even when no Command or Session log exists, such as a host/time-stamped stderr or scheduler-log instruction. Do not invent a log path or require the default cron layout. Keep the overall failure nonzero and do not treat a failed query as no available work or a successful skip.Key interfaces:
Acceptance criteria:
not started; a safe Cause identifies GitHub rate-limit exhaustion. The configured Harness/Model is not presented as observed session execution.Out of scope:
Quota cooldowns, retry scheduling, idle-polling optimization and repeated-alert policy (#739); changing the user's crontab; altering Security findings, fix dispatch or safeguards; broadening disclosure of private diagnostics; changing post-audit recording-stage semantics.
Verification performed during triage:
cargo test --test security_run failed_security_audits_send_one_notification_with_a_safe_status -- --exactpassed (1 test, 0 failures; scenario execution 3.59 seconds). Its controlled pre-audit GitHub failure confirms the current misleading status and omitted Cause; passing here reproduces the defect because the existing assertions encode that behavior. This did not reproduce live quota exhaustion or test a fix. The incident's rate-limit cause is established separately by the scheduler log.Report Security run pre-audit failures truthfully (#740)
JacobStephens2 commentedon Oct 19, 2026
Correction: this comment originally put the occurrence below under #740, but it was a scheduled Architect run, not a Security run. In the cron log, the Security run started at 00:02:12 UTC and the seven Architect runs at 00:02:21–22. That matches the notification's
Took: 12sand itsResult: review failed / Review: failedshape, which only Architect runs produce. So this is the #739 shape (an Architect run failing before its Architecture review starts), caused here by a transient GitHub 5xx rather than quota. PR #767 does not cover it.Occurrence:
stephens, 2026-10-19 00:02:34 UTC,Cause: gh issue list failed: HTTP 504: 504 Gateway Timeout (https://api.github.com/graphql). Retrying the read is filed as #786.