Conversation
Adds a reactive, backend-agnostic app-crash signal that classifies a scenario's terminal step failure as an app crash (never a poll, never on the run/CI verdict path) and captures the platform's own crash evidence into runs/<run_id>/<sid>/app-crash/. - base.AppCrashedError / base.AppCrashSignal / base.AppCrashPollResettable: narrow, opt-in capability protocols alongside StepOutcome.app_crashed and app_crash_artifacts. - _step_runner.py: a shared _finish_outcome settles every step kind through one seam, backed by three scenario-scoped latches (deliberate-termination, unconfirmed-launch, confirmed-crash) that bound when the probe fires and prevent misclassifying a relaunch or a never-foregrounded launch as a crash. - iOS (XCUITest, Simulator only): a new /app/state runner route reads XCUIApplication.state; a bounded sweep matches the crashed process's .ips report by executable name, udid, and a launch marker re-stamped at all four launch sites (cold spawn, relaunch, crawl reset, warm cross-lease resume). - Android (adb, API 30+): pidof corroborated by a bounded ApplicationExitInfo poll; a logcat crash-buffer extraction (managed and native formats); a root-gated tombstone pull deferred to pipeline.py's post-return scan, since firing adb root mid-scenario would kill the resident channel a later step still needs. - pipeline.py scans every phase's outcomes for the confirmed crash and writes the redacted report; manifest.py excludes the raw-bytes field from the JSON round trip. - crawl integration: the UI-tree heuristic stays the detector; the capture is gated on the driver positively confirming the event, so a covering system alert doesn't pay a full-timeout sweep. - Showcase fixtures (iOS SwiftUI + Android Compose) crash on a launch-env flag; a non-gating app-crash CI lane on both platforms proves the platform mechanisms this item depends on against a real device, asserted via scripts/assert_app_crash_evidence.py. - Bilingual docs (docs/evidence.md, docs/ci.md, docs/architecture.md + docs/ja/ mirrors). Implements all 13 units of BE-0424. A 3-round self-review pass found and fixed 2 critical Android parsing bugs, a protocol-isinstance gap, and several CI/Makefile mechanical issues before this PR opened. make check: 7954 passed, 6 skipped, all coverage floors held. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Refs: BE-0424
Adds the Implementing PR row and Log entry for #2012, both language files, per CLAUDE.md's reciprocal-link requirement. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Refs: BE-0424
…match Addresses three non-blocking findings from the automated PR review on #2012: - `_step_runner._classify_app_crash` called `app_crash_signal()` bare; any exception besides `BackendCrashError` (e.g. `XcuitestDriver`'s `/app/state` raising a plain `XcuitestChannelError` on a truncated reply) escaped `run_scenario` and discarded every scenario's result, not just the one being probed. Now caught and logged, matching the other two `capture_app_crash` call sites' own `except Exception` guard. - `extract_crash_block`'s managed-crash match on `f"Process: {package}"` was an unanchored prefix test, so a sibling package whose name starts with the target's (`com.foo.dev` beside `com.foo`) could have its own crash block misattributed. Anchored on the trailing comma `logcat`'s managed header always renders. - `_exit_info_reason`'s docstring claimed only the newest `dumpsys activity exit-info` entry is read, contradicting the fall-through loop beneath it and this PR's own `test_newest_exit_info_skips_a_block_whose_reason_names_no_known_token`. Corrected to say what the code (and the caller's time-bound guard) actually do. Also records a newly confirmed `app-crash (xcuitest)` failure pattern in investigate-ci-failure's reference: rebuilding the showcase app and XCUITest runner from this commit and driving `app_crash.yaml` against a freshly booted, uncontended Simulator reproduces the correct crash classification every time, ruling out a code defect; the failing CI run's own log shows a concurrent `video-no-bytes` render-service stall, the same host fault class as pattern #9, here silently dropping the crash-triggering tap's delivery instead of blocking a screenshot or launch. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Refs: BE-0424
Job 105041168033 (run 35164359245, attempt 3) classified the app-under-test crash correctly (app_crashed: true) but its .ips sweep came up empty, so assert_app_crash_evidence.py failed on a missing app-crash/ directory — a different signature from the already-catalogued pattern #11 for this job. A concurrent video-no-bytes stall in the same run's log is same-run evidence of host contention, the class already confirmed for #9/#11, so this is recorded as pattern #12 rather than treated as a code defect. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Refs: BE-0424
Member
Author
|
@claude review |
The XCUITest app-crash job (BE-0424) failed 2 of 2 CI attempts on commit 07c117a, both with the same symptom: the crash was classified correctly (app_crashed: true in manifest.json) but scripts/assert_app_crash_evidence.py still failed on "no app-crash/ directory", because XcuitestEnvironment._collect_app_crash_reports found no matching .ips within _APP_CRASH_REPORT_TIMEOUT (5.0s). A local reproduction against a freshly built app/runner on an uncontended Simulator captures the report every time, and the third CI attempt showed no concurrent host stall at all (unlike the first two, see the reference-pattern doc update in 07c117a) — ruling out both a code defect in the sweep itself and, for that third attempt, an unrelated render-service wedge. The remaining explanation is that ReportCrash simply takes longer than 5s to finish writing the .ips under a shared GitHub Actions runner than on a dedicated dev Mac. Widened the bound to 15s; it is a condition wait; not a fixed sleep, so the common case (report already present) pays nothing extra. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Refs: BE-0424
…h splice The app-crash/BE-0424 paragraph inserted earlier in this PR sits between the crash-diagnostics discussion and the sentence that resumes it, so "The runner"/「ランナーの出力を集めるのは」 now reads as continuing the app-crash paragraph instead. Name crash-diagnostics/ again to re-anchor, per the live Claude review's suggestions on both docs/ci.md and docs/ja/ci.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Refs: BE-0424
Member
Author
|
@claude review |
A wrapping if/forEach outcome's reason is already the inner step's own finished failure string (copied verbatim by _run_if/_run_for_each), which already carries "— the app under test crashed: <signal>" once _classify_app_crash confirmed it further in. The "already confirmed" branch appended the note again unconditionally, so a nested if -> forEach -> tap crash repeated it once per wrapping level. Guard on the note already being present before appending it, and add regression assertions that it appears exactly once for both the flat and nested cases. Also fold two outstanding review-comment fixes into this same push: reword known-ci-failure-patterns.md row #12 to measure a recurrence against the current 15s _APP_CRASH_REPORT_TIMEOUT bound instead of the 5s one this PR already raised, record today's CI recurrence (run 35174153612) as a fresh confirmation at that bound, and fix two forced-translation/inconsistent terms in the ja roadmap doc's changelog entry. Recording today's investigation also adds a new e2e-known-flake pattern (#13: a GitHub Actions hashFiles() timeout on codegen (xcuitest), unrelated to any code here) and updates SKILL.md's own step 3/5 wording so a future session is actually pointed at all three catalog tables. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Refs: BE-0424
Member
Author
|
@claude review |
…424-d849c1 # Conflicts: # BajutsuKit/Tests/BajutsuRunnerTests/FakeElementProvider.swift # bajutsu/common/drivers/xcuitest/xcuitest_driver.py # demos/showcase/ios/swiftui/Sources/AppModel.swift # demos/showcase/ios/swiftui/Sources/RootView.swift
_restore_unroot() ran `adb unroot` on every path out of app_crash_tombstone(), with no record of whether adbd was already root when _pull_tombstone arrived. demos/showcase/android/Makefile's e2e / e2e-conformance targets both root the device for a whole session (gestures requires it), so an app crash mid-run would drop that root device-wide for every later scenario in the same make invocation. Remember the privilege level found before rooting and skip the restore when it was already root. Also add test coverage for scripts/assert_app_crash_evidence.py's violations(), the sole verdict for the app-crash (xcuitest) / app-crash (adb) CI jobs (both discard their wrapped run's exit code on purpose), which previously shipped with none. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Refs: BE-0424
「声高に失敗させる」was a direct rendering of the English "failing loudly" whose object binding reads wrong in Japanese, and "声高に" itself describes loudness of speech, not a test failure. Match the term this repo's Japanese docs already use for the same concept elsewhere (docs/ja/recording.md, docs/ja/scenarios.md, docs/ja/configuration.md). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Refs: BE-0424
… job actuation (xcuitest) (run 35185561508, job 105090800802) failed on a plain wait timeout for an unrelated element (horse.title), with two recordVideo/video-no-bytes stalls and a Simulator reboot after a failed cold runner spawn in its log — the same render/window-server host fault class pattern 9 already names, but the first confirmation on this PR of the generic pattern itself rather than one of its app-crash-specific derivatives (#11/#12). Reran the job. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Refs: BE-0424
app-crash (xcuitest) has failed twice cleanly (no concurrent video-no-bytes stall) with the same "no app-crash/ directory" symptom pattern #12 already anticipates as evidence the 15s _APP_CRASH_REPORT_TIMEOUT bound may itself be too tight — but confirming that needs host telemetry this job never uploaded. Its --runs-dir pointed at tmp/showcase-app-crash-runs instead of runs/, so BE-0361's runs/diagnostics/ sweep (which every sibling job's existing runs/ upload rides for free) never picked it up, and the job called collect-ios-diagnostics nowhere at all. Move the scenario run tree to runs/app-crash (still fresh-removed per run, still gitignored the same as tmp/ — gitleaks dir . sweeps both regardless of .gitignore, so that reasoning never actually depended on the tmp/ prefix), add the Start/Collect iOS diagnostics steps every other raw-bajutsu-run job already has, and widen the upload to runs/ so stall-NN-*, render-probe.txt, and host-telemetry.log ride along automatically. A future no-stall recurrence now has real evidence to diagnose the timeout against instead of the log-line-only confirmation the last two occurrences were limited to. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Refs: BE-0424
…424-d849c1 # Conflicts: # .apm/skills/investigate-ci-failure/references/known-ci-failure-patterns.md
… 30s Four app-crash (xcuitest) CI recurrences after BE-0424's diagnostics-gap fix all carried direct evidence of host/render-service contention (render-probe.txt's screenshot-probe kill, a stall capture's backboardd sample timeout) recorded before any scenario ran. Widening the budget cannot fix a fully wedged render pipeline, but gives ReportCrash more room on a host that recovers within tens of seconds rather than staying wedged for the probe's own window. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Refs: BE-0424
Analyze (java-kotlin) on PR #2012 (run 35250805835, job 105302464362) failed Gradle dependency resolution with a blanket 403 Forbidden from both repo.maven.apache.org and plugins.gradle.org, across every artifact the Android showcase build needs. The PR's diff touches no Android/Gradle config, so a code regression cannot explain every coordinate being rejected by the registry at once — an external registry/network fault, not a code defect. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Refs: BE-0424
… download smoke (adb) on PR #2012 (run 35252166515, job 105307465870) failed before any scenario script ran: sdkmanager's system-images install logged "Error on ZipFile unknown archive" for the google_apis x86_64 image, and the emulator step then failed outright with the emulator never listening on port 5554. A corrupted download from Google's SDK package repository, not this repository's code — folded into the existing "emulator never comes up" pattern (#7). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Refs: BE-0424
…n a scroll-distance test conformance (adb) on PR #2012 (run 35253562344, job 105311875152) failed test_one_scroll_step_travels_the_distance_it_was_asked_for with the same WARNING bajutsu.adb.resident: read lag signature pattern #14 already names, this time surfacing through a direct travel-distance measurement rather than an expect assertion, with a widening lag across the test's three measurements. The PR's adb_driver.py diff in this window is pure addition (BE-0424 crash-signal probing) and none of it touches the resident-read or scroll-actuation path. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Refs: BE-0424
…ccurrence PR #2012 run 35253562594 reconfirms pattern #9 far more severely than before: app-crash (xcuitest) shows a failed cold runner spawn forcing a Simulator reboot, a recordVideo that never started, a mid-run runner exit (code 65), and a render-probe watchdog with a 15s kill bound that doesn't report the kill until 26s — the bash polling loop itself running behind, not just the render service. actuation (xcuitest) failed in the same run ~13 minutes later with the same video-no-bytes signature. Cross-job corroboration, within one run, of a degraded batch of assigned runners rather than one unlucky VM. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Refs: BE-0424
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements BE-0424: when the app under test crashes mid-scenario,
bajutsu runnow fails the step with a message that names the event, and writes the platform's own crash report intoruns/<run_id>/<sid>/app-crash/— instead of the genericElementNotFounda contributor previously had to rule out by elimination. This PR ships all 13 units of the item and moves its Status to Implemented in bothroadmaps/BE-0424-app-crash-diagnostics/BE-0424-app-crash-diagnostics.mdand its Japanese mirror.The detection is reactive — a driver is only ever asked once a step has already failed, never polled — so a green run pays nothing for a failure mode that is rare by construction. Classification happens in-band, inside the step loop itself, so it never touches the
run/CI verdict path and never triggers the existing backend-crash retry (that path is for infrastructure faults; this one is a likely defect in the app being tested).What changed
bajutsu/common/drivers/base/(new:app_crashed_error.py,app_crash_signal.py,app_crash_poll_resettable.py) — a narrow, opt-inAppCrashSignalcapability protocol (mirroringInterruptionPolicyTarget/SettledCacheInvalidator's existing shape) so only XCUITest and adb implement it, with no stub required onDriver's other implementers.bajutsu/common/orchestrator/loop/_step_runner.py,app_crash_latches.py(new) — a shared_finish_outcomesettles every step kind (including the one early-return exit an earlier draft is documented to have missed) through one seam, backed by three scenario-scoped latches that bound when the probe fires: a deliberaterelaunchtermination, an unconfirmed app launch (the bare-count readiness rung can't tell the app from SpringBoard), and a confirmed crash (so at most one outcome per scenario ever carries the classification).bajutsu/common/drivers/xcuitest/,bajutsu/common/platform_lifecycle/environments/xcuitest/) — a newPOST /app/staterunner route readsXCUIApplication.state; scoped to the Simulator only (a real device'snotRunningcan mean an OS memory-pressure kill instead). A bounded sweep matches the crashed process's.ipsreport by executable name (read fromInfo.plist) and Simulator udid (parsed from the report's own payload), against a launch marker re-stamped at all four launch sites — cold spawn, arelaunchstep, a crawl frontier reset, and BE-0291's cross-lease warm resume.bajutsu/common/drivers/adb/,bajutsu/common/platform_lifecycle/environments/android/) —pidofcorroborated by a boundedApplicationExitInfopoll (API 30+ only); alogcatcrash-buffer extraction covering both managed and native crash formats; a root-gated tombstone pull deliberately deferred topipeline.py's post-return scan, since firingadb rootmid-scenario would kill the resident channel a later step still needs.bajutsu/common/runner/pipeline.py,bajutsu/common/report/manifest.py— a post-return scan over every phase's outcomes finds the confirmed crash and writes its (already-captured) evidence through the redacting text path; the raw-bytesfield is excluded from the manifest's JSON round trip the same waywall_offset_salready is.bajutsu/crawl/— the UI-tree heuristic stays the detector unchanged; the artifact capture is gated on the driver positively confirming the event, so a covering system alert doesn't pay a full-timeout sweep on every false positive.demos/showcase/ios/swiftui/Sources/CrashView.swift,demos/showcase/android/compose/.../CrashScreen.kt,demos/showcase/scenarios/app_crash.yaml, new) — a launch-env-gated affordance that faults the app on demand, plusscripts/assert_app_crash_evidence.pyto check the finished run's manifest and itsapp-crash/directory..github/workflows/ios-e2e.yml,android-e2e.yml) — a non-gatingapp-crashjob on each lane, proving the platform mechanisms (ReportCrash,logcat's crash buffer,ApplicationExitInfo) this item depends on against a real Simulator/emulator, the same path every other new signal on these lanes has taken before promotion.docs/evidence.md,docs/ci.md,docs/architecture.md+docs/ja/mirrors) — the new evidence kind, including its two documented and deliberately-accepted blind spots (a crash on a scenario's own last step with nothing declared after it; the same shape with only a scenario-levelexpectto surface it, which lands red but unclassified).Scope
The web (Playwright) backend is explicitly left to a follow-up item —
PlaywrightDriverimplements noAppCrashSignal, so the check is a no-op there by construction, as the roadmap item's own Introduction scopes it.Prime-directive compliance
XCUIApplication.state,pidof/ApplicationExitInfo), never inferred or guessed.base.deadline_ticks(the.ipssweep, the exit-info poll, thelogcatre-dump); no fixedsleepanywhere in the new code. An ambiguous or missing selector still fails the step immediately.targets.<name>config, never baked into the deterministic core;pipeline.py's tombstone call is deliberately unconditional rather than branching onresult.backend.Verification
166 new tests across
tests/orchestrator/test_app_crash.py,tests/test_app_crash_drivers.py,tests/test_app_crash_environments.py,tests/runner/test_app_crash_pipeline.py,tests/test_crawl_app_crash.py, plus additions totests/test_crawl_repro.pyandtests/runner/test_platform_lifecycle_seam.py, cover the step-loop classification and all three latches, the iOS.ipssweep and launch-marker ordering at all four launch sites, the Android signal (including tolerant exit-info reason parsing, since the exact AOSP rendering could not be verified against a live device — the parser matches known reason tokens as substrings rather than betting on one exact format), the pipeline write and manifest exclusion, and the crawl gate.Not exercised by this gate: the Swift (
BajutsuKit/) and Kotlin (showcase app) changes could not be compiled or run in this session (no Xcode/Gradle toolchain in the environment they were written in) — the newapp-crash (xcuitest)/app-crash (adb)CI jobs on this PR are the first real verification of the runner route, the showcase affordances, and the on-device platform mechanisms (ReportCrash,logcat,ApplicationExitInfo) this item depends on.Notes
This PR went through implement-be's 3-round self-review cap before opening. Round 1 found and fixed two correctness bugs (an Android exit-info parser field-order assumption that didn't match real
dumpsysoutput order, and alogcatcrash-block match window that could absorb a neighboring process's crash) plus a protocol-isinstance gap. Round 2, re-verifying round 1, found four new blocking issues the first pass missed — including that the exit-info reason-token format still couldn't be confirmed without a live device, so the parser was redesigned to match known tokens tolerantly rather than betting on one exact rendering — plus CI/script bugs that would have made both newapp-crashjobs permanently red (a nonexistent--run-idCLI flag, a wrong manifest key, a stale Swift route-count assertion). Round 3 found only mechanical CI/Makefile issues (all fixed): an untracked run directory the new jobs would have left at the repo root, a shell guard blind spot, missinguv syncsteps, and a Gradle build running inside the timed emulator-boot window. Per the skill's 3-round cap, a 4th review round was not run.Two small, genuinely unrelated issues surfaced during review and were filed as separate follow-ups rather than fixed here: a pre-existing test-isolation flake in
tests/test_xcuitest_bundled_runner.pythat surfaces only under the full suite's parallel worker scheduling, and a stale route-count comment in a Swift file this PR's diff never touches.🤖 Generated with Claude Code