Skip to content

[BE-0424] feat(driver): detect an app-under-test crash and capture its report - #2012

Draft
0x0c wants to merge 18 commits into
mainfrom
claude/implement-be-0424-d849c1
Draft

0x0c wants to merge 18 commits into
mainfrom
claude/implement-be-0424-d849c1

Conversation

@0x0c

@0x0c 0x0c commented Sep 16, 2026

Copy link
Copy Markdown
Member

Summary

Implements BE-0424: when the app under test crashes mid-scenario, bajutsu run now fails the step with a message that names the event, and writes the platform's own crash report into runs/<run_id>/<sid>/app-crash/ — instead of the generic ElementNotFound a contributor previously had to rule out by elimination. This PR ships all 13 units of the item and moves its Status to Implemented in both roadmaps/BE-0424-app-crash-diagnostics/BE-0424-app-crash-diagnostics.md and its Japanese mirror.

The detection is reactive — a driver is only ever asked once a step has already failed, never polled — so a green run pays nothing for a failure mode that is rare by construction. Classification happens in-band, inside the step loop itself, so it never touches the run/CI verdict path and never triggers the existing backend-crash retry (that path is for infrastructure faults; this one is a likely defect in the app being tested).

What changed

  • bajutsu/common/drivers/base/ (new: app_crashed_error.py, app_crash_signal.py, app_crash_poll_resettable.py) — a narrow, opt-in AppCrashSignal capability protocol (mirroring InterruptionPolicyTarget/SettledCacheInvalidator's existing shape) so only XCUITest and adb implement it, with no stub required on Driver's other implementers.
  • bajutsu/common/orchestrator/loop/_step_runner.py, app_crash_latches.py (new) — a shared _finish_outcome settles every step kind (including the one early-return exit an earlier draft is documented to have missed) through one seam, backed by three scenario-scoped latches that bound when the probe fires: a deliberate relaunch termination, an unconfirmed app launch (the bare-count readiness rung can't tell the app from SpringBoard), and a confirmed crash (so at most one outcome per scenario ever carries the classification).
  • iOS (bajutsu/common/drivers/xcuitest/, bajutsu/common/platform_lifecycle/environments/xcuitest/) — a new POST /app/state runner route reads XCUIApplication.state; scoped to the Simulator only (a real device's notRunning can mean an OS memory-pressure kill instead). A bounded sweep matches the crashed process's .ips report by executable name (read from Info.plist) and Simulator udid (parsed from the report's own payload), against a launch marker re-stamped at all four launch sites — cold spawn, a relaunch step, a crawl frontier reset, and BE-0291's cross-lease warm resume.
  • Android (bajutsu/common/drivers/adb/, bajutsu/common/platform_lifecycle/environments/android/) — pidof corroborated by a bounded ApplicationExitInfo poll (API 30+ only); a logcat crash-buffer extraction covering both managed and native crash formats; a root-gated tombstone pull deliberately deferred to pipeline.py's post-return scan, since firing adb root mid-scenario would kill the resident channel a later step still needs.
  • bajutsu/common/runner/pipeline.py, bajutsu/common/report/manifest.py — a post-return scan over every phase's outcomes finds the confirmed crash and writes its (already-captured) evidence through the redacting text path; the raw-bytes field is excluded from the manifest's JSON round trip the same way wall_offset_s already is.
  • bajutsu/crawl/ — the UI-tree heuristic stays the detector unchanged; the artifact capture is gated on the driver positively confirming the event, so a covering system alert doesn't pay a full-timeout sweep on every false positive.
  • Showcase fixtures (demos/showcase/ios/swiftui/Sources/CrashView.swift, demos/showcase/android/compose/.../CrashScreen.kt, demos/showcase/scenarios/app_crash.yaml, new) — a launch-env-gated affordance that faults the app on demand, plus scripts/assert_app_crash_evidence.py to check the finished run's manifest and its app-crash/ directory.
  • CI (.github/workflows/ios-e2e.yml, android-e2e.yml) — a non-gating app-crash job on each lane, proving the platform mechanisms (ReportCrash, logcat's crash buffer, ApplicationExitInfo) this item depends on against a real Simulator/emulator, the same path every other new signal on these lanes has taken before promotion.
  • Docs (docs/evidence.md, docs/ci.md, docs/architecture.md + docs/ja/ mirrors) — the new evidence kind, including its two documented and deliberately-accepted blind spots (a crash on a scenario's own last step with nothing declared after it; the same shape with only a scenario-level expect to surface it, which lands red but unclassified).

Scope

The web (Playwright) backend is explicitly left to a follow-up item — PlaywrightDriver implements no AppCrashSignal, so the check is a no-op there by construction, as the roadmap item's own Introduction scopes it.

Prime-directive compliance

  • AI never judges — no LLM anywhere on this path; classification is a driver-level positive confirmation (XCUIApplication.state, pidof/ApplicationExitInfo), never inferred or guessed.
  • Determinism first — every wait is a condition wait through base.deadline_ticks (the .ips sweep, the exit-info poll, the logcat re-dump); no fixed sleep anywhere in the new code. An ambiguous or missing selector still fails the step immediately.
  • App-agnostic — the package name and platform-specific signals are threaded as plain constructor keywords from targets.<name> config, never baked into the deterministic core; pipeline.py's tombstone call is deliberately unconditional rather than branching on result.backend.

Verification

make check
  7954 passed, 6 skipped, 298 warnings
  Total coverage: 95.01% (floor 93.0%)
  coverage-floors: 521 file(s) at or above their recorded floor.

166 new tests across tests/orchestrator/test_app_crash.py, tests/test_app_crash_drivers.py, tests/test_app_crash_environments.py, tests/runner/test_app_crash_pipeline.py, tests/test_crawl_app_crash.py, plus additions to tests/test_crawl_repro.py and tests/runner/test_platform_lifecycle_seam.py, cover the step-loop classification and all three latches, the iOS .ips sweep and launch-marker ordering at all four launch sites, the Android signal (including tolerant exit-info reason parsing, since the exact AOSP rendering could not be verified against a live device — the parser matches known reason tokens as substrings rather than betting on one exact format), the pipeline write and manifest exclusion, and the crawl gate.

Not exercised by this gate: the Swift (BajutsuKit/) and Kotlin (showcase app) changes could not be compiled or run in this session (no Xcode/Gradle toolchain in the environment they were written in) — the new app-crash (xcuitest) / app-crash (adb) CI jobs on this PR are the first real verification of the runner route, the showcase affordances, and the on-device platform mechanisms (ReportCrash, logcat, ApplicationExitInfo) this item depends on.

Notes

This PR went through implement-be's 3-round self-review cap before opening. Round 1 found and fixed two correctness bugs (an Android exit-info parser field-order assumption that didn't match real dumpsys output order, and a logcat crash-block match window that could absorb a neighboring process's crash) plus a protocol-isinstance gap. Round 2, re-verifying round 1, found four new blocking issues the first pass missed — including that the exit-info reason-token format still couldn't be confirmed without a live device, so the parser was redesigned to match known tokens tolerantly rather than betting on one exact rendering — plus CI/script bugs that would have made both new app-crash jobs permanently red (a nonexistent --run-id CLI flag, a wrong manifest key, a stale Swift route-count assertion). Round 3 found only mechanical CI/Makefile issues (all fixed): an untracked run directory the new jobs would have left at the repo root, a shell guard blind spot, missing uv sync steps, and a Gradle build running inside the timed emulator-boot window. Per the skill's 3-round cap, a 4th review round was not run.

Two small, genuinely unrelated issues surfaced during review and were filed as separate follow-ups rather than fixed here: a pre-existing test-isolation flake in tests/test_xcuitest_bundled_runner.py that surfaces only under the full suite's parallel worker scheduling, and a stale route-count comment in a Swift file this PR's diff never touches.

🤖 Generated with Claude Code

Adds a reactive, backend-agnostic app-crash signal that classifies a
scenario's terminal step failure as an app crash (never a poll, never
on the run/CI verdict path) and captures the platform's own crash
evidence into runs/<run_id>/<sid>/app-crash/.

- base.AppCrashedError / base.AppCrashSignal / base.AppCrashPollResettable:
  narrow, opt-in capability protocols alongside StepOutcome.app_crashed
  and app_crash_artifacts.
- _step_runner.py: a shared _finish_outcome settles every step kind
  through one seam, backed by three scenario-scoped latches
  (deliberate-termination, unconfirmed-launch, confirmed-crash) that
  bound when the probe fires and prevent misclassifying a relaunch or
  a never-foregrounded launch as a crash.
- iOS (XCUITest, Simulator only): a new /app/state runner route reads
  XCUIApplication.state; a bounded sweep matches the crashed process's
  .ips report by executable name, udid, and a launch marker re-stamped
  at all four launch sites (cold spawn, relaunch, crawl reset, warm
  cross-lease resume).
- Android (adb, API 30+): pidof corroborated by a bounded
  ApplicationExitInfo poll; a logcat crash-buffer extraction (managed
  and native formats); a root-gated tombstone pull deferred to
  pipeline.py's post-return scan, since firing adb root mid-scenario
  would kill the resident channel a later step still needs.
- pipeline.py scans every phase's outcomes for the confirmed crash and
  writes the redacted report; manifest.py excludes the raw-bytes field
  from the JSON round trip.
- crawl integration: the UI-tree heuristic stays the detector; the
  capture is gated on the driver positively confirming the event, so
  a covering system alert doesn't pay a full-timeout sweep.
- Showcase fixtures (iOS SwiftUI + Android Compose) crash on a
  launch-env flag; a non-gating app-crash CI lane on both platforms
  proves the platform mechanisms this item depends on against a real
  device, asserted via scripts/assert_app_crash_evidence.py.
- Bilingual docs (docs/evidence.md, docs/ci.md, docs/architecture.md
  + docs/ja/ mirrors).

Implements all 13 units of BE-0424. A 3-round self-review pass found
and fixed 2 critical Android parsing bugs, a protocol-isinstance gap,
and several CI/Makefile mechanical issues before this PR opened.

make check: 7954 passed, 6 skipped, all coverage floors held.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Refs: BE-0424
@0x0c
0x0c deployed to claude-review September 16, 2026 11:24 — with GitHub Actions Active
@github-actions github-actions Bot added topic:contribution topic:dogfood Dogfood fixtures (demo apps) (roadmap topic) topic:platform Platform expansion (Android / Web / Flutter) (roadmap topic) labels Sep 16, 2026
Comment thread bajutsu/common/drivers/base/app_crash_signal.py
Comment thread bajutsu/common/drivers/base/app_crash_poll_resettable.py
Adds the Implementing PR row and Log entry for #2012, both language
files, per CLAUDE.md's reciprocal-link requirement.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Refs: BE-0424
Comment thread bajutsu/common/orchestrator/loop/_step_runner.py Outdated
Comment thread bajutsu/common/backend_cli/adb/_functions.py Outdated
Comment thread bajutsu/common/backend_cli/adb/_functions.py Outdated
T and others added 2 commits September 17, 2026 08:53
…match

Addresses three non-blocking findings from the automated PR review on #2012:

- `_step_runner._classify_app_crash` called `app_crash_signal()` bare; any
  exception besides `BackendCrashError` (e.g. `XcuitestDriver`'s `/app/state`
  raising a plain `XcuitestChannelError` on a truncated reply) escaped
  `run_scenario` and discarded every scenario's result, not just the one being
  probed. Now caught and logged, matching the other two `capture_app_crash`
  call sites' own `except Exception` guard.
- `extract_crash_block`'s managed-crash match on `f"Process: {package}"` was
  an unanchored prefix test, so a sibling package whose name starts with the
  target's (`com.foo.dev` beside `com.foo`) could have its own crash block
  misattributed. Anchored on the trailing comma `logcat`'s managed header
  always renders.
- `_exit_info_reason`'s docstring claimed only the newest `dumpsys activity
  exit-info` entry is read, contradicting the fall-through loop beneath it
  and this PR's own `test_newest_exit_info_skips_a_block_whose_reason_names_no_known_token`.
  Corrected to say what the code (and the caller's time-bound guard) actually
  do.

Also records a newly confirmed `app-crash (xcuitest)` failure pattern in
investigate-ci-failure's reference: rebuilding the showcase app and XCUITest
runner from this commit and driving `app_crash.yaml` against a freshly booted,
uncontended Simulator reproduces the correct crash classification every time,
ruling out a code defect; the failing CI run's own log shows a concurrent
`video-no-bytes` render-service stall, the same host fault class as pattern
#9, here silently dropping the crash-triggering tap's delivery instead of
blocking a screenshot or launch.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Refs: BE-0424
Job 105041168033 (run 35164359245, attempt 3) classified the app-under-test
crash correctly (app_crashed: true) but its .ips sweep came up empty, so
assert_app_crash_evidence.py failed on a missing app-crash/ directory — a
different signature from the already-catalogued pattern #11 for this job.
A concurrent video-no-bytes stall in the same run's log is same-run evidence
of host contention, the class already confirmed for #9/#11, so this is
recorded as pattern #12 rather than treated as a code defect.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Refs: BE-0424
@0x0c

0x0c commented Sep 17, 2026

Copy link
Copy Markdown
Member Author

@claude review

The XCUITest app-crash job (BE-0424) failed 2 of 2 CI attempts on commit
07c117a, both with the same symptom: the crash was classified correctly
(app_crashed: true in manifest.json) but scripts/assert_app_crash_evidence.py
still failed on "no app-crash/ directory", because
XcuitestEnvironment._collect_app_crash_reports found no matching .ips within
_APP_CRASH_REPORT_TIMEOUT (5.0s).

A local reproduction against a freshly built app/runner on an uncontended
Simulator captures the report every time, and the third CI attempt showed no
concurrent host stall at all (unlike the first two, see the reference-pattern
doc update in 07c117a) — ruling out both a code defect in the sweep itself
and, for that third attempt, an unrelated render-service wedge. The remaining
explanation is that ReportCrash simply takes longer than 5s to finish writing
the .ips under a shared GitHub Actions runner than on a dedicated dev Mac.
Widened the bound to 15s; it is a condition wait; not a fixed sleep, so the
common case (report already present) pays nothing extra.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Refs: BE-0424
Comment thread docs/ci.md Outdated
Comment thread docs/ja/ci.md Outdated
…h splice

The app-crash/BE-0424 paragraph inserted earlier in this PR sits between the
crash-diagnostics discussion and the sentence that resumes it, so "The
runner"/「ランナーの出力を集めるのは」 now reads as continuing the app-crash
paragraph instead. Name crash-diagnostics/ again to re-anchor, per the live
Claude review's suggestions on both docs/ci.md and docs/ja/ci.md.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Refs: BE-0424
@0x0c

0x0c commented Sep 17, 2026

Copy link
Copy Markdown
Member Author

@claude review

Comment thread bajutsu/common/orchestrator/loop/_step_runner.py Outdated
Comment thread .apm/skills/investigate-ci-failure/references/known-ci-failure-patterns.md Outdated
Comment thread roadmaps/BE-0424-app-crash-diagnostics/BE-0424-app-crash-diagnostics-ja.md Outdated
A wrapping if/forEach outcome's reason is already the inner step's own
finished failure string (copied verbatim by _run_if/_run_for_each), which
already carries "— the app under test crashed: <signal>" once
_classify_app_crash confirmed it further in. The "already confirmed"
branch appended the note again unconditionally, so a nested if -> forEach
-> tap crash repeated it once per wrapping level. Guard on the note
already being present before appending it, and add regression assertions
that it appears exactly once for both the flat and nested cases.

Also fold two outstanding review-comment fixes into this same push:
reword known-ci-failure-patterns.md row #12 to measure a recurrence
against the current 15s _APP_CRASH_REPORT_TIMEOUT bound instead of the
5s one this PR already raised, record today's CI recurrence (run
35174153612) as a fresh confirmation at that bound, and fix two
forced-translation/inconsistent terms in the ja roadmap doc's changelog
entry. Recording today's investigation also adds a new e2e-known-flake
pattern (#13: a GitHub Actions hashFiles() timeout on codegen
(xcuitest), unrelated to any code here) and updates SKILL.md's own step
3/5 wording so a future session is actually pointed at all three
catalog tables.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Refs: BE-0424
@0x0c

0x0c commented Sep 17, 2026

Copy link
Copy Markdown
Member Author

@claude review

…424-d849c1

# Conflicts:
#	BajutsuKit/Tests/BajutsuRunnerTests/FakeElementProvider.swift
#	bajutsu/common/drivers/xcuitest/xcuitest_driver.py
#	demos/showcase/ios/swiftui/Sources/AppModel.swift
#	demos/showcase/ios/swiftui/Sources/RootView.swift
Comment thread scripts/assert_app_crash_evidence.py
Comment thread docs/ja/architecture.md
T and others added 3 commits September 17, 2026 14:14
_restore_unroot() ran `adb unroot` on every path out of app_crash_tombstone(),
with no record of whether adbd was already root when _pull_tombstone arrived.
demos/showcase/android/Makefile's e2e / e2e-conformance targets both root the
device for a whole session (gestures requires it), so an app crash mid-run
would drop that root device-wide for every later scenario in the same make
invocation. Remember the privilege level found before rooting and skip the
restore when it was already root.

Also add test coverage for scripts/assert_app_crash_evidence.py's violations(),
the sole verdict for the app-crash (xcuitest) / app-crash (adb) CI jobs (both
discard their wrapped run's exit code on purpose), which previously shipped
with none.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Refs: BE-0424
「声高に失敗させる」was a direct rendering of the English "failing loudly"
whose object binding reads wrong in Japanese, and "声高に" itself describes
loudness of speech, not a test failure. Match the term this repo's Japanese
docs already use for the same concept elsewhere (docs/ja/recording.md,
docs/ja/scenarios.md, docs/ja/configuration.md).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Refs: BE-0424
… job

actuation (xcuitest) (run 35185561508, job 105090800802) failed on a plain
wait timeout for an unrelated element (horse.title), with two
recordVideo/video-no-bytes stalls and a Simulator reboot after a failed cold
runner spawn in its log — the same render/window-server host fault class
pattern 9 already names, but the first confirmation on this PR of the
generic pattern itself rather than one of its app-crash-specific derivatives
(#11/#12). Reran the job.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Refs: BE-0424
T and others added 7 commits September 17, 2026 17:20
app-crash (xcuitest) has failed twice cleanly (no concurrent video-no-bytes
stall) with the same "no app-crash/ directory" symptom pattern #12 already
anticipates as evidence the 15s _APP_CRASH_REPORT_TIMEOUT bound may itself
be too tight — but confirming that needs host telemetry this job never
uploaded. Its --runs-dir pointed at tmp/showcase-app-crash-runs instead of
runs/, so BE-0361's runs/diagnostics/ sweep (which every sibling job's
existing runs/ upload rides for free) never picked it up, and the job called
collect-ios-diagnostics nowhere at all.

Move the scenario run tree to runs/app-crash (still fresh-removed per run,
still gitignored the same as tmp/ — gitleaks dir . sweeps both regardless of
.gitignore, so that reasoning never actually depended on the tmp/ prefix),
add the Start/Collect iOS diagnostics steps every other raw-bajutsu-run job
already has, and widen the upload to runs/ so stall-NN-*, render-probe.txt,
and host-telemetry.log ride along automatically. A future no-stall recurrence
now has real evidence to diagnose the timeout against instead of the
log-line-only confirmation the last two occurrences were limited to.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Refs: BE-0424
…424-d849c1

# Conflicts:
#	.apm/skills/investigate-ci-failure/references/known-ci-failure-patterns.md
… 30s

Four app-crash (xcuitest) CI recurrences after BE-0424's diagnostics-gap
fix all carried direct evidence of host/render-service contention
(render-probe.txt's screenshot-probe kill, a stall capture's backboardd
sample timeout) recorded before any scenario ran. Widening the budget
cannot fix a fully wedged render pipeline, but gives ReportCrash more
room on a host that recovers within tens of seconds rather than staying
wedged for the probe's own window.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Refs: BE-0424
Analyze (java-kotlin) on PR #2012 (run 35250805835, job 105302464362)
failed Gradle dependency resolution with a blanket 403 Forbidden from
both repo.maven.apache.org and plugins.gradle.org, across every
artifact the Android showcase build needs. The PR's diff touches no
Android/Gradle config, so a code regression cannot explain every
coordinate being rejected by the registry at once — an external
registry/network fault, not a code defect.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Refs: BE-0424
… download

smoke (adb) on PR #2012 (run 35252166515, job 105307465870) failed
before any scenario script ran: sdkmanager's system-images install
logged "Error on ZipFile unknown archive" for the google_apis x86_64
image, and the emulator step then failed outright with the emulator
never listening on port 5554. A corrupted download from Google's SDK
package repository, not this repository's code — folded into the
existing "emulator never comes up" pattern (#7).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Refs: BE-0424
…n a scroll-distance test

conformance (adb) on PR #2012 (run 35253562344, job 105311875152) failed
test_one_scroll_step_travels_the_distance_it_was_asked_for with the same
WARNING bajutsu.adb.resident: read lag signature pattern #14 already
names, this time surfacing through a direct travel-distance measurement
rather than an expect assertion, with a widening lag across the test's
three measurements. The PR's adb_driver.py diff in this window is pure
addition (BE-0424 crash-signal probing) and none of it touches the
resident-read or scroll-actuation path.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Refs: BE-0424
…ccurrence

PR #2012 run 35253562594 reconfirms pattern #9 far more severely than
before: app-crash (xcuitest) shows a failed cold runner spawn forcing a
Simulator reboot, a recordVideo that never started, a mid-run runner
exit (code 65), and a render-probe watchdog with a 15s kill bound that
doesn't report the kill until 26s — the bash polling loop itself
running behind, not just the render service. actuation (xcuitest)
failed in the same run ~13 minutes later with the same video-no-bytes
signature. Cross-job corroboration, within one run, of a degraded
batch of assigned runners rather than one unlucky VM.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Refs: BE-0424

This branch was successfully deployed

1 active (outdated) deployment
claude-review — f7daef3e Deployed Sep 16, 2026 by 0x0c via claude review #9238
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

topic:contribution topic:dogfood Dogfood fixtures (demo apps) (roadmap topic) topic:platform Platform expansion (Android / Web / Flutter) (roadmap topic)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant