fix: stop xcodebuild collecting a sysdiagnose the user did not ask for by pgyula · Pull Request #281 · bitrise-steplib/steps-xcode-test

pgyula · 2026-09-18T12:58:31Z

Checklist

I've read and followed the Contribution Guidelines
step.yml and README.md is updated with the changes (if needed)

Version

Context

Workarounds unit test execution hanging intermittently due to xcodebuild's built-in diagnostics collection.

Since Xcode 26, xcodebuild collects a simulator sysdiagnose of its own after a failing test run.
This is similar to what the Step collected on its own. The Input Collect Simulator diagnostics (collect_simulator_diagnostics) set to never now disables xcodebuild's diagnostic too.

To keep Simulator diagnostics collection, set the Input to on_failure, or project_setting to preserve test plan settings (the default is collection on failure).
Our collection is disabled for favor of xcodebuild's built-in, when avaialble. (The underlying command is a similar simctl diagnose)

Investigation details

Test plan encoding: "diagnosticCollectionPolicy": "Never" | "Always" in defaultOptions or any configurations[].options; absent = on failure. The policy is stored only in .xctestplan files, not in the scheme and not as a build setting. A fresh Xcode 27 project (autocreated scheme, implicit plan) resolves to on-failure. The CLI flag overrides the test plan

Decisions

Map the existing input onto the flag instead of adding a third diagnostics knob.
Keep always as an accepted value: removing it would reject existing configs (MAJOR bump) for a debugging knob; on Xcode 26+ it equals on_failure.
Projects that set the policy in their test plan opt out of the override with project_setting instead of the Step parsing test plans: no project parsing, explicit in the config, and no new library dependency.

Follow-up

bitrise-step-xcode-test-without-building has the same Xcode 26 exposure (it passes no diagnostics option).

Since Xcode 26, xcodebuild collects a simulator sysdiagnose of its own after a failing test run, by shelling out to `simctl diagnose --timeout=600`. This is a separate mechanism from the diagnostics this Step collects during teardown, and it runs even when the user set `collect_simulator_diagnostics: never` - the default. It usually presents as tests "hanging" at the end of the run. Measured on osx-xcode-27.0.x with a two-file SPM package and a single XCTFail, the tests themselves take ~1s either way: default -collect-test-diagnostics never IDETestOperationsObserverDebug 618.801s 21.746s Xcode Test step 10.7 min 45.62 sec .xcresult.zip artifact 16.58 MiB 55.92 KiB The collection does not even succeed; it ends with IDETestOperationsObserverDebug: Failure collecting diagnostics from simulator: Timed out after 600.0 seconds while waiting for a response from the invoked process so the default spends ten minutes and 16 MB per failing build to produce a timeout error. Rather than add a third diagnostics knob, map the existing `collect_simulator_diagnostics` input onto xcodebuild's option, so the Step honours the contract that input already advertises: never (default) -> -collect-test-diagnostics never on_failure -> -collect-test-diagnostics on-failure always -> -collect-test-diagnostics on-failure xcodebuild accepts `on-failure|never` only - there is no `always` - and its collection is failure-triggered anyway, so `always` maps onto `on-failure`. Guarded two ways: - The option does not exist before Xcode 26, so it is only passed for major >= 26. main.go already treats an unreadable Xcode version as non-fatal, leaving major at 0, which skips the option. - An explicit `-collect-test-diagnostics` in `xcodebuild_options` wins; the Step does not add its own. User options are also still appended last. Users who want xcodebuild's collection back set `collect_simulator_diagnostics` to `on_failure` or `always`, or pass the option directly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ofalvai · 2026-09-21T08:43:21Z

    category: Debugging
    title: Collect Simulator diagnostics
    summary: If this input is set, the simulator verbose logging will be enabled and the simulator diagnostics log will be exported.
+    description: |-


Let's rewrite this description so that the default and simple description explains the new (Xcode 26+) behavior, and the older behavior is explained as an exception. We'll phase out those older Xcodes in a few years and will be left with only the new behavior.

rephrased once more

Copilot

Copilot review overview

🟡 Changes recommended

The generated README remains stale and omits the newly documented Xcode 26 behavior.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1

Open (1)

README input table omits updated Xcode 26 behavior · New

What changed in this PR

Adds Xcode 26+ control over automatic test diagnostics, respecting the existing simulator diagnostics setting.

Changes:

Maps diagnostics preferences to -collect-test-diagnostics.
Preserves explicit user overrides and older Xcode compatibility.
Adds mapping and argument-generation tests.

File	Description
`main.go`	Supplies the detected Xcode version.
`step/step.go`	Carries version and diagnostics configuration.
`step/utils.go`	Maps diagnostics settings to xcodebuild values.
`step/step_test.go`	Updates parser test fixtures.
`step/collecttestdiagnostics_test.go`	Tests version, mapping, and override behavior.
`xcodebuild/utils.go`	Adds the xcodebuild argument.
`xcodebuild/xcodebuild_test.go`	Updates expected test arguments.
`step.yml`	Documents the updated input behavior.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

xcodebuild only understands the two-token form, `-collect-test-diagnostics never`. It accepts `-collect-test-diagnostics=never` (and any `-name=value` token) without an error and silently ignores it, verified on Xcode 26.5 and 27.0 by reading DiagnosticCollectionPolicy back from the generated xctestrun. With the `=` form counted as an override the Step dropped its own flag, xcodebuild dropped the user's, and the default on-failure collection ran even though both asked for `never`. Only the exact two-token form suppresses the Step's flag now. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…later Since Xcode 26 xcodebuild collects the Simulator diagnostics itself after a failing run, driven by -collect-test-diagnostics, and stores them in the xcresult the Step already exports. The Step's own `simctl diagnose` during teardown produced a second copy of the same data as a separate zip and doubled the time spent collecting on a failing build. On Xcode 26 and later the Step no longer collects on its own. Simulator verbose logging is still enabled when the input is not `never`, so xcodebuild's bundle benefits from it. Xcode 16 and earlier keep the previous behaviour. `always` consequently behaves like `on_failure` on Xcode 26 and later, which the input description now states. Rewrote the input description so the Xcode 26 behaviour is the primary text and older Xcode is the exception, as requested in review, and regenerated README.md. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

xcodebuild's -collect-test-diagnostics knows on-failure and never only, and the Step no longer collects on its own on Xcode 26 and later, so `always` cannot mean more than `on_failure` there. Keep the value accepted (dropping it would break existing configs) and say what it does per Xcode version. README regenerated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The Step's default `never` overrides whatever the project's test plan says about diagnostics collection, since the -collect-test-diagnostics option wins over the plan in both directions (verified on Xcode 26.5 and 27.0). Projects that opted in through their plan had no way to keep that choice short of passing the option in xcodebuild_options. `project_setting` passes no -collect-test-diagnostics at all, so xcodebuild follows the test plan: collect on failure unless the plan says otherwise. Simulator verbose logging is still enabled, as for every value other than `never`. On Xcode 16 and earlier there is nothing to follow and the Step does not collect on its own for this value. Input description rewritten around the new default behaviour; README regenerated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Replace the inline check workflow with the steps-check include, as in bitrise-steplib/steps-git-clone#250. The shared checks surfaced a few things, fixed here so the first run is green: staticcheck ST1005 (capitalised error strings in main.go), QF1004 (strings.ReplaceAll in testaddon), and yamlfmt (`{}` spelling in e2e/bitrise.yml). Verified locally with `bitrise merge`, the shared .golangci.yml (0 issues), the shared .ymlfmt (clean) and stepman audit. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

They are user-facing log lines, not wrapped errors, so exempt them from staticcheck ST1005 instead of lowercasing them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

logger.Errorf received the formatted error chain as its format string, so a `%` inside an error message would have been misrendered. Use "%s" as in steps-xcode-archive. Also fix the Xcode version warning, which used the %w verb in a logger call where only %s is valid. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Describe the mapping as it behaves after this change instead of the history behind it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

shouldStepCollectDiagnostics decides whether the Step runs its own simctl diagnose; xcodebuildDiagnosticsOverride is the value the Step passes to xcodebuild's -collect-test-diagnostics, carried as Config.XcodebuildDiagnosticsOverride and TestParams.XcodebuildDiagnosticsOverride. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Copilot

Copilot review overview

🟢 Approval recommended

The core behavior is coherent and well tested; the remaining misleading legacy-Xcode log message is non-blocking.

Review effort: Balanced
Findings: 1 · 1

Open (2)

Gate diagnostics extraction notice on Xcode version · New
README input table omits updated Xcode 26 behavior

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The teardown notice that xcodebuild collected the diagnostics into the xcresult also fired for a failed project_setting run on Xcode 16 and earlier, where nothing collects anything. Gate it on the Xcode version. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

pgyula and others added 2 commits September 17, 2026 16:05

Merge branch 'master' into fix/xcode26-collect-test-diagnostics

6b3b738

gergely-sallai previously approved these changes Sep 18, 2026

View reviewed changes

ofalvai reviewed Sep 21, 2026

View reviewed changes

ofalvai requested a balanced review from Copilot September 21, 2026 08:51

Copilot started reviewing on behalf of ofalvai September 21, 2026 08:52 View session

Copilot AI reviewed Sep 21, 2026

View reviewed changes

Comment thread step.yml

comments cleanup

dd24e56

lpusok dismissed gergely-sallai’s stale review via dd24e56 September 21, 2026 11:22

lpusok and others added 8 commits September 21, 2026 16:31

Inline the -collect-test-diagnostics option name

2442790

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Keep the top-level error messages capitalised

c2a4eaa

They are user-facing log lines, not wrapped errors, so exempt them from staticcheck ST1005 instead of lowercasing them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

lpusok force-pushed the fix/xcode26-collect-test-diagnostics branch from 86d6ca4 to f3085ed Compare September 21, 2026 15:26

lpusok and others added 2 commits September 21, 2026 17:29

Compact the collectTestDiagnosticsValue doc comment

79d17bd

Describe the mapping as it behaves after this change instead of the history behind it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

lpusok requested review from gergely-sallai and ofalvai and a balanced review from Copilot September 21, 2026 15:52

Copilot started reviewing on behalf of lpusok September 21, 2026 15:53 View session

Replace the xcodebuildDiagnosticsOverride doc comment with inline notes

b6e7dfb

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

lpusok force-pushed the fix/xcode26-collect-test-diagnostics branch from 219b77c to b6e7dfb Compare September 21, 2026 15:55

Copilot AI reviewed Sep 21, 2026

View reviewed changes

Comment thread step/step.go Outdated

lpusok and others added 3 commits September 21, 2026 17:55

Replace the shouldStepCollectDiagnostics doc comment with an inline note

7895290

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Name the xcodebuild-collects decision in teardown

82be39b

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

benbitrise approved these changes Sep 21, 2026

View reviewed changes

Merge branch 'master' into fix/xcode26-collect-test-diagnostics

3f68c64

lpusok merged commit 8b80091 into bitrise-steplib:master Sep 22, 2026
1 check passed

lpusok mentioned this pull request Sep 22, 2026

Add collect_simulator_diagnostics to stop xcodebuild collecting a sysdiagnose the user did not ask for bitrise-steplib/bitrise-step-xcode-test-without-building#19

Open

2 tasks

Navigation Menu

fix: stop xcodebuild collecting a sysdiagnose the user did not ask for - #281

fix: stop xcodebuild collecting a sysdiagnose the user did not ask for#281
lpusok merged 18 commits into
bitrise-steplib:masterfrom
pgyula:fix/xcode26-collect-test-diagnostics

pgyula commented Sep 18, 2026 •

edited by lpusok

Loading

Uh oh!

ofalvai Sep 21, 2026

Uh oh!

lpusok Sep 21, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

pgyula commented Sep 18, 2026 • edited by lpusok Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Version

Context

Investigation details

Decisions

Follow-up

Uh oh!

ofalvai Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

lpusok Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟢 Approval recommended

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

pgyula commented Sep 18, 2026 •

edited by lpusok

Loading