fix(profiling): stop losing TraceLens reports to two avoidable outcomes

[Cemberk]

Stacked on #172. Two things were still costing us TraceLens reports on real MI300X traces, both found by running the pipelines on hardware rather than by reading the code.

What was going wrong

A torchrun run always reported a failure it should not have. dynolog configures every process that registered with it, and for a torchrun job that includes the launcher, which supervises its children and runs no kernels. TraceLens is asked to analyze its trace, correctly says No GPU events found in the trace, and madengine turned that into a FAILURE row sitting next to the real report, plus a non-zero exit. Now that case is reported as SKIPPED with the reason spelled out, and no output path, since nothing was written.

--short_kernel_study could throw away the whole PyTorch workbook. On some real traces the short-kernel sheets come out with MultiIndex columns, and TraceLens writes every sheet with index=False, which pandas refuses:

NotImplementedError: Writing to Excel with MultiIndex columns and no index ('index'=False) is not yet implemented

Every CSV is written first, so the analysis is complete and only the workbook is lost. It is data-dependent: the same flags succeed on a duration-based capture of the same workload and fail on an iteration-based one. We no longer request the flag for the PyTorch report; it stays reachable through the analyzer's trailing extra args. Reported upstream as TraceLens#938.

What hardware showed

On MI300X (banff-cyxtera-s83-5), one torchrun job now produces exactly the intended mix, and the run exits 0:

trace	status	output
libkineto_trace_896.json, the rank that ran kernels	SUCCESS	workbook + 7 CSV sheets
libkineto_trace_728.json, the torchrun launcher	SKIPPED	none, with the reason
collective report	SKIPPED	none, dynolog names traces after the pid

The rocprofv3 pipeline still succeeds with all six CSV sheets, including the two short-kernel ones.

One thing I got wrong, and what it taught us

I first extended the flag removal to the rocprof report on the theory that it shares the same Excel writer. It does share the code, but it writes either the CSVs or the workbook:

if output_csvs_dir:
    ...  # write CSVs
else:
    ...  # write the workbook

Since madengine asks for the CSVs, that writer is unreachable and the flag was harmless there; removing it only deleted short_kernels_summary.csv and short_kernel_histogram.csv. That change is not in this PR. The same either/or also means rocprof runs never produce an .xlsx even though we pass --output_xlsx_path, which is now documented and noted next to the arguments so nobody drops the flag twice.

Tests

Both behaviors are covered without a GPU, through the dummy TraceLens fixture from #172, which now validates GPU events the way the real tool does:

• test_a_trace_without_gpu_activity_is_skipped_not_failed — unit
• test_a_launcher_trace_is_skipped_rather_than_failed — e2e, one worker trace and one launcher trace through the whole pipeline
• test_no_report_requests_the_short_kernel_study and a companion asserting it still works when asked for explicitly

Test plan

• 105 CI-safe tests pass on the GPU node: dummy TraceLens and dynolog pipelines, unit, integration
• GPU-gated suite: 9 passed, 2 skipped (those two cover the TraceLens-missing path)
• Real duration-based dynolog capture on a torchrun job: SUCCESS for the rank, SKIPPED for the launcher, exit 0
• Real rocprofv3 to TraceLens: SUCCESS with all six CSV sheets
• Local suite on Windows: 69 unit and integration tests

[Copilot]

Pull request overview

Improves the TraceLens profiling pipeline so madengine doesn’t treat “no GPU activity” traces (e.g., torchrun launcher traces captured by dynolog) as failures, and avoids losing the PyTorch workbook due to TraceLens’ --short_kernel_study Excel-writing limitation.

Changes:

• Treat TraceLens’ “No GPU events found in the trace” as a SKIPPED outcome (no output path, exit remains 0 unless real failures occurred).
• Stop requesting --short_kernel_study for the PyTorch TraceLens report by default (still possible via extra args).
• Extend the dummy TraceLens fixture + add unit/e2e coverage for launcher/no-GPU-event traces; update profiling docs accordingly.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/unit/test_tracelens_analyze.py	Adds unit coverage for skipping no-GPU-activity traces and for omitting --short_kernel_study by default.
tests/fixtures/dummy_tracelens/TraceLens/Reporting/_dummy.py	Updates dummy TraceLens contract and adds GPU-activity validation for PyTorch traces.
tests/e2e/test_tracelens_dummy_pipeline.py	Adds an end-to-end scenario with a launcher trace that has no GPU events and must be skipped.
src/madengine/scripts/common/tools/tracelens_analyze.py	Implements SKIPPED handling for “no GPU events” and removes --short_kernel_study from PyTorch args.
docs/profiling.md	Documents dynolog/torchrun empty traces and rocprof CSV-only behavior.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🟡 Changes recommended

Two moderate issues remain unresolved, in addition to a documentation nit.

Review effort: Lite
Findings: 2 · 2

Open (4)

• Avoid attributing all missing GPU events to the launcher · New
• _check_gpu_events() currently swallows JSON/IO problems and returns success. For gzipped Kineto…
• Fix grammatical error in launcher message · New
• _pytorch_args() comment says the flag can be passed via the analyzer’s trailing extra args, but…

[Copilot]

Copilot review overview

🔵 Needs a closer look

Unresolved moderate findings remain in the analyzer behavior, CLI documentation, and profiling documentation.

Review effort: Lite
Findings: 2 · 2

Open (4)

• Avoid attributing all missing GPU events to the launcher
• _check_gpu_events() currently swallows JSON/IO problems and returns success. For gzipped Kineto…
• Fix grammatical error in launcher message
• _pytorch_args() comment says the flag can be passed via the analyzer’s trailing extra args, but…

[Copilot]

Copilot review overview

🔵 Needs a closer look

The caller must distinguish all-skipped results from cases with no supported trace artifacts.

Review effort: Lite
Findings: None

Resolved since last review (4)

• Avoid attributing all missing GPU events to the launcher
• _check_gpu_events() currently swallows JSON/IO problems and returns success. For gzipped Kineto…
• Fix grammatical error in launcher message
• _pytorch_args() comment says the flag can be passed via the analyzer’s trailing extra args, but…