feat(profiling): integrate TraceLens trace analysis

[Cemberk]

Turn the trace artifacts a run leaves behind into TraceLens operator, kernel, roofline, and collective reports, and add on-demand torch.profiler capture so PyTorch workloads can produce those traces without editing the model script.

Analysis runs in either of two places. On the host, madengine report tracelens keeps TraceLens' pinned protobuf and xprof out of workload images entirely. In-container, the tracelens tool installs into an isolated virtualenv for the same reason. SLURM and Kubernetes collection now gather torch_profiler_output/ and tracelens_output/ per node.

The scripts/ and *.json ignore rules are anchored to the repo root: unanchored they also matched packaged source, which silently excluded the five runtime scripts these tool definitions depend on.

[Copilot]

Pull request overview

Integrates TraceLens-based trace analysis into madengine, providing both host-side reporting (madengine report tracelens / tracelens-compare) and in-container tooling to capture PyTorch Kineto traces on-demand (dynolog) and generate TraceLens reports. This fits into the reporting + tools pipeline by turning collected profiler artifacts into actionable operator/kernel/roofline/collective outputs and ensuring distributed collectors gather the needed directories.

Changes:

• Add a stdlib-only TraceLens analyzer script plus a host-side wrapper module and new report CLI commands (tracelens, tracelens-compare).
• Add in-container tools (torch_profiler_dynolog, tracelens and mode-specific variants) with supporting pre/post scripts and TraceLens-venv isolation.
• Extend SLURM/Kubernetes artifact collection, add examples/docs, and add unit/integration/e2e coverage.

Reviewed changes

Copilot reviewed 24 out of 25 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/unit/test_tracelens_report.py	Unit tests for host-side TraceLens wrapper functions and report CLI behavior.
tests/unit/test_tracelens_analyze.py	Unit tests for the standalone analyzer script (discovery, command construction, summaries).
tests/integration/test_tracelens_tools_config.py	Integration tests validating tools.json wiring and tool stacking behavior.
tests/e2e/test_tracelens_workflows.py	E2E coverage for host reporting and (GPU) container tool workflows.
src/madengine/scripts/common/tools/tracelens_analyze.py	Standalone (stdlib-only) analyzer that discovers traces and runs TraceLens entry points out-of-process.
src/madengine/scripts/common/tools/dynolog_trigger.sh	Background trigger script to request torch.profiler traces via dynolog.
src/madengine/scripts/common/tools.json	Registers torch_profiler_dynolog and TraceLens tool variants and their scripts/env.
src/madengine/scripts/common/pre_scripts/trace.sh	Adds installers for dynolog and TraceLens (isolated venv + optional traceconv staging).
src/madengine/scripts/common/pre_scripts/dynolog_start.sh	Starts dynolog daemon and arms the trigger alongside the workload.
src/madengine/scripts/common/post_scripts/tracelens.sh	Runs the analyzer post-workload to generate TraceLens reports without failing the run.
src/madengine/scripts/common/post_scripts/trace.sh	Collects new output directories (torch_profiler_output, tracelens_output) as artifacts.
src/madengine/scripts/common/post_scripts/dynolog_stop.sh	Stops dynolog/trigger and reports captured trace count + tail logs.
src/madengine/reporting/tracelens_report.py	Host-side driver for running the analyzer and surfacing install guidance/errors.
src/madengine/deployment/templates/slurm/job.sh.j2	Collects additional per-node tool output directories on successful SLURM runs.
src/madengine/deployment/templates/kubernetes/job.yaml.j2	Copies additional per-pod tool output directories into results for K8s runs.
src/madengine/deployment/k8s_scripts.py	Ensures helper scripts referenced by pre/post scripts under scripts/common/tools are bundled for K8s.
src/madengine/deployment/k8s_results.py	Collects .pftrace and additional known output directories via kubectl cp.
src/madengine/cli/commands/report.py	Adds report tracelens and report tracelens-compare commands with Rich output.
pyproject.toml	Adds optional dependency extra madengine[tracelens] pinned to a TraceLens git ref.
examples/profiling-configs/tracelens_rocprofv3.json	Example config for rocprofv3_lightweight + TraceLens analysis.
examples/profiling-configs/torch_profiler_tracelens.json	Example config for dynolog capture + TraceLens PyTorch analysis.
examples/profiling-configs/README.md	Documents TraceLens/dynolog example configs and produced artifacts.
docs/profiling.md	Adds documentation for dynolog-based capture and TraceLens host/container analysis workflows.
docs/cli-reference.md	Adds CLI reference entries for report tracelens and report tracelens-compare.
.gitignore	Anchors scripts/ and *.json ignore rules to repo root to avoid ignoring packaged runtime scripts.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[coketaste]

Two pre-merge nits found while reviewing this PR as the base of the TraceLens stack (#167 -> #170 -> #172 -> #173). Neither is a design issue, both are quick fixes. See inline comments.

[Copilot]

Copilot review overview

🟡 Changes recommended

Critical credential exposure risks and blocking profiling, analysis, collection, and CLI failures remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 3 · 4 · 2

Open (9)

• Prevent Dynolog URL credentials from xtrace output · New
• Prevent TraceLens credential leakage through xtrace · New
• Remove unsupported Dynolog fail-on-no-process flag · New
• Treat analyzer subprocess failures as report failures · New
• Match Dynolog libkineto trace filenames · New
• Restrict collective analysis to discovered PyTorch traces · New
• Dynolog install path runs apt-get install -f without an apt-get update first. On minimal…
• This re.findall(...) line exceeds Black’s configured 88-character limit, which can cause…
• This function definition exceeds the configured Black line length (88 chars), which can cause…

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved moderate and critical findings affect artifact collection, failure reporting, credential safety, analysis correctness, and supported Python versions.

Review effort: Lite
Findings: 4 · 4 · 1

Open (9)

• list[str] breaks imports on Python 3.8 · New
• Remove unsupported Dynolog fail-on-no-process flag
• Prevent TraceLens credential leakage through xtrace
• Prevent Dynolog URL credentials from xtrace output
• TraceLens fallback may use the wrong Python environment · New
• Restrict collective analysis to discovered PyTorch traces
• Match Dynolog libkineto trace filenames
• Treat analyzer subprocess failures as report failures
• This re.findall(...) line exceeds Black’s configured 88-character limit, which can cause…

Resolved since last review (2)

• Dynolog install path runs apt-get install -f without an apt-get update first. On minimal…
• This function definition exceeds the configured Black line length (88 chars), which can cause…

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved moderate findings affect analyzer failures, helper bundling, artifact collection, dynolog triggering, and TraceLens input handling.

Review effort: Lite
Findings: 4 · 6 · 1

Open (11)

• list[str] breaks imports on Python 3.8
• Remove unsupported Dynolog fail-on-no-process flag
• Prevent TraceLens credential leakage through xtrace
• Prevent Dynolog URL credentials from xtrace output
• Propagate analyzer failure when summary is missing · New
• Require complete rank mapping for collective analysis · New
• TraceLens fallback may use the wrong Python environment
• Restrict collective analysis to discovered PyTorch traces
• Match Dynolog libkineto trace filenames
• Treat analyzer subprocess failures as report failures
• This re.findall(...) line exceeds Black’s configured 88-character limit, which can cause…

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved profiling isolation, artifact collection, analyzer argument, stale-summary, and SLURM workflow issues remain.

Review effort: Lite
Findings: 1 · 1

Open (2)

• Shared dynolog state causes concurrent runs to interfere · New
• This re.findall(...) line exceeds Black’s configured 88-character limit, which can cause…

Resolved since last review (10)

• list[str] breaks imports on Python 3.8
• Remove unsupported Dynolog fail-on-no-process flag
• Prevent TraceLens credential leakage through xtrace
• Prevent Dynolog URL credentials from xtrace output
• Require complete rank mapping for collective analysis
• Propagate analyzer failure when summary is missing
• TraceLens fallback may use the wrong Python environment
• Restrict collective analysis to discovered PyTorch traces
• Match Dynolog libkineto trace filenames
• Treat analyzer subprocess failures as report failures

Previously missed (2)

In code that hasn't changed since last review

Stale TraceLens summaries can falsely report rerun success

src/​madengine/​reporting/​tracelens_report.py:94

An existing tracelens_summary.json is accepted regardless of the new analyzer's result. If a rerun crashes before writing a summary, the stale file is loaded and its old counters can make the CLI report success. Remove the prior summary before launching the analyzer (or verify that the file was freshly written).

Terminate analyzer option parsing before forwarding extra arguments

src/​madengine/​reporting/​tracelens_report.py:167

Option-like values in extra_args are appended directly to the analyzer's argparse input, so a value such as --detect_recompute is rejected as an unknown analyzer option before TraceLens runs. Terminate this parser's options before forwarding the extra flags.