DR-008 Option 4: two-stage test-execution workflow (PR 2 of 2) by Subramanian-K812 · Pull Request #280 · eclipse-score/reference_integration

Subramanian-K812 · 2026-07-07T08:48:54Z

DR-008 Option 4: two-stage test-execution workflow (PR 2 of 2)

Closes #264. This is the second of two PRs. It is stacked on PR 1
(Subramanian-K812_resolve_override_mechanism, "resolved-dependency resolve + override
mechanism") — please review/merge that one first. Until PR 1 merges, this PR's base branch
is the PR-1 branch, so the diff shown here is only the test-execution delta.

What this PR does

Rewires the quality workflow to the DR-008 Option 4 flow, using the resolve + override
mechanism added in PR 1:

Stage 1 — integration: builds the platform, runs Feature Integration Tests, and
exports the resolved dependency set as the stage1-resolved-deps artifact
(resolved_versions.json).
prepare_matrix: derives the Stage-2 module list dynamically from
known_good.json (target_sw) — never hardcoded.
Stage 2 — per module: checks the module out at its known_good commit, overrides
its MODULE.bazel with the Stage-1 resolved set (PR 1's mechanism), and runs the
module's own unit tests + coverage as the Bazel root (//...).
aggregate: consolidates Stage 1 + Stage 2 into one quality report (and the
release-tag test-report ZIP).

Files

Workflow & runners

.github/workflows/test_and_docs.yml — two-stage restructure (+ parallel docs build)
scripts/quality_runners.py — module-context mode (--module-dir / --resolved-deps); calls ResolvedDependencies.overwrite
scripts/aggregate_quality_report.py — new, consolidated report
scripts/known_good/list_modules.py — new, dynamic Stage-2 matrix from known_good.json
scripts/integration_test.py — derive build targets from known_good.json (drop build_config.json)

Resolved pins + their config ripple (kept together so the tree is self-consistent)

known_good.json — pin bumps (baselibs, lifecycle) + per-module bazel_config + lifecycle code_root_path/extra_test_config
bazel_common/score_basic_bazel.MODULE.bazel — flatbuffers bump
bazel_common/score_modules_target_sw.MODULE.bazel — regenerated (baselibs, lifecycle)
MODULE.bazel.lock — updated lock
rust_coverage/BUILD — lifecycle query //src/... → //score/... (layout moved by the bump)
showcases/simple_lifecycle/BUILD — lifecycle bin path //src/... → //score/launch_manager
.bazelrc — coverage atomic-gcov / -no-coverage; eb-aarch64 outline-atomics link fix

Known Stage 2 integration findings (surfaced, not fixed here)

The workflow surfaces two module-owned integration failures caused by the
baselibs bump (score_logging CheckSizeValid stale uint8_t assumption;
score_persistency to_string on std::string_view).

github-actions · 2026-08-03T04:15:18Z

The created documentation from the pull request is available at: docu-html

PiotrKorkus

as the workflow has been completely reworked please create a private fork and execute it there, so we can see the proof of working in CI

pull_request_target wont allow to execute it here

PiotrKorkus · 2026-08-14T13:46:45Z

@@ -0,0 +1,221 @@
+# Stage 2 — Constraint Sheet
+
+Living document. Amend it when a constraint turns out to be wrong, and record why in the


If thats AI generated and maintained, where is a skill / agent for that purpose?

PiotrKorkus · 2026-08-14T13:47:09Z

where is this file generated from? All modules should be from known_good

not generated — known_good.json pins first-party repos by commit, these are registry deps pinned by version (confirmed via bazel mod graph --verbose: 2 of the 3 pins do move MVS's selection)

PiotrKorkus · 2026-08-14T13:47:54Z

+# 0.9.1 declares the four *_rules_rust_miri toolchains score_lifecycle_health and
+# score_persistency register in their own .bazelrc; 0.8.0 has none. ref_int cannot un-register
+# them, so the version must match here. Locked in score_test_artifact_versions.MODULE.bazel.


remove this comment

PiotrKorkus · 2026-08-14T13:50:32Z

+      - name: Set up Python 3
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'


why needed?

PiotrKorkus · 2026-08-14T13:51:25Z

+
+# One label for every Python unit test, so CI runs all of them by naming a single target.
+test_suite(
+    name = "python_tests",


change name, its meaningless without looking into target

Renamed to all_python_unit_tests.

PiotrKorkus · 2026-08-14T13:52:40Z

  push:
    branches:
      - main
+      - Subramanian-K812_test_execution_upgrade # TEMP: remove after CI verified


it doesnt work because the same workflow name is with pull_request_target on main

PiotrKorkus · 2026-08-14T13:53:36Z

+    runs-on: ubuntu-latest
+    steps:
+      - name: Clean disk space
+        uses: eclipse-score/more-disk-space@v1


use 1.1 version

Bumped to v1.1

PiotrKorkus · 2026-08-14T13:55:11Z

+          if [ ! -s _module/module_graph.json ]; then
+            echo "::warning::no module graph captured for ${{ matrix.module.name }}"; exit 0
+          fi
+          PYTHONPATH=scripts python3 scripts/known_good/verify_stage2_resolution.py \


if script needs a PYTHONPATH to be set its very bad design. Fix with proper modules imports

PiotrKorkus · 2026-08-14T14:14:25Z

+                    "bazel_config": [
+                        "stage2-gcc",
+                        "stage2-rust"


it feels like it doesnt need to be placed in this json as its common for every module

score_time and score_communication are exceptions for the above.

PiotrKorkus · 2026-08-14T14:14:42Z

                        "//score/mw/com/impl/bindings/lola/tracing:tracing_runtime_test"
+                    ],
+                    "bazel_config": [
+                        "stage2-rust"


why com has only rust?

score_communication passes use_base_constraints_only=True to its own gcc.toolchain(), which changes its generated toolchain target name leading to referencing a target that doesn't exist.

Subramanian-K812 · 2026-08-17T10:48:14Z

Ran the reworked workflow end-to-end on a private fork:
https://github.com/Subramanian-K812/reference_integration-ci-verify/actions/runs/32220961400

…nsition

…job unchanged

PiotrKorkus · 2026-08-25T07:57:54Z

this cannot be removed

PiotrKorkus · 2026-08-25T07:58:08Z

            "//feature_integration_tests/configs:qemu_bridge_config.json",
            "//images/qnx_x86_64:image",
        ],
-    }) + ["conftest.py"],


PiotrKorkus · 2026-08-26T13:59:19Z

Bazel version is not dependency to resolve. It is a given from ref_int

PiotrKorkus · 2026-08-26T14:05:48Z

None of the .bazelrc should be patched. We always take common config from ref_int

PiotrKorkus · 2026-08-26T14:05:51Z

None of the .bazelrc should be patched. We always take common config from ref_int

PiotrKorkus · 2026-08-26T14:05:55Z

None of the .bazelrc should be patched. We always take common config from ref_int

dcalavrezo-qorix · 2026-08-27T11:13:18Z

every job has continue-on-error: true

this means that the workflow can never go red. kind of a no-op as PR check

dcalavrezo-qorix · 2026-08-27T11:15:38Z

+# Do not flood CI with unneeded previous runs in PR
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
+  cancel-in-progress: ${{ github.ref_name != 'main' && !startsWith(github.ref_name, 'release/') }}


github.ref_name is the base branch / main so this evaluates to false for PRs -> old runs never get cancelled.

can you check?

OliverHeilwagen

review focus ./scripts/known_good, excluding tests.
Transitive dependency are now correctly pinned 👍 .

Patches from reference integration are locally applied but patching in Stage 2 uses single_version_override and git_override without patches, which leads to missing patches for module dependencies, which were present in Stage 1.

OliverHeilwagen · 2026-08-24T12:28:10Z

+        resolved: dict[str, Module] = {}
+        provenance: dict[str, str] = {}
+        declared_overrides: dict[str, dict[str, str]] = {}
+        declared_deps: dict[str, dict[str, object]] = {}


It would be nice to have more descriptive data classes or enum classes to improve the readability.
May also be taken up as a follow up cleanup.

e.g.:
provenance:

class PinProvenance(StrEnum): ASSERTED = "asserted" INCIDENTAL = "incidental" provenance: dict[str, PinProvenance] = {}

declared_overrides:

@dataclass(frozen=True) class UncarriedOverride: module_name: str kind: str source_file: str reason: str consequence: str def to_dict(self) -> dict[str, str]: return { "module": self.module_name, "kind": self.kind, "file": self.source_file, "reason": self.reason, "consequence": self.consequence, } declared_overrides: dict[str, UncarriedOverride] = {}

declared_deps:

@dataclass(frozen=True) class DependencySpec: module_name: str version: str | None dev_dependency: bool source_file: str declared_deps: dict[str, DependencySpec] = {}

OliverHeilwagen · 2026-08-24T12:28:20Z

+
+        # The completeness guard: every override ref_int declared that did not become a manifest entry.
+        uncarried = [entry for name, entry in sorted(declared_overrides.items()) if name not in resolved]
+        report = _build_report(resolved, provenance, uncarried, declared_by, _internal_drift(resolved, declared_deps))


as mentioned above report would also benefit from a dataclass approach e.g.:

@dataclass(frozen=True) class PinReport: pin: dict[str, str] provenance: PinProvenance declared_versions: list[str] verdict: Literal["unknown", "by_commit", "agree", "differs"] = "unknown" direction: Literal["ref_int_lower", "ref_int_higher"] | None = None def to_dict(self) -> dict[str, object]: return { "pin": self.pin, "provenance": self.provenance.value, "declared_versions": self.declared_versions, "verdict": self.verdict, "direction": self.direction, }

OliverHeilwagen · 2026-08-24T13:12:57Z

+        if entry["verdict"] == "differs":
+            conflicts.append(name)


conflicts appears to be a too strong term for the report entry.
I would expect ref_int_lower or ref_int_higher a typical case.
Can we stick to differs instead, this would also be more self-explanatory.

OliverHeilwagen · 2026-08-28T06:47:07Z

        return cls({name: Module.from_dict(name, md) for name, md in entries.items()})

    @staticmethod
    def _parse_override_file(text: str) -> list[Module]:


The existing known_good.json makes use of patches (git_override or local_override).
Please also collect the patches so we are capable of injecting them in stage 2.

Related comment is in quality_runner.py
We apply the pachtes locally but the override mechanism is loosing them for dependencies since we use git_override.

OliverHeilwagen · 2026-08-28T07:06:21Z

        """
        module_bazel = Path(module_bazel)
        original = self._strip_injection(module_bazel.read_text())



Please add a warning generation in case the module is using:

*_override

highlight patches?

non bazel version resolution dependency

http_archive

http_file

git_repository

new_git_repository

local_repository

Overrides / Patches used by the module may conflict with the version imposed by reference integration.
Ideally we should do such a scan as a initial sanity check if the versions of known_good.json are updated.

If a selected module contain such elements it may cause build / test conflicts, being aware early could reduce time for issue identification.

OliverHeilwagen · 2026-08-28T10:13:49Z

+            raise SystemExit(f"{args.modules_to_test[0]} is not a target_sw module in {args.known_good_path}")
+        print_centered(f"QR: Applying {module_under_test.name}'s own bazel_patches to {workspace}")
+        try:
+            apply_module_patches(module_under_test, workspace.resolve(), REF_INT_ROOT, log=print_centered)


patches are applied locally but the resolved.overwrite() patches are using single_version_override or git_override which leads to missing patches for dependencies in the stage 2 module run.

Ideally we can transport a patch of a dependency into the overwrite patching functionality.

OliverHeilwagen · 2026-08-28T10:37:25Z

+  contents: write
+  pull-requests: write
+on:
+  pull_request_target: # Allow forks to access secrets


This should only be required once we extend to QNX build and test in this workflow.
Since we use --config=linux-x86_64 shouldn't the following be sufficient?
Then we can also check the run in this PR.

Suggested change

pull_request_target: # Allow forks to access secrets

pull_request:

OliverHeilwagen · 2026-08-28T10:47:47Z

+    return tuple(int(part) for part in version.split(".")) if _RELEASE_RE.fullmatch(version) else None
+
+
+def resolve_stage2_bazel_version(ref_int_version: str, module_version: str | None) -> str:


Why don't we demand a .bazelversion to be present under root of the module?
This way we could simplify the call and just give the root path of ref_int and module the module.
We could then also have the bazel version patching logic in this python file.

OliverHeilwagen · 2026-08-28T10:56:45Z

+      - name: Export resolved dependency manifest
+        if: always()
+        run: |
+          mkdir -p artifacts/stage1-resolved-deps


Please provide all the steps also in a README for local replication of the workflow.

OliverHeilwagen · 2026-08-28T11:15:31Z

+    actual = module_graph_versions(args.mod_graph)
+    resolved = json.loads(args.resolved.read_text(encoding="utf-8"))
+    resolved = resolved.get("modules", resolved)


since we base the check upon the bazel mod graph of the module and the resolved_version.json which both do not contain information about patches demanded by reference integration, the check is incomplete.

Subramanian-K812 force-pushed the Subramanian-K812_test_execution_upgrade branch from c9462ea to fd3df76 Compare July 7, 2026 08:49

Subramanian-K812 force-pushed the Subramanian-K812_test_execution_upgrade branch 2 times, most recently from d293277 to b6df34d Compare July 24, 2026 06:56

Subramanian-K812 force-pushed the Subramanian-K812_test_execution_upgrade branch 2 times, most recently from e51b340 to 640dfee Compare August 3, 2026 03:26

Subramanian-K812 force-pushed the Subramanian-K812_test_execution_upgrade branch 4 times, most recently from 04a7a73 to e816620 Compare August 11, 2026 10:22

PiotrKorkus requested changes Aug 14, 2026

View reviewed changes

Subramanian-K812 force-pushed the Subramanian-K812_test_execution_upgrade branch 3 times, most recently from b22c037 to a992474 Compare August 21, 2026 09:58

PiotrKorkus mentioned this pull request Aug 24, 2026

Add resolved-dependency override mechanism for DR-008 #278

Merged

Subramanian-K812 added 3 commits August 24, 2026 13:04

feat: two-stage DR-008 test workflow with per-module Stage 2 execution

4624b81

ci: run new two-stage workflow alongside unchanged old job during tra…

715f615

…nsition

fix: give central-mode test runs their own exclusion list

35d83cd

Subramanian-K812 force-pushed the Subramanian-K812_test_execution_upgrade branch from 587dcbb to 35d83cd Compare August 25, 2026 04:05

Subramanian-K812 marked this pull request as ready for review August 25, 2026 04:08

Subramanian-K812 requested review from AlexanderLanin, FScholPer, MaximilianSoerenPollak, antonkri, dcalavrezo-qorix, lurtz, nradakovic, opajonk and pawelrutkaq as code owners August 25, 2026 04:08

ci: split DR-008 test execution into its own workflow file, keep old …

38dd214

…job unchanged

PiotrKorkus reviewed Aug 25, 2026

View reviewed changes

fix: restore itf conftest.py

05184ee

Subramanian-K812 requested a review from PiotrKorkus August 25, 2026 13:19

PiotrKorkus requested changes Aug 26, 2026

View reviewed changes

dcalavrezo-qorix reviewed Aug 27, 2026

View reviewed changes

OliverHeilwagen suggested changes Aug 28, 2026

View reviewed changes

		@@ -0,0 +1,221 @@
		# Stage 2 — Constraint Sheet

		Living document. Amend it when a constraint turns out to be wrong, and record why in the

	pull_request_target: # Allow forks to access secrets
	pull_request:

		return tuple(int(part) for part in version.split(".")) if _RELEASE_RE.fullmatch(version) else None


		def resolve_stage2_bazel_version(ref_int_version: str, module_version: str \| None) -> str:

Uh oh!

Conversation

Subramanian-K812 commented Jul 7, 2026

DR-008 Option 4: two-stage test-execution workflow (PR 2 of 2)

What this PR does

Files

Known Stage 2 integration findings (surfaced, not fixed here)

Uh oh!

github-actions Bot commented Aug 3, 2026

Uh oh!

PiotrKorkus left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Subramanian-K812 commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

OliverHeilwagen left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Subramanian-K812 commented Aug 17, 2026 •

edited

Loading

OliverHeilwagen left a comment •

edited

Loading