feat: add first-class baseline/candidate compatibility regression comparison

[ErenAri]

feat: add first-class baseline/candidate compatibility regression comparison

Summary

Add first-class baseline → candidate compatibility comparison to BPFCompat.

Today BPFCompat can produce structured compatibility reports for an artifact or project loader across a kernel matrix, but consumers that want to answer:

"Did this candidate introduce a new kernel compatibility regression compared with the last known-good release?"

still need to implement their own comparison and gating logic.

Inspektor Gadget's upstream compatibility lane is a concrete example: it had to implement project-side aggregation logic for regression detection, existing limitations, improvements, missing reports, and completeness checks.

This issue moves those semantics into BPFCompat itself so future integrations can consume a stable comparison contract instead of rebuilding it.

---

Goal

Given two BPFCompat reports produced against the same logical compatibility matrix:

• baseline: last known-good release / reference artifact
• candidate: current build / release candidate

classify every comparable target and produce a machine-readable and human-readable regression report.

The primary question is:

Which previously working kernel targets changed from PASS to FAIL?

---

Classification semantics

For every comparable artifact/profile pair:

Baseline	Candidate	Classification
PASS	PASS	stable
PASS	FAIL	regression
FAIL	FAIL	existing_limitation
FAIL	PASS	improvement
known result	missing/error	incomplete
missing/error	known result	incomplete
missing/error	missing/error	incomplete

Infrastructure failures must remain distinct from compatibility failures.

A target must never be classified as a compatibility regression solely because the candidate run suffered an infrastructure error.

---

Proposed CLI

Initial explicit-report form:

bpfcompat compare \
  --baseline baseline.json \
  --candidate candidate.json \
  --out comparison.json \
  --markdown comparison.md

CI gating:

bpfcompat compare \
  --baseline baseline.json \
  --candidate candidate.json \
  --fail-on-regression \
  --fail-on-incomplete

Expected exit behavior:

0 = comparison completed and configured policy passed
1 = configured compatibility policy failed
2 = invalid input / schema / comparison configuration
3 = comparison could not be completed because of infrastructure/internal error

Exact exit codes may be adjusted to match existing BPFCompat conventions.

---

Output

The comparison JSON should be stable and versioned independently from internal Go structs.

Example shape:

{
  "schema_version": "1",
  "baseline": {
    "report_sha256": "...",
    "artifact": {
      "digest": "sha256:..."
    }
  },
  "candidate": {
    "report_sha256": "...",
    "artifact": {
      "digest": "sha256:..."
    }
  },
  "summary": {
    "stable": 17,
    "regressions": 1,
    "existing_limitations": 4,
    "improvements": 2,
    "incomplete": 0
  },
  "results": [
    {
      "profile_id": "ubuntu-22.04-5.15",
      "baseline_status": "pass",
      "candidate_status": "fail",
      "classification": "regression",
      "candidate_failure": {
        "classification": "UNSUPPORTED_ATTACH_TYPE"
      }
    }
  ]
}

The Markdown output should prioritize regressions first and clearly separate:

1. regressions
2. incomplete / infrastructure results
3. improvements
4. existing limitations
5. stable targets

---

Profile identity

Comparison must use immutable logical profile identity rather than display names alone.

At minimum, reports should preserve enough metadata to determine that the baseline and candidate represent the same intended target, including:

profile_id
architecture
distribution
distribution release
kernel release
profile revision / definition identity

Exact kernel releases may legitimately differ when comparing rolling/freshened vendor profiles.

Therefore the comparison model should distinguish:

logical profile identity

from:

exact execution environment identity

Both must remain visible in the evidence.

---

Artifact identity

Comparison output should preserve artifact identity where available:

• local artifact SHA-256
• OCI resolved manifest digest
• loader/binary digest
• BPFCompat version
• source report digest

Artifact equality must not be used as a prerequisite for regression detection.

A real PASS → FAIL transition must still be reported when one or both reports lack optional artifact hash metadata.

---

Missing and incomplete results

Missing results need explicit semantics.

Examples:

REFERENCE_NOT_FOUND
NO_EBPF_LAYER
NO_REPORT
MISSING_PROFILE_RESULT
INFRA_ERROR
TIMEOUT
SKIPPED

The comparison engine should retain the underlying reason and classify the comparison as incomplete unless the project policy explicitly declares the missing case expected.

Expected missing results should be configurable rather than hard-coded.

---

Policy

Initial policy controls:

--fail-on-regression
--fail-on-incomplete

Potential follow-up options:

--allow-expected-missing <file>
--require-profile <profile>
--required-only

The comparison engine should separate classification from policy:

comparison answers what changed
policy decides whether CI should fail

This keeps the report useful outside GitHub Actions.

---

GitHub Action integration

Once the CLI contract is stable, expose the comparison behavior through the Action.

Conceptual future usage:

- uses: Kernel-Guard/bpfcompat@<pinned-sha>
  with:
    baseline-report: baseline.json
    candidate-report: candidate.json
    fail-on-regression: true

A later higher-level workflow may support resolving the baseline and candidate artifacts directly, but that is not required for this issue.

---

Acceptance criteria

• Compare two valid BPFCompat JSON reports.
• Produce stable, regression, existing_limitation, improvement, and incomplete classifications.
• Detect PASS → FAIL even when artifact SHA metadata is unavailable.
• Keep infrastructure errors distinct from compatibility failures.
• Reject reports that cannot be meaningfully aligned rather than silently comparing unrelated targets.
• Preserve exact baseline and candidate execution metadata.
• Emit a versioned JSON comparison report.
• Emit a readable Markdown report.
• Support --fail-on-regression.
• Support policy failure for unexpected incomplete results.
• Deterministically order comparison output.
• Add unit tests covering every transition in the classification table.
• Add tests for missing profile results and infrastructure errors.
• Add tests for profile identity mismatches.
• Document the comparison semantics and CI usage.

---

Suggested test matrix

Minimum transition tests:

PASS → PASS
PASS → FAIL
FAIL → FAIL
FAIL → PASS

PASS → INFRA_ERROR
FAIL → INFRA_ERROR
MISSING → PASS
PASS → MISSING
MISSING → MISSING

Additional correctness tests:

same logical profile, newer exact vendor kernel
different architecture
different profile IDs
missing optional artifact SHA
identical artifact, different execution result
different OCI manifest digests
unexpected missing report
expected missing report
deterministic output ordering

---

Non-goals

This issue does not include:

• hosted execution
• multi-tenant SaaS
• billing
• dashboards
• GitHub issue automation
• automatic git bisection
• compatibility attestations
• defining whether a project officially supports a kernel

BPFCompat reports evidence. Project maintainers own their support policy.

---

Why this belongs in core

Regression semantics are project-independent.

Falco, Inspektor Gadget, KubeArmor, bpfman, Tracee, and other eBPF projects may have very different loaders and artifacts, but the release question is the same:

Did a kernel target that worked before stop working in the candidate?

Making that a stable BPFCompat primitive should substantially reduce the amount of integration-specific CI glue required by future adopters.

This is also a prerequisite for higher-level features such as:

release-vs-main gates
historical compatibility timelines
automated regression issues
compatibility attestations
managed/private compatibility testing

The comparison contract should therefore be designed as a durable public interface rather than an Inspektor Gadget-specific aggregator port.