Skip to content

test(e2e): add CLI conformance baseline - #2871

Closed
elezar wants to merge 2 commits into
mainfrom
codex/2183-cli-conformance-minimal
Closed

test(e2e): add CLI conformance baseline#2871
elezar wants to merge 2 commits into
mainfrom
codex/2183-cli-conformance-minimal

Conversation

@elezar

@elezar elezar commented Aug 21, 2026

Copy link
Copy Markdown
Member

Summary

Establish a minimal, portable CLI conformance baseline that certifies the complete CLI -> gateway -> driver deployment path. The standalone conformance task consumes the gateway already selected in the CLI configuration and does not provision a gateway, choose an installation method, or select a compute driver.

Phase 1 contains one hardened smoke scenario. It establishes reusable command execution, polling, diagnostics, explicit resource tracking, and cleanup mechanics before additional lifecycle workflows are considered for conformance.

This is stack 1/2. PR #2873 adds the direct API conformance framework on top.

Related Issue

Changes

  • add the e2e-cli-conformance profile and focused mise run e2e:cli-conformance task
  • run the standalone task against an already configured gateway without provisioning one or selecting a driver
  • require OPENSHELL_BIN so the suite executes the explicitly selected CLI binary rather than assuming a repository build location
  • add an OpenShell-specific runner with generated run identity, fluent command steps, structured results, typed execution errors, JSON parsing, bounded polling, sanitized diagnostics, and focused fake-CLI tests
  • make connected gateway status a reusable runner precondition while retaining the human-readable openshell status command as part of the scenario
  • create one uniquely named detached sandbox with --from base, verify readiness through sandbox get <name> --output json, verify list visibility, execute a run-specific marker, delete the sandbox, and verify its disappearance
  • track the unique sandbox name explicitly and combine functional and cleanup failures through the consuming finish() call; phase 1 does not require labels
  • make Docker, Podman, Kubernetes, and VM Cargo features imply CLI conformance so driver wrappers do not repeat the conformance feature
  • keep VM overlay and TLS-key permission assertions in the VM-specific vm_overlay target
  • document the phase-1 contract and interrupted-run resource recovery procedure

Testing

  • 14 focused conformance helper tests
  • 36 complete E2E harness library tests
  • smoke target compilation with only e2e-cli-conformance enabled
  • smoke target compilation through Docker, Podman, Kubernetes, and VM driver features
  • strict Clippy for the helper library and smoke target
  • shell syntax checks for the affected driver wrappers
  • complete mise run pre-commit
  • missing-gateway precondition fails immediately without attempting sandbox cleanup
  • live driver-backed smoke run (no gateway is currently selected in the local CLI configuration)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Conformance is independent of gateway installation method and compute driver
  • Driver-specific assertions remain outside conformance
  • Cleanup is scoped to the explicitly tracked, uniquely named sandbox
  • Documentation updated

@elezar elezar added the test:e2e Requires end-to-end coverage label Aug 21, 2026
@copy-pr-bot

copy-pr-bot Bot commented Aug 21, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions

Copy link
Copy Markdown

Label test:e2e applied, but pull-request/2871 does not exist yet. A maintainer needs to comment /ok to test 4180d62b99b1e3d9891f8281cd0a1adb2e10ac3b to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

@elezar elezar changed the title test(e2e): add minimal CLI conformance profile test(e2e): add CLI conformance baseline Aug 21, 2026
@elezar
elezar force-pushed the codex/2183-cli-conformance-minimal branch 2 times, most recently from 7b57176 to 3d802c2 Compare August 25, 2026 10:21
@elezar
elezar marked this pull request as ready for review August 25, 2026 10:27
@elezar
elezar requested review from a team, derekwaynecarr, mrunalp and sjenning as code owners August 25, 2026 10:27
@elezar
elezar force-pushed the codex/2183-cli-conformance-minimal branch 2 times, most recently from 184ffbc to e794b2c Compare August 25, 2026 13:06
elezar added 2 commits August 25, 2026 16:08
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
@elezar
elezar force-pushed the codex/2183-cli-conformance-minimal branch from e794b2c to a257f6a Compare August 25, 2026 14:21
@elezar

elezar commented Aug 26, 2026

Copy link
Copy Markdown
Member Author

Closing unmerged: superseded by #2925, which now carries the full conformance/e2e stack against main through 2a2e632.

@elezar elezar closed this Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant