feat(ci): run e2e tests from the runner via Holodeck remoteAccess

[abrarshivani]

Runs the Kubernetes-facing parts of the e2e suite on the GitHub Actions runner instead of inside the Holodeck EC2 node. Draft until CI has run against it.

What changes

kubernetes.remoteAccess (NVIDIA/holodeck#818) gives the runner a usable kubeconfig, so helm, kubectl, the case scripts and log collection all run there. The repo rsync, the scp of the values override, the in-VM tooling installs and pull.sh are gone.

Two host-mutating operations stay on the node: sudo modprobe -a i2c_core ipmi_msghandler, and the operator container kill in test_restart_operator. Both go through a new tests/scripts/node-exec.sh, which streams the self-contained node-operations.sh over SSH. With NODE_SSH_HOST unset it runs locally, so the developer path still works. Container-selection logic is moved verbatim.

This is not a revert. Unlike NVIDIA/k8s-device-plugin#1948, the rsync + ssh model here is the original 2021 design for laptop-driven testing, and CI was already using it on Holodeck v0.2.18, before the kubeconfig permission change.

local.sh, ci-run-e2e.sh, push.sh, pull.sh, sync.sh, remote.sh and prerequisites.sh are kept, since they are the documented developer workflow in tests/README.md. CI just stops calling them.

Polling loops, worth a close look

The loops in checks.sh and update-nvidiadriver.sh bounded themselves by counting sleeps rather than elapsed time. That held when an iteration cost about 5s on the node. From the runner it does not: the 45 minute bound stretches past the job's 90 minute cap, and a cancelled job skipped the failure()-gated artifact upload, so a hung run produced no logs at all.

Twelve loops now measure elapsed time with SECONDS against a baseline taken at loop entry. Bounds, messages and exit paths are unchanged. In wait_for_driver_upgrade_done the current_time % 30 debug cadence became a due-time check, since irregular elapsed jumps can miss a modulo window entirely. check_gpu_pod_ready was re-fetching every container's full log for every pod every 5s; collection now runs on a 30s cadence, forced before both exit paths, while readiness polling stays at 5s. The artifact upload moves to always().

Still not enforceable: a job runs roughly 31 of these loops at 45 minutes each against a 90 minute cap, so two slow ones back to back can exhaust it. That needs shorter budgets or a shared deadline, which felt like a separate discussion.

Testing

Both e2e jobs pass against real Holodeck infrastructure. The run exercised the
whole new path end to end: the OCI chart published and installed, the runner
read the remoteAccess kubeconfig and reached the API server, node-exec.sh
loaded the kernel modules over SSH, and the operator restart test killed the
container on the node with crictl while the runner watched it come back.
Diagnostics, artifact upload and credential cleanup all ran. Artifacts came out
at roughly 735 KB and 336 KB.

Not exercised yet: use_values_override: true, which is the one changed branch
the run skipped, and the timeout/cancellation path that always() exists for.

Locally, shellcheck reports no new findings on any modified file, and the
wall-clock conversions were checked against stubs — the old code overran a 10s
budget by 3.8x where the new code exits on time.

[rahulait]

few nits, sorry forgot to submit review before

[coderabbitai]

📝 Walkthrough

Walkthrough

The E2E workflows now run case scripts on the GitHub Actions runner with runner-side kubeconfig access. Helm, kubectl, and jq versions are pinned. Node operations use a local-or-SSH dispatcher. Kubernetes checks use request timeouts, deadline-based polling, API failure handling, and periodic diagnostics. Driver update and migration checks dynamically query GPU nodes and tolerate diagnostic command failures. Test logs and cluster diagnostics are archived after each run.

Merge Risk: 🔵 Low · up to 18444

The workflow change is mergeable with owner awareness that diagnostic collection can fail when the log directory path contains whitespace; quoting that path should prevent the bounded artifact-collection issue.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

🟡 Other comments (1)

tests/scripts/checks.sh-135-135 (1)

135-135: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Quote log_dir when creating the log directory.

When log_dir contains whitespace, this command creates separate paths. Later artifact writes use the intended quoted path and fail. The || true handlers then hide the missing diagnostics.

Proposed fix

-	mkdir -p ${log_dir}
+	mkdir -p "${log_dir}"

Source: Linters/SAST tools

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: QUIET

Plan: Team

Run ID: 8e7f88bc-3165-4ffa-9c15-35e5f16d86fc

📥 Commits

Reviewing files that changed from the base of the PR and between f12acb8 and 18444e9.

📒 Files selected for processing (10)

• .github/workflows/e2e-tests.yaml
• tests/README.md
• tests/holodeck.yaml
• tests/scripts/.definitions.sh
• tests/scripts/checks.sh
• tests/scripts/migrate-clusterpolicy-to-nvidiadriver.sh
• tests/scripts/node-exec.sh
• tests/scripts/node-operations.sh
• tests/scripts/update-clusterpolicy.sh
• tests/scripts/update-nvidiadriver.sh

Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review.