diff --git a/REUSE.toml b/REUSE.toml index 3be5549c2..4a725f3e4 100644 --- a/REUSE.toml +++ b/REUSE.toml @@ -265,3 +265,29 @@ SPDX-License-Identifier = "CC0-1.0" path = "dstack/crates/qemu-acpi/fixtures/*.bin" SPDX-FileCopyrightText = "NONE" SPDX-License-Identifier = "CC0-1.0" + +[[annotations]] +path = "test-suites/catalog/source-inventory.json" +SPDX-FileCopyrightText = "© 2026 Phala Network " +SPDX-License-Identifier = "Apache-2.0" + +[[annotations]] +path = "test-suites/catalog/configuration-inventory.json" +SPDX-FileCopyrightText = "© 2026 Phala Network " +SPDX-License-Identifier = "Apache-2.0" + +[[annotations]] +path = "test-suites/catalog/api-inventory.json" +SPDX-FileCopyrightText = "© 2026 Phala Network " +SPDX-License-Identifier = "Apache-2.0" + +[[annotations]] +path = "test-suites/catalog/source-coverage-map.json" +SPDX-FileCopyrightText = "© 2026 Phala Network " +SPDX-License-Identifier = "Apache-2.0" + +[[annotations]] +path = "test-suites/**" +precedence = "aggregate" +SPDX-FileCopyrightText = "© 2026 Phala Network " +SPDX-License-Identifier = "Apache-2.0" diff --git a/docs/testing/dstack-test-methodology.md b/docs/testing/dstack-test-methodology.md new file mode 100644 index 000000000..3e52268f1 --- /dev/null +++ b/docs/testing/dstack-test-methodology.md @@ -0,0 +1,145 @@ + + + +# dstack Test Methodology + +This document defines the common process for dstack release testing, from change analysis and risk assessment through execution, evidence collection, and release decisions. See the [test-case authoring specification](test-case-authoring-spec.md#dstack-test-case-authoring-spec) and [report output specification](test-report-output-spec.md#dstack-test-report-output-spec) for normative formats. + + +## 1. Objectives + +Testing must produce reproducible, auditable, and traceable release evidence—not merely show that a script once exited successfully. A conclusion must be traceable from a requirement or risk to a case, step, original command evidence, observation, and attachment. + +Testing is complete only when: + +1. every relevant change, requirement, and material risk has explicit coverage; +2. an executor unfamiliar with the implementation can reproduce each case; +3. the native AI session preserves executed commands and their raw output; +4. simulated and physical-hardware results are reported separately; +5. tools can recompute aggregate status from atomic case results; and +6. references, attachment digests, and statistics are machine-verifiable. + + +## 2. Artifact layers + +Do not mix these four layers: + +| Layer | Purpose | Immutable after execution starts | +|---|---|---:| +| Change audit | Establishes changed behavior, dependencies, and risks | Yes | +| Test plan | Defines scope, topology, cases, and execution order | Yes | +| Case specification | Defines preconditions, actions, and expected results | Yes | +| `results//` | Records versions, native sessions, observations, and attachments | No, while running | + +A plan uses exactly three semantic levels: chapter, section, and case. Its machine-readable execution order is defined by `index.json`; its top-level `README.md` is the executor's environment guide. + + +## 3. Workflow + +### 3.1 Audit the release delta + +Compare the previous released tag with the candidate commit. Inspect commits, pull requests, schemas, RPCs, command-line interfaces, configuration defaults, systemd units, image recipes, deployment manifests, migrations, and dependency changes. For every change record: + +- the user-visible or operational behavior; +- affected components and interfaces; +- compatibility direction and version combinations; +- failure modes and security impact; +- the requirement and risk IDs used by test cases; and +- whether physical TEE hardware is required. + +Generated changelogs alone are insufficient. Follow data and control flow across component boundaries. + +### 3.2 Build a risk-based coverage matrix + +Classify coverage as: + +- **new or changed functionality**: full positive, boundary, and relevant negative coverage; +- **regression**: behavior likely to be affected by shared code, configuration, images, protocols, or lifecycle changes; +- **compatibility**: supported mixed-version combinations and upgrade order; +- **security**: trust boundaries, identity, attestation, key handling, authorization, and secret disclosure; +- **operations**: install, upgrade, restart, recovery, logging, and diagnostics. + +Prioritize by impact, likelihood, detectability, and breadth. `P0` covers release-blocking trust, data-loss, availability, or primary-path risks; `P1` covers important supported behavior; `P2` covers lower-risk variants. + +### 3.3 Define environments + +The plan guide must describe topology, component endpoints, credentials, test data, health checks, concurrency constraints, cleanup, and prohibited operations. Record common software versions once in run-level context. A case records a version override only when it deliberately uses a different component version. + +Environment levels are: + +- **UNIT**: isolated code-level validation; +- **SIMULATOR**: no-TEE or mock-attestation execution; +- **INTEGRATION**: deployed multi-component system; +- **HARDWARE**: physical supported TEE hardware. + +Simulation may follow `docs/development-without-tee.md`; a no-TEE development guest may independently use `key_provider=tpm` when the SGX local key provider is unavailable. This does not run local-key-provider in a TPM mode or cover its SGX behavior. Simulation never proves hardware-specific boot, measurement, attestation, sealing, or device behavior. Such unconfirmed items must be called out separately in the report. + +### 3.4 Author and review cases + +Each case validates one independently decidable behavior and references at least one requirement or risk. Prefer three to eight logical steps. Every step defines an action and exact observable expected results. Do not write a separate failure criterion: any result that does not fully match the expected result is `FAIL`. + +Review the plan for change coverage, regression breadth, compatibility matrices, security boundaries, operational recovery, test-data isolation, and cleanup before execution. + +### 3.5 Execute + +The `run-plan` orchestration agent must first read the guide, index, and every +case specification. It processes cases in index order, starts an independent +case-agent session for each runnable case, and reads the completed result before +deciding about later cases. It may mark a later case `SKIPPED` without launching +it only when a recorded earlier non-PASS result demonstrably makes the later +case's prerequisite false or its result meaningless. Similarity, expected cost, +or a mere possibility of failure is not sufficient. Independent cases continue. + +Each case executor must: + +1. read the plan `README.md` and `index.json`; +2. execute cases in index order unless the guide explicitly permits parallelism; +3. start a fresh Codex or Claude session for each case; +4. execute real commands rather than infer outcomes; +5. preserve the native JSONL session as step evidence; +6. write only a shallow atomic `result.json`; and +7. continue to later independent cases after a case-level failure. + +The executor name and model are recorded by the runner. Secrets must never be emitted into sessions or artifacts. + + +## 4. Status model + +Case and step status is one of: + +- `PASS`: every expected result was fully observed; +- `FAIL`: at least one expected result was not fully observed; +- `BLOCKED`: an external prerequisite prevented the tested behavior from starting; +- `NOT_RUN`: execution was not attempted; +- `SKIPPED`: omission was explicitly authorized and explained. + +`PARTIAL` is forbidden. A completed run may contain any terminal case status. A run is `INCOMPLETE` only when required case result artifacts are missing. + +Product failure and test-infrastructure failure must be distinguished. A healthy system returning the wrong response is `FAIL`; an unavailable required laboratory host before the tested action begins is `BLOCKED`. + + +## 5. Evidence and traceability + +Every logical step must be supported by observed commands and raw output in the native session. Screenshots or other files are attachments, not replacements for command evidence where machine-readable evidence is available. Preserve timestamps, exit codes, stdout, stderr, and tool errors as supplied by the agent CLI. + +Use explicit HTML anchors for all chapters, sections, cases, and steps. Do not rely on renderer-specific heading slugs. `index.json` is the authority for ordering and paths; IDs remain stable after publication. + + +## 6. Compatibility testing + +Derive version combinations from supported deployment behavior rather than testing arbitrary permutations. For a rolling upgrade, cover at least: + +- latest control-plane services with both previous and latest guest images; +- persisted state created by the previous release and consumed by the candidate; +- protocol/schema defaults when one side omits newly introduced fields; +- upgrade order, restart behavior, and rollback where supported; and +- explicit rejection of unsupported combinations with actionable diagnostics. + +For dstack v0.6.0, the expected online topology includes latest VMM, KMS, and gateway components while instances may use a mixture of old and new images. + + +## 7. Release decision + +The final report must provide coverage by requirement and risk, status counts, unresolved failures, blocked or skipped cases, simulation-only results, unconfirmed hardware items, and material deviations from the plan. Release acceptance criteria belong in the plan guide and must state which statuses or open risks block release. + +Before publishing, run `dstack-test validate`, render the self-contained HTML report, and package the selected run. The package is an immutable review artifact and must not include secrets or results from unrelated run IDs. diff --git a/docs/testing/test-case-authoring-spec.md b/docs/testing/test-case-authoring-spec.md new file mode 100644 index 000000000..36a73eb6d --- /dev/null +++ b/docs/testing/test-case-authoring-spec.md @@ -0,0 +1,299 @@ + + + +# dstack Test-Case Authoring Specification + +This document defines the normative layout and content of a dstack test plan and its `case.md` files. See the [test methodology](dstack-test-methodology.md#dstack-test-methodology) and [report output specification](test-report-output-spec.md#dstack-test-report-output-spec). + + +## 1. Plan layout + +```text +/ +├── index.json +├── README.md +├── results// +└── / + ├── README.md # optional + └──
/ + ├── README.md # optional + └── / + ├── case.md + ├── fixtures/ # optional + ├── scripts/ # optional + └── results// +``` + +Only chapter, section, and case are semantic organization levels. Each case has its own directory and a specification named `case.md`. + + +## 2. IDs and anchors + +All referenceable objects use explicit, stable, globally unique ASCII IDs. Use lowercase letters, digits, and hyphens. IDs must not change when titles change. Place `` before each referenceable heading and use relative paths with fragments for cross-file links. + +Recommended forms: + +```text +chapter-gateway +section-gateway-proxy-protocol +tc-gw-pp-001 +tc-gw-pp-001-step-01 +req-gw-pp-001 +risk-gw-spoofing-001 +``` + + +## 3. Top-level guide + +The top-level `README.md` is the first document an executor reads. It must state: + +1. objectives and scope; +2. system topology and system under test; +3. hardware, software, account, and external-service requirements; +4. reproducible common setup commands; +5. shared preconditions and health checks; +6. status rules and release acceptance criteria; +7. evidence, redaction, and attachment rules; +8. order, concurrency, and stop conditions; +9. cleanup and recovery; and +10. validation, packaging, and rendering commands. + +Avoid non-reproducible instructions such as “configure a working KMS.” + + +## 4. `index.json` + +The index is authoritative for discovery and order. A minimal example is: + +```json +{ + "schema_version": "1.0", + "id": "dstack-v0-6-0-release", + "title": "dstack v0.6.0 Release Test Plan", + "guide": {"path": "README.md", "anchor": "release-test-guide"}, + "chapters": [ + { + "id": "chapter-gateway", + "title": "Gateway", + "order": 1, + "path": "01-gateway", + "sections": [ + { + "id": "section-gateway-proxy-protocol", + "title": "Proxy Protocol", + "order": 1, + "path": "01-gateway/01-proxy-protocol", + "cases": [ + { + "id": "tc-gw-pp-001", + "title": "Forward a Proxy v1 client address over TLS termination", + "order": 1, + "priority": "P0", + "path": "01-gateway/01-proxy-protocol/tc-gw-pp-001", + "spec": { + "path": "01-gateway/01-proxy-protocol/tc-gw-pp-001/case.md", + "anchor": "tc-gw-pp-001" + }, + "requirements": ["req-gw-pp-001"], + "risks": ["risk-gw-spoofing-001"], + "tags": ["gateway", "proxy-protocol"] + } + ] + } + ] + } + ] +} +``` + +Array order must agree with `order`. Paths must remain below the plan root and must not contain absolute paths or `..` traversal. + +### Fixture and executor declarations + +Cases may declare an isolated fixture contract and the product actions that +fixture setup must not perform: + +```json +{ + "fixture": { + "profile": "vmm-empty-control-plane", + "capabilities": ["create_vm", "remove_vm"] + }, + "actions_under_test": ["Vmm.CreateVm"] +} +``` + +The fixture supplies substrate, dependencies, resource capacity, and a verified +initial state. The case performs the declared product action through the real +product interface. Mutable release tests must not reuse a long-lived shared +guest or control-plane instance. + +Execution defaults to the configured Agent. A deterministic case can instead +declare an executable entrypoint: + +```json +{ + "execution": { + "entrypoint": "01-gateway/01-proxy-protocol/tc-gw-pp-001/automation/run-test.py", + "args": [], + "timeout_seconds": 600 + } +} +``` + +The path is relative to the plan root, must remain inside that root, must be a +regular executable file, and must contain a shebang. Arguments are passed as an +argv array without a shell. The script writes the same `result.json`, evidence, +and attachments as an Agent case. Its exit code describes executor health; the +validated `result.json` describes the product result. + + +## 5. Required `case.md` structure + +Use this order: + +```markdown + +# TC-EXAMPLE-001: Title + +## Metadata + +## Objective + +## Preconditions + +## Test Data + +## Steps + + +### Step 1: Step title + +Action instructions. + +**Expected results:** + +- First observable result. +- Second observable result. + +## Postconditions +``` + + +## 6. Metadata + +At minimum include case ID, priority (`P0`, `P1`, or `P2`), type (for example Functional, Security, Compatibility, Regression, or Performance), minimum environment level, automation suitability, and requirement/risk references. + +Do not repeat common versions in every case. When a case requires an old or special component version, add a **Special Version Requirements** field and record the actual value as a result-level version override. + + +## 7. Objective + +Define one independently decidable behavior: the relevant configuration or state, the action, and the essential externally observable result. Split a case when its title contains multiple independent “and” clauses. An implementation function name or a script's zero exit status is not a product objective. + + +## 8. Preconditions + +Preconditions must be verifiable and distinct from the tested action. Put shared environment conditions in the plan guide and only case-specific conditions in the case. If a prerequisite fails before the tested behavior starts, preserve evidence and report `BLOCKED`; do not report a product `FAIL` for setup failure. + + +## 9. Test data + +Prefer JSON blocks for protocol fields, boundary values, and expected values. Document how random values are generated and preserve the actual values in results. Use RFC documentation address ranges for security-test addresses. Never put reusable credentials, tokens, private keys, or production secrets in plan files. + + +## 10. Steps and expected results + +Prefer three to eight logical steps. One logical step may invoke several mechanical commands, but it validates one phase. Every step must have: + +1. an explicit unique anchor; +2. a reproducible action; +3. precise, observable, comparable expected results; and +4. at least one item of command evidence in the native session. + +Do not write a separate failure criterion. If the actual result does not fully satisfy the expected result, the step and case are `FAIL`. Replace vague words such as “normal,” “correct,” or “without errors” with exact states, fields, addresses, digests, counts, or response codes. + + +## 11. Postconditions + +State which data is removed, which services are stopped or retained, how modified policy/configuration is restored, which state is intentionally retained for later cases, and how cleanup failures are recorded. Cleanup failure does not erase the original test result but must be reported. + + +## 12. Proxy Protocol example + +````markdown + +# TC-GW-PP-001: A PP-enabled application port receives the client address + +## Metadata + +- Priority: P0 +- Type: Functional, Regression, Security +- Environment: INTEGRATION +- Requirements: req-gw-pp-001 +- Risks: risk-gw-spoofing-001 + +## Objective + +Verify that, when inbound Proxy Protocol is enabled and application port 8443 declares `pp=true`, gateway forwards the Proxy v1 client address to the application and completes the following HTTP request. + +## Preconditions + +1. Gateway has `inbound_pp_enabled=true`. +2. The guest is registered and port 8443 has `pp=true`. +3. The capture backend is ready and gateway has loaded the instance port policy. + +## Test Data + +```json +{ + "source": "198.51.100.27:45678", + "destination": "203.0.113.10:8443", + "request_id": "tc-gw-pp-001" +} +``` + + +### Step 1: Check effective policy + +Query the guest and gateway port policy. + +**Expected results:** The instance ID matches; port 8443 exists and has `pp=true`. + + +### Step 2: Reset capture state + +Clear earlier capture records. + +**Expected results:** The backend is ready and contains zero records. + + +### Step 3: Send the request + +Send a Proxy v1 header, then complete TLS and HTTP on the same connection. + +**Expected results:** TLS succeeds, HTTP returns 200, and the request ID matches. + + +### Step 4: Inspect the capture + +Query the backend capture records. + +**Expected results:** Exactly one new record exists; source and destination match the test data and the HTTP request is complete. +```` + + +## 13. Review checklist + +Before submission, confirm that: + +- directory, case ID, and index entry agree; +- explicit anchors are present and unique; +- each case references a requirement or risk; +- the objective has one core behavior; +- preconditions are verifiable; +- every step has precise expected results and raw evidence; +- step count is reasonable; +- simulation and physical-hardware requirements are explicit; +- postconditions restore the environment; and +- no secret or environment-private value is present. diff --git a/docs/testing/test-report-output-spec.md b/docs/testing/test-report-output-spec.md new file mode 100644 index 000000000..4f50ded3e --- /dev/null +++ b/docs/testing/test-report-output-spec.md @@ -0,0 +1,300 @@ + + + +# dstack Test Report Output Specification + +This document defines the normative format for AI sessions, case summaries, run summaries, attachments, cross-references, packages, and self-contained HTML reports. See the [test methodology](dstack-test-methodology.md#dstack-test-methodology) and [case authoring specification](test-case-authoring-spec.md#dstack-test-case-authoring-spec). + + +## 1. Principles + +1. `run-plan` uses one AI orchestration session to drive the `next-case` loop, + and each case it executes runs in an independent Codex or Claude session. +2. The agent's native JSONL is the primary source for commands, tool calls, and raw output. +3. The agent writes one shallow `result.json`; it does not copy command output into that file. +4. The runner generates `runner.json`, timestamps, exit code, checksums, and run aggregates. +5. Common versions and environment information appear once at run level. +6. Screenshots, long logs, and binary captures are separate attachments. +7. Stable anchors make every object linkable in one offline HTML report. + + +## 2. Command interface + +The single public command is `dstack-test`, with consistently named subcommands: + +```text +dstack-test run-case +dstack-test run-plan +dstack-test finalize +dstack-test validate +dstack-test package +dstack-test render +``` + +Options use kebab-case. Execution defaults to Codex; select Claude with +`--agent claude`. Do not introduce separate `--codex` or `--claude` switches. +`run-case` and `run-plan` generate a unique run ID when `--run-id` is omitted. +Commands that operate on an existing run still require its ID. + + +## 3. Case execution + +```bash +dstack-test run-case \ + --plan \ + --case \ + --workdir \ + -- "Additional execution constraints" +``` + +The runner supplies the plan guide, `case.md`, output location, status rules, and result schema in the prompt. The agent must read the guide before the case and must not modify plan specifications. + +For a dependency-driven skip, the orchestrator creates a synthetic one-event +case session containing the reason and causal earlier case IDs. It does not +pretend that the skipped case was executed. The full decision process remains +available in the run-level `orchestrator.jsonl`. + + +## 4. Result layout + +```text +/results// +├── run.json +├── context.json # optional +├── case-manifests/ +├── case-lifecycle/ +├── leases/ +├── attempts/ +├── cases/ + └── /
// + ├── prompt.md + ├── session.jsonl + ├── agent-stderr.log + ├── runner.json + ├── result.json + ├── artifacts/ + ├── fixture/ + │ ├── runtime-manifest.json + │ ├── lease.json + │ └── cleanup.json + └── SHA256SUMS +└── SHA256SUMS +``` + +One run is one self-contained directory. Its `cases/` tree mirrors the indexed +chapter, section, and case specification paths. A non-empty case result must +not be overwritten unless the caller explicitly supplies `--overwrite`. + + +## 5. `session.jsonl` + +For Agent execution, the runner stores the native Agent CLI event stream +without rewriting it. For script execution it stores `process.started`, +`stdout`, `stderr`, and `process.exited` JSON objects. Every non-empty line is +one complete JSON object. Renderer adapters normalize these formats only for +display; the stored file remains unchanged. + +The agent should include the complete step ID when beginning and completing a step: + +```text +tc-gw-pp-001-step-01 +``` + +The renderer links matching session events to the step. If no marker is found, it exposes the complete session as fallback evidence rather than inventing a narrower association. + + +## 6. `runner.json` + +Only `dstack-test` writes this file: + +```json +{ + "schema_version": "1.0", + "run_id": "run-20260723-001", + "case_id": "tc-gw-pp-001", + "executor": {"type": "agent", "agent": "codex", "model": "gpt-5-codex"}, + "session": { + "format": "codex-jsonl", + "path": "session.jsonl", + "events": 42 + }, + "prompt_path": "prompt.md", + "result_path": "result.json", + "started_at": "2026-07-23T18:00:00.000Z", + "finished_at": "2026-07-23T18:04:32.000Z", + "duration_ms": 272000, + "exit_code": 0, + "result_valid": true, + "result_error": null +} +``` + +Historical files may retain the legacy top-level `agent` field. New files use +`executor`. A script executor additionally records its entrypoint, argv, and +entrypoint SHA-256. Extract an Agent model name from the session when possible, +otherwise use explicit `--model`, then `unknown`; never guess. The exit code +describes executor infrastructure, not product status. A valid product `FAIL` +may accompany any executor exit code. + +Fixture success is separate from product success. `lease.json` records exact +resource ownership, while `cleanup.json` records forced teardown. A product +`PASS` with failed fixture cleanup retains the product observation but makes +the overall execution `INFRA_ERROR`; validation and packaging must reject the +run until the leak is reconciled. + + +## 7. Shallow `result.json` + +Before exiting, the agent atomically writes: + +```json +{ + "schema_version": "1.0", + "case_id": "tc-gw-pp-001", + "status": "PASS", + "summary": "Gateway forwarded the Proxy v1 address to the PP-enabled application port.", + "steps": [ + { + "id": "tc-gw-pp-001-step-01", + "status": "PASS", + "observed": "Gateway's cached port 8443 policy had pp=true." + }, + { + "id": "tc-gw-pp-001-step-02", + "status": "PASS", + "observed": "The capture backend was ready and initially contained zero records." + } + ], + "artifacts": [ + { + "name": "Backend capture", + "path": "artifacts/backend-capture.json" + } + ], + "remarks": "" +} +``` + +Constraints: + +- status is `PASS`, `FAIL`, `BLOCKED`, `NOT_RUN`, or `SKIPPED`; +- `PARTIAL` is forbidden; +- `PASS` requires at least one step and all steps must be `PASS`; +- step IDs must come from `case.md`; +- `observed` is a concise observation, not copied raw command output; +- artifact paths must remain inside the result directory; and +- results must not contain tokens, private keys, or other secrets. + + +## 8. Attachments + +Put screenshots, long logs, JSON responses, and binary captures under `artifacts/`; reference them by name and relative path in `result.json`. Generate `SHA256SUMS` during finalization. The renderer displays images inline, shows text and JSON in collapsible blocks, and provides embedded download links for other files. Absolute paths and `..` traversal are forbidden. + + +## 9. Run summary + +After all cases, `dstack-test finalize` scans case outputs and generates `run.json`: + +```json +{ + "schema_version": "1.0", + "id": "run-20260723-001", + "anchor": "run-20260723-001", + "plan_id": "dstack-v0-6-0-release", + "status": "COMPLETED", + "started_at": "2026-07-23T18:00:00.000Z", + "finished_at": "2026-07-23T20:00:00.000Z", + "executors": [{"type": "codex", "model": "gpt-5-codex"}], + "software_under_test": { + "repository": "Dstack-TEE/dstack", + "candidate": "0123456789abcdef", + "previous_release": "v0.5.11" + }, + "environment": {"level": "INTEGRATION", "simulated": true}, + "summary": { + "total": 1, + "completed": 1, + "by_status": { + "PASS": {"count": 1, "case_refs": ["#result-tc-gw-pp-001"]}, + "FAIL": {"count": 0, "case_refs": []}, + "BLOCKED": {"count": 0, "case_refs": []}, + "NOT_RUN": {"count": 0, "case_refs": []}, + "SKIPPED": {"count": 0, "case_refs": []} + } + }, + "case_results": [ + { + "id": "tc-gw-pp-001", + "anchor": "result-tc-gw-pp-001", + "status": "PASS", + "result_path": "../../01-gateway/01-proxy-protocol/tc-gw-pp-001/results/run-20260723-001/result.json" + } + ] +} +``` + +Supply common versions and environment data through `--context `. Put exceptional component versions in the relevant case result only. + + +## 10. Validation + +```bash +dstack-test validate --plan --run-id +``` + +Validation must cover at least: + +1. plan paths, IDs, anchors, and index order; +2. one JSON object per non-empty session line; +3. required runner/result files and matching case/run IDs; +4. consistent case and step statuses; +5. safe, existing artifact paths; +6. recomputable run statistics; +7. no missing case in a completed run; and +8. complete and correct `SHA256SUMS` files. +9. released fixture leases and successful cleanup for every fixture-backed case. + + +## 11. Packaging + +```bash +dstack-test package \ + --plan \ + --run-id \ + --output -.tar.gz +``` + +Supported formats are `.tar.gz`, `.tgz`, `.tar`, and `.zip`. Validate before packaging. Include the complete plan and selected run, but exclude every other historical run stored beside it. + + +## 12. Self-contained HTML + +```bash +dstack-test render \ + --plan \ + --run-id \ + --output report.html +``` + +The output must work offline and inline all CSS, JavaScript, session events, text, JSON, images, and downloadable attachments. It must provide: + +- chapter/section/case navigation; +- the original guide and `case.md` requirements; +- common versions, environment, and executor model; +- status summaries, search, and status filters; +- each case summary and step observation; +- step-to-session-event links; +- collapsible original agent messages, tool calls, command output, and errors; +- every attachment and the complete raw session; and +- stable cross-reference anchors. + + +## 13. Live and historical dashboard + +`run-plan --web` starts a read-only HTTP dashboard for case status and the +native JSONL output of the orchestrator and every case agent. The browser polls +incremental byte ranges so active output appears without rewriting session +files. `dstack-test serve --plan --run-id ` exposes the same view +for a completed or interrupted run. The server has no built-in authentication; +non-loopback binding is permitted only on a trusted network or behind an +authenticated tunnel. diff --git a/test-suites/PROGRAMMATIC-EXECUTION.md b/test-suites/PROGRAMMATIC-EXECUTION.md new file mode 100644 index 000000000..dff6d8c0d --- /dev/null +++ b/test-suites/PROGRAMMATIC-EXECUTION.md @@ -0,0 +1,174 @@ +# Programmatic execution of this test plan + +This plan used to be driven by an LLM orchestrator that decided, case by case, +what to run. Measured over run `central-fixtures-20260724T032131Z` that cost +43 of 72 wall-clock hours in orchestrator stalls and per-decision turns, at a +maximum concurrency of one. The loop is now deterministic and the AI is out of +the execution path. + +## Running the plan + +```sh +# Deterministic driver (default). Replaces the LLM orchestrator with the loop +# it was already constrained to: next-case, then run-case or complete-case. +test-suites/runner/dstack-test run-plan --plan --driver program + +# Fast regression over every case that owns a checked-in harness. +test-suites/runner/dstack-test sweep --plan --run-id --workers 8 \ + --runtime-manifest /runtime-manifest.json + +# Registry integrity: every promoted case must be backed by a harness that +# actually handles it. +test-suites/runner/dstack-test verify-registry --plan +``` + +On a physical TDX host, create the runtime manifest with +`shared/automation/prepare-hardware-run.sh` rather than calling `prepare-run.sh` +directly. The wrapper makes every external provider and deterministic tool/data +prerequisite part of the prepared run; plain preparation cannot provision those +lab-specific inputs. + +A scripted case averages 0.4s against 178s for an agent-driven one, so the +whole scripted set sweeps in seconds. That is what makes "fix, then re-verify" +cheap enough to do on every change. + +## The rule that matters + +A case is only scripted when its harness reproduces what the case claims to +test. Two mechanisms enforce this, both added after the registry was found +asserting things that were not true: + +- `verify-registry` rejects a promotion whose harness does not handle the case. + It caught nine cases registered against `passed-rpc-case.py` whose table + never listed them; every rerun had been dying with `KeyError` while the + registry reported them as deterministic passes. +- `shared/automation/mine-passing-attempt.py` refuses to emit a spec when it cannot + template every recorded operation. Under the earlier permissive rule, five of + eight verified specs were replaying only part of their recorded operations. + +Prefer an honest `BLOCKED` or an unregistered case over a harness that passes +without exercising the behaviour. + +## Adding a harness + +Most cases fall into a family that already has a table-driven harness in +`shared/automation/`. Extending a table is cheaper and more reviewable than writing a +new script: + +| family | harness | +| --- | --- | +| guest-agent simulator RPC | `passed-rpc-case.py` | +| VMM RPC | `passed-vmm-empty-rpc-case.py` | +| gateway RPC | `passed-gateway-empty-rpc-case.py` | +| gateway ZT domains | `passed-gateway-zt-domain-case.py` | +| KMS RPC | `passed-kms-rpc-case.py` | +| replay of a mined attempt | `replay-case.py` + `shared/automation/replay/.json` | + +A harness reads `DSTACK_TEST_CASE_MANIFEST` for its lease-owned fixture, prints +`STEP`/`EVIDENCE` markers, and writes `result.json` plus artifacts. Never hard-code +a port, a workspace path, or the candidate repository: they differ per lease. + +### Contract limits to respect + +The RPC harnesses call each method over JSON, then protobuf, then once more, +and require every call to succeed. That fits idempotent methods only. +`Vmm.RemoveVm` is not idempotent; `Vmm.ShutdownVm`, `Vmm.SvStop` and +`Vmm.SvRemove` need a running guest and supervisor process that the prepared +stopped VM does not have. Those need a harness that models a state transition +rather than repeating one call. + +Do not assert determinism without checking. `Vmm.GetAppEnvEncryptPubKey` +returns a timestamp and signatures over it, so two identical requests match +byte for byte only within the same second: it passes alone and fails under a +parallel sweep. + +## Substrate settings belong to the run + +Fixture providers read lab-specific locations from the environment. Declare +them in `an operator-owned lab manifest`, which `prepare-run.sh` merges into +`runtime-manifest.json`; `run-case` exports them before provisioning. A missing +variable used to surface as a fixture `INFRA_ERROR` indistinguishable from a +real capability gap — 60 `BLOCKED` and 15 `INFRA_ERROR` results in one sweep +were nothing but an unset variable. + +Use `environment` for plain values and `environment_path_prepend` for toolchain +directories, since `PATH` is always set and cannot use the set-when-unset rule. + +## Script coverage + +The suite currently contains 358 cases. Distributed case metadata declares a +checked-in execution entrypoint for 357 of them. The remaining macvtap +connectivity case is agent-driven until it has a reproducible harness. + +## Known substrate defect + +`physical-tdx` targets an external shared VMM through `DSTACK_TEST_VMM_URL`, +default `127.0.0.1:12000`. That instance runs from a deleted working directory, +so `CreateVm` fails with "Failed to load image" and every hardware case errors. +The provider should own a per-lease VMM the way `isolated-component` does. + +When copying that provider's VMM startup, note that it passes a +`simulator_seed` unconditionally, which appends a `[cvm.tee_simulator]` block +and yields software-simulated quotes. A provider that exists to produce real +hardware quotes must pass an empty seed. + +## The 86 BLOCKED/SKIPPED results are not a finished category + +Run `central-fixtures-20260724T032131Z` left 86 cases BLOCKED or SKIPPED. It is +tempting to read those as settled — a capability the lab does not have. Reading +every summary shows otherwise. They almost all say some variant of *the fixture +did not provide this*: + +- "the fixture lacks the case-scoped local PCCS/key-provider lifecycle" +- "the prepared no-tee-dev simulator fixture lacks the required case-owned TPM + simulator/proxy endpoint" +- "the gateway cluster was healthy, but the fixture lacked the ACME/DNS + issuance path" +- "the case-owned fixture declares no cryptographically verifiable attested + client" +- "the image-assembly fixture provides no documented audit invocation + parameters" + +That is unprovisioned fixture work, not an unavailable capability, so those +cases do not qualify for a capability-based BLOCKED. Only nine mention +something the host genuinely may not offer — GPU, SEV-SNP, or hugepages: + + tc-gos-attestatio-006 tc-gos-gpupolicy-007 tc-gos-platform-009 + tc-gos-setup-011 tc-gos-yocto-006 tc-kms-release-010 + tc-ver-input-plat-005 tc-vmm-compute-ne-003 tc-vmm-compute-ne-004 + +and even those need a probe that demonstrates the absence rather than an +assertion that it is absent. + +The practical consequence: the remaining work is not only the 241 behavioural +harnesses. It also includes provisioning the capabilities those fixtures are +missing — a local PCCS, a TPM/vTPM proxy, an ACME/DNS issuance path, attested +KMS clients, image-assembly audit handles. Budget for that before treating the +BLOCKED column as closed. + +## What "no AI in the execution loop" does and does not mean today + +The orchestrator is gone: `run-plan --driver=program` decides what to run in +process, and `sweep` re-runs the scripted set with no model involved at all. +That is the path used for every result quoted here. + +It does not yet mean the whole plan runs without a model. `run_case` dispatches +on whether the case owns an entrypoint (test-suites/runner/dstack-test, around +line 1126): + + if case.execution is not None: + value = run_script_case(...) + else: + value = run_agent_case(...) + +So running the full plan today spawns an agent only for the single case without +an execution entrypoint. The other 357 cases execute deterministically. + +Two consequences worth keeping in mind: + +- Quote the scripted count alongside any "programmatic" claim. "356 of 358 cases + run deterministically" is true; "the entire plan runs without AI" is not yet. +- `run-plan --driver=program --require-script` refuses the first case without + an entrypoint instead of falling back, which makes the boundary enforceable + rather than conventional. It currently halts at `tc-vmm-compute-ne-009` + instead of spawning an agent. diff --git a/test-suites/README.md b/test-suites/README.md new file mode 100644 index 000000000..3257e4158 --- /dev/null +++ b/test-suites/README.md @@ -0,0 +1,216 @@ + + + +# dstack Core Components Full Test Plan + +The post-baseline merged-PR review is recorded in [`audit/core-components-post-baseline-pr-audit.md`](audit/core-components-post-baseline-pr-audit.md). + +## 1. Objective and scope + +This plan is a source-derived, full functional audit of the dstack guest OS, VMM, KMS, gateway, verifier, and their trust and compatibility boundaries. It covers every protobuf RPC method present at authoring time plus non-RPC boot, configuration, storage, networking, cryptographic, measurement, proxy, certificate, cluster, UI, operational, recovery, upgrade, and security behavior found in the component source trees. + +Execution order is discovered by sorting chapter, section, and case directory +names. Each directory owns its `metadata.json`. Traceability is in +`catalog/feature-audit.md`; the raw repository scan is `catalog/source-inventory.json` and the +mandatory 214-field configuration matrix is `catalog/configuration-inventory.json`, the complete protobuf field matrix is `catalog/api-inventory.json`, and reverse file-to-case traceability is `catalog/source-coverage-map.json`. +A source reference means the case must be reviewed when that implementation +surface changes. Passing existing unit tests is evidence for a step only when +the case explicitly runs them; it never substitutes for product-level expected +results. + +### Suite layout + +```text +test-suites/ +├── runner/ # dstack-test CLI, dashboard, and runner unit tests +├── cases/ # case specifications and case-local entrypoints +├── shared/ +│ ├── automation/ # multi-case or location-sensitive harnesses +│ └── fixtures/ # fixture profiles, providers, and test images +├── catalog/ # API, configuration, source, and coverage inventories +├── manifests/ # environment-specific run manifests +├── audit/ # retained historical audit evidence +└── metadata.json # suite-level identity and guide metadata +``` + +An entrypoint used by one case lives beside that case as `run.py`, `run.sh`, +or `run.cjs`. An entrypoint shared by multiple cases, or one that owns a group +of related helper files, lives under `shared/automation/`. A case's local +`metadata.json` is the authoritative machine-readable binding between its +specification and entrypoint. The runner scans and validates the distributed +metadata at startup; paths and order are derived from the directory tree. + +## 2. Repository scope + +| Chapter | Primary source roots | +|---|---| +| Guest OS | `os/`, `dstack/guest-agent`, `guest-api`, `supervisor`, `dstack-util`, `local-key-provider`, `tee-simulator` | +| VMM | `dstack/vmm`, `dstack/host-api` | +| KMS | `dstack/kms` including mock/simple/Ethereum authorization implementations | +| Gateway | `dstack/gateway`, `dstack/certbot` | +| Verifier | `dstack/verifier`, `dstack-mr`, `dstack-attest`, image artifact specification | +| Integration | `dstack/tests/e2e`, all cross-component protocols and persisted state | + +Before a release run, update `catalog/source-inventory.json`, compare RPC/config/source changes with this plan, and add or amend cases before execution. + +## 3. Required topology + +Prepare isolated namespaces and credentials for: + +1. one control host with the candidate repository and `dstack-test`; +2. at least two VMM nodes when cluster/failover behavior is tested; +3. at least three KMS/gateway nodes for rolling-upgrade and partition cases; +4. pinned `v0.5.4`, `v0.5.8`, `v0.5.11`, and candidate guest images; +5. a private OCI registry capable of bearer authentication and fault injection; +6. DNS zones and an ACME staging account, never a production ACME account; +7. controllable HTTP/TCP/TLS/Proxy-Protocol capture backends; +8. an Ethereum development chain and deployed test authorization contract; +9. a fault-injection network supporting latency, loss, partition, and clock-control; +10. a log/artifact sink with secrets redaction. + +Use unique run-scoped domains, ports, app IDs, instance names, DNS records, registry tags, and storage paths. Never point destructive Admin, Exit, Clear, Remove, Delete, or certificate cases at production. + +## 4. Environment levels + +- `UNIT`: repository build/test tools and committed fixtures only. +- `SIMULATOR`: follow `docs/development-without-tee.md`. If the SGX local key provider is unavailable, a no-TEE development guest may independently use `key_provider=tpm`; this does not run local-key-provider in a TPM mode and does not cover SGX local-key-provider behavior. +- `INTEGRATION`: deployed multi-component environment; a TEE simulator is allowed only when the case does not claim hardware properties. +- `HARDWARE`: supported physical TDX/TDX-lite, SEV-SNP, GCP TDX, Nitro TPM, or GPU hardware as named by the case. + +Simulation is not confirmation of measured boot, quote/certificate collateral, physical device isolation, sealing, TPM/PCR behavior, GPU attestation, or platform firmware measurements. A simulator result must be labeled simulated. If a hardware case is run only under simulation, report it separately as unconfirmed; do not mark the hardware case PASS. + +## 5. Common setup and context + +Record actual component commits, image digests, firmware/QEMU/kernel versions, authorization implementation and contract, registry, DNS provider, ACME directory, TEE hardware, and topology once in the run context: + +```json +{ + "software_under_test": { + "repository": "Dstack-TEE/dstack", + "candidate": "", + "compatibility_releases": ["v0.5.4", "v0.5.8", "v0.5.11"], + "guest_images": { + "v0.5.4": "", "v0.5.8": "", + "v0.5.11": "", "candidate": "" + }, + "vmm": "", "kms": "", "gateway": "", "verifier": "" + }, + "environment": { + "level": "HARDWARE", + "simulated": false, + "topology": "" + } +} +``` + +Use the generated pRPC clients or a pinned generic pRPC helper. Preserve request and response bodies after redacting credentials. Capture effective TOML, systemd unit state, QEMU command line, VM configuration, image/compose hashes, component health, and synchronized clocks before case execution. + +## 6. Execution rules + +1. Read this guide and the current case before acting. +2. Execute cases in discovered directory order unless the orchestrator proves a recorded dependency makes a later case meaningless. +3. Every executed case gets an independent Agent session. Commands and raw outputs remain in `session.jsonl`. +4. A case is PASS only when every expected result is fully observed. There is no separate failure criterion. +5. Use BLOCKED only when an external prerequisite prevents the tested behavior from starting. +6. Use SKIPPED only for an authorized omission or a proven dependency consequence, with causal case IDs. +7. Do not change a product configuration merely to force an expected result unless the case instructs that change. +8. Do not restart physical hosts; cases requiring it must use VM/service/device-level recovery or be reported unconfirmed. +9. Stop a destructive case immediately if its target identity is not the isolated run-scoped environment. +10. Continue independent chapters after failures. + +### 6.1 Prepared execution environment + +Prepare immutable build inputs once before starting a run: + +```bash +run_id= +shared/automation/prepare-run.sh \ + "$(git rev-parse --show-toplevel)" \ + "results/$run_id/runtime-manifest.json" +``` + +Export the resulting path as `DSTACK_TEST_RUNTIME_MANIFEST` when invoking the +runner. `dstack-test` also discovers this standard run-relative path +automatically and exports its shared Cargo target and cache directory to every +case Agent. + +Every case contains a **Prepared execution knowledge** section. Together with +[`shared/automation/execution-guide.md`](shared/automation/execution-guide.md), it is the +complete initial execution specification. Agents must use the prepared binary +and case-scoped simulator helpers rather than copying Cargo registries, +creating private target trees, browsing earlier sessions, or rebuilding the +same candidate for each RPC method. Clean-build cases remain clean and must not +claim cached output as build evidence. + +Run with the live dashboard: + +```bash +test-suites/runner/dstack-test run-plan \ + --plan test-suites \ + --context run-context.json \ + --web \ + -- "Do not restart physical hosts" +``` + +Resume an interrupted run with its printed run ID: + +```bash +test-suites/runner/dstack-test run-plan \ + --plan test-suites \ + --run-id --resume --web +``` + +## 7. Evidence and redaction + +Each logical step must have at least one observed command/tool result in the native Agent session. Attach packet captures, screenshots, QEMU arguments, measurement calculations, certificates, manifests, synchronized cluster snapshots, or long logs under the case result `artifacts/` directory. + +Never retain admin tokens, private keys, disk/env plaintext keys, DNS secrets, ACME account keys, Ethereum private keys, reusable cookies, or decrypted application secrets. Quotes, public certificates, public keys, hashes, and redacted configuration may be retained. For a redaction test, record hashes or sentinel-presence checks rather than the secret itself. + +## 8. Compatibility policy + +Compatibility cases keep VMM on the candidate release by default. Guest images +and online KMS, gateway, and verifier consumers may simultaneously include +`v0.5.4`, `v0.5.8`, `v0.5.11`, and the candidate. Test request distribution, +node loss, restart, state synchronization, old/new client-server directions, +protobuf optional and unknown fields, persisted old state, rolling cutover and +explicit rejection of unsupported combinations. Record the exact tag, commit, +image digest, QEMU, firmware, and backported patch set for every historical +node as a case-level override. + +### 8.1 KMS onboarding to the 0.6.0 candidate + +Follow the validated matrix in [PR #705](https://github.com/Dstack-TEE/dstack/blob/203e09bcbce27e566f157d2b6ed4657eb949459a/docs/operations/kms-upgrade-plan.md): + +| Source KMS | Required path to the 0.6.0 candidate | +|---|---| +| `v0.5.4` | `0.5.4 → 0.5.7 bridge → 0.6.0` | +| `v0.5.8` | direct to `0.6.0` | +| `kms-v0.5.11` | direct to `0.6.0`; record whether PR #693 is included | + +The candidate target must boot on its matching candidate OS with **legacy TDX +attestation**, never lite or an `auto` decision that resolves to lite, while an +old source verifies it. The latest VMM must use +`qemu_single_pass_add_pages=true` and `qemu_pic=true`. Both source and target +`mrAggregated` values and the target image hash must be authorized; the source +must download the target verifier archive. A healthy onboard preserves the CA, +root k256 public key, existing application keys, and certificate trust. + +Direct `0.5.4 → 0.6.0` is a required negative test: it must fail before key +transfer because 0.5.4 cannot extract the versioned RA-TLS attestation OID. +Use QEMU 9.1.50-era `dstack-acpi-tables` when diagnosing 0.5.4 measurements. +Upgrade gateway only after KMS 0.6.0 key and certificate operations pass, and +retain old KMS/gateway nodes for a tested rollback window. + +## 9. Cleanup + +Delete run-scoped VMs, workdirs, taps, port mappings, GPU bindings, registry artifacts, DNS records, ACME staging orders, WaveKV objects, authorization contracts/state, temporary KMS nodes, certificates, storage volumes, firewall rules, and fault-injection rules. Verify host devices and services returned to their baseline. Preserve only redacted report artifacts. + +## 10. Finalization + +```bash +test-suites/runner/dstack-test validate --plan test-suites --run-id +test-suites/runner/dstack-test render --plan test-suites --run-id --output report.html +test-suites/runner/dstack-test package --plan test-suites --run-id --output report.tar.gz +``` + +The release summary must list every FAIL, BLOCKED, SKIPPED, NOT_RUN, simulation-only result, hardware-unconfirmed item, compatibility gap, and deviation from this plan. diff --git a/test-suites/audit/core-components-post-baseline-pr-audit.md b/test-suites/audit/core-components-post-baseline-pr-audit.md new file mode 100644 index 000000000..288726d68 --- /dev/null +++ b/test-suites/audit/core-components-post-baseline-pr-audit.md @@ -0,0 +1,51 @@ + + +# Core component post-baseline pull request audit + +PR #841 was last fully exercised against `next` at `cb961ad7877b0f2f60abfba73fdcd6dbc11b5c39` with candidate head `c7364e9e84410097ff6fa0952750af697938df0c`. This audit covers first-parent merges through `89fe3184ba46143324e27acf94b762db4e393e6c`. + +| PR | Change area | Acceptance coverage after audit | +| --- | --- | --- | +| #837 | Libvirt-filtered VMM networking | `tc-vmm-compute-ne-001`, `tc-vmm-compute-ne-007` | +| #1023 | Pre-launch ordering documentation | Documentation-only; no executable behavior added | +| #1025, #1026 | Branch/CI/repository rename | Repository workflow validation; no runtime case added | +| #1027 | Atomic Gateway refresh failover | `tc-gos-observabil-003`, `tc-gos-setup-009` | +| #1038 | TDX V2 event preimage integrity | `tc-gos-setup-018`, `tc-ver-input-plat-003` | +| #1040 | dstackup CID-window allocation | `tc-vmm-internal-002` | +| #1039 | Simulator vTPM device/state race | `tc-gos-setup-013`, `tc-gos-setup-015` | +| #1034 | Auth-mock dependency update | Existing KMS authorization and build gates | +| #1030 | Named MessagePack encoding | `tc-gos-attestatio-002`, `tc-int-mixed-007` | +| #1036 | Gateway sync authentication and bounds | `tc-gw-cluster-ad-002` | +| #1037 | Gateway Prometheus metrics | `tc-gw-cluster-ad-004`, `tc-gos-observabil-001` | +| #1043 | Guest SELinux parity | `tc-gos-platform-005` | +| #1042 | Guest nftables/netfilter parity | `tc-gos-platform-005`, `tc-gos-observabil-003` | +| #1035 | Gateway KV validation and recovery | `tc-gw-kv-009`, `tc-gw-cluster-ad-001` | +| #1044 | Administrative CVM removal | New `tc-gw-admin-034` | +| #1046 | Rejected-record and node recovery APIs | New `tc-gw-admin-035`, `tc-gw-admin-036` | +| #1048 | GPU secondary-bus-reset sanitization | `tc-vmm-compute-ne-004` | +| #1050, #1052, #1053 | Rust QEMU ACPI generation and profiles | `tc-vmm-compute-ne-007`, `tc-ver-image-meas-003` | +| #1051 | Lite-TDX ACPI verification | `tc-ver-image-meas-003`, `tc-ver-input-plat-003` | +| #1057 | Auth-mock lockfile synchronization | Dependency lock only; KMS authorization and build gates apply | +| #1056 | Lite-TDX ACPI digest generation | `tc-gos-setup-018`, `tc-ver-image-meas-003`, `tc-ver-input-plat-003` | +| #1059 | s2n-quic dependency update | Dependency-only; Gateway build, RPC, proxy, and cluster gates apply | +| #1054 | Streaming `dstack-util` encrypt/decrypt | New `tc-gos-setup-025` | +| #1064 | Explicit netd bridge preparation RPC | `tc-vmm-compute-ne-001`, `tc-vmm-compute-ne-009` | +| #1060 | Multiple Gateway clusters | `tc-gos-setup-009` and multi-cluster Gateway integration cases | +| #1067 | Materialized WaveKV proxy winner | `tc-gw-kv-009`, `tc-gw-cluster-ad-001` | +| #1031 | WaveKV v2 delta-state synchronization | `tc-gw-admin-010`, `tc-gw-cluster-ad-001`, `tc-gw-cluster-ad-002`, `tc-gw-kv-009` | +| #1061 | Netd-managed macvtap networking | `tc-vmm-compute-ne-009` | +| #1068 | Restricted deployment network overrides | `tc-vmm-compute-ne-009` | +| #1069 | Secure netd socket activation | `tc-vmm-compute-ne-009` | +| #1070 | KMS RPC endpoint normalization | Corrected `tc-gos-setup-006` | +| #1071 | SEV-SNP simulator ABI semantics | `tc-gos-setup-014` | +| #1072 | Stable Certbot certificate ordering | Extended `tc-gw-certbot-005` | +| #1073 | Guest image builder provenance | New `tc-gos-build-001` | +| #1074 | Source-local component tests and fixtures | Existing component cases consume the tests and prepared fixture binary; no new runtime behavior | + +The three new Gateway Admin methods were absent from the previous API inventory and were the only newly merged public RPC surface without a dedicated case. This branch adds their complete API inventory, case specifications, deterministic authenticated smoke coverage, and recovery matrices. Existing cases are tightened below for the non-RPC regression surfaces. + +The audit also refreshes deterministic harness expectations invalidated by the +merged source changes: the renamed TDX simulator atomicity test, the renamed +lite-TDX verifier test, the expanded verifier and RA-TLS unit-test totals, and +the QEMU 10 RTMR0 delta and supported hugepage/NUMA row introduced by the new +ACPI generator. diff --git a/test-suites/audit/core-components-product-pr-accounting.md b/test-suites/audit/core-components-product-pr-accounting.md new file mode 100644 index 000000000..224dcf10d --- /dev/null +++ b/test-suites/audit/core-components-product-pr-accounting.md @@ -0,0 +1,13 @@ +# Core component product commit accounting + +This inventory accounts for every commit formerly carried by product PR #840. + +- `RETAINED`: the stable patch is present on one or more split product branches. +- `MANUAL`: the behavior is retained, but conflict resolution or upstream adaptation changed its stable patch ID. +- `EXISTING_PR`: an already-open independent PR carries the change. +- `SUPERSEDED`: a newer implementation is already present upstream. +- `REVERTED`: the historical commit and its revert have no net product effect. +- `REJECTED`: review determined that the historical change weakens the intended behavior, so it is deliberately not carried forward. +- `TEST_ONLY`: test-only, formatting, fixture, or test dependency work; it is not a standalone product bug. + +The TSV is the authoritative per-commit inventory. It contains all 332 commits with no unclassified rows. diff --git a/test-suites/audit/core-components-product-pr-accounting.tsv b/test-suites/audit/core-components-product-pr-accounting.tsv new file mode 100644 index 000000000..e778659e5 --- /dev/null +++ b/test-suites/audit/core-components-product-pr-accounting.tsv @@ -0,0 +1,333 @@ +commit subject disposition target +275031817 fix(os): omit built-in FUSE module package RETAINED codex/fix-os-fuse-package +83ec8739b fix(os): fail multi-flavor builds on first error RETAINED codex/fix-os-multiflavor-failure +dc549cfaa fix(simulator): support current FUSE soname MANUAL codex/fix-simulator-fuse-soname,codex/fix-simulator-tdx-configfs-shadow +52befa748 fix(simulator): tolerate udev TPM node race REJECTED discarded; master platform selection already enforces strict device creation +450e51b8d fix(simulator): wait for GCP vTPM readiness RETAINED codex/fix-simulator-gcp-vtpm-readiness +dc0a1cf2e fix(os): install TPM device TCTI for simulator RETAINED codex/fix-os-simulator-tpm-tcti +a83750d83 fix(simulator): expose GCP TPM event log MANUAL codex/fix-simulator-gcp-event-log +846f72194 fix(simulator): shadow securityfs for GCP event log RETAINED codex/fix-simulator-gcp-event-log +d547a559d test(simulator): log NitroTPM vendor commands RETAINED codex/fix-simulator-nitrotpm-pcrs +93a00d6d9 fix(simulator): advertise NitroTPM vendor command RETAINED codex/fix-simulator-nitrotpm-pcrs +703f9fecb style(simulator): apply repository rustfmt RETAINED codex/fix-simulator-nitrotpm-pcrs +0d3d6adba fix(vmm): generate simulated SEV-SNP mr_config RETAINED codex/fix-vmm-simulated-snp-mr-config +caf875fdc fix(vmm): pass cloud image measurements to guests RETAINED codex/fix-vmm-cloud-image-measurements,codex/fix-vmm-image-artifact-confinement,codex/fix-vmm-simulated-nitrotpm-measurement +d570101b9 style(vmm): apply repository rustfmt RETAINED codex/fix-vmm-cloud-image-measurements,codex/fix-vmm-image-artifact-confinement,codex/fix-vmm-simulated-nitrotpm-measurement +266e647b1 fix(vmm): align simulated NitroTPM measurement MANUAL codex/fix-vmm-simulated-nitrotpm-measurement +b019aa9d3 style(vmm): apply repository rustfmt RETAINED codex/fix-vmm-simulated-nitrotpm-measurement +7b936abfa fix(simulator): retry interrupted vTPM reads RETAINED codex/fix-simulator-interrupted-vtpm-read +93b75b1b3 fix(simulator): report live NitroTPM PCRs RETAINED codex/fix-simulator-nitrotpm-pcrs +523fdd197 chore(simulator): remove NitroTPM debug output RETAINED codex/fix-simulator-nitrotpm-pcrs +d3ce0365f fix(simulator): notify after NSM registration RETAINED codex/fix-simulator-measured-nitro-pcrs,codex/fix-simulator-nitro-pcr-state,codex/fix-simulator-nsm-readiness +9c9b83b9d fix(simulator): publish NSM device before ready RETAINED codex/fix-simulator-measured-nitro-pcrs,codex/fix-simulator-nitro-pcr-state,codex/fix-simulator-nsm-readiness +7ea071ec8 fix(simulator): stabilize NSM node before ready RETAINED codex/fix-simulator-measured-nitro-pcrs,codex/fix-simulator-nitro-pcr-state,codex/fix-simulator-nsm-readiness +31f6cbb98 fix(simulator): detect Nitro Enclave DMI RETAINED codex/fix-nitro-enclave-platform-detection +90d5edb84 fix(vmm): expose simulated platform through SMBIOS RETAINED codex/fix-nitro-enclave-platform-detection +ef923dcee fix(guest): skip TDX config check on Nitro Enclave RETAINED codex/fix-nitro-enclave-platform-detection +764e798c5 test(fixtures): require mkosi guest images TEST_ONLY move/account in test-infrastructure PR +a85d8bfa0 fix(guest): avoid ZFS cache writes on immutable root REVERTED net-zero historical pair +a7d524d39 "Revert ""fix(guest): avoid ZFS cache writes on immutable root""" REVERTED net-zero historical pair +88466ed3d fix(guest): report ZFS pool creation errors RETAINED codex/fix-guest-zfs-pool-errors,codex/fix-guest-zfs-root-mountpoint +57f430e76 fix(guest): disable the ZFS pool root mountpoint RETAINED codex/fix-guest-zfs-root-mountpoint +0fc9488c9 fix(os/mkosi): install the volume helper RETAINED codex/fix-mkosi-volume-helper +a30fd23eb fix(os/mkosi): stage the volume helper RETAINED codex/fix-mkosi-volume-helper +5a895283c fix(os/mkosi): enable Docker IPv4 NAT modules RETAINED codex/fix-mkosi-docker-ipv4-nat +dbfd6f385 fix(os/mkosi): use current xtables NAT symbols RETAINED codex/fix-mkosi-docker-ipv4-nat +6a06b1a54 fix(vmm): stop VM launchers gracefully through SvStop REVERTED net-zero historical pair +63d402278 "Revert ""fix(vmm): stop VM launchers gracefully through SvStop""" REVERTED net-zero historical pair +a72084da2 fix(vmm): reap VM launcher children on SvStop RETAINED codex/fix-vmm-svstop-child-reaping +93fccce42 fix(vmm): keep SvStop unknown IDs rejected RETAINED codex/fix-vmm-svstop-child-reaping +0f1748cd4 fix(vmm): resize uninitialized stopped VM manifests RETAINED codex/fix-vmm-resize-validation,codex/fix-vmm-stopped-manifest-resize +f68dcf7c3 test(vmm): cover per-instance TEE simulator matrix TEST_ONLY move/account in test-infrastructure PR +c7c2322b4 test(vmm): complete swtpm simulator decision matrix TEST_ONLY move/account in test-infrastructure PR +3a85ae4a7 fix(vmm): keep Host API on private vsock listener RETAINED codex/fix-vmm-private-host-api +12c2d4da6 fix(vmm): reject empty and zero ResizeVm updates MANUAL codex/fix-vmm-resize-validation +07b38ea5b fix(vmm): reject conflicting host port mappings MANUAL codex/fix-vmm-host-port-conflicts +9d6706ff4 fix(testing): import port protocol in conflict test TEST_ONLY move/account in test-infrastructure PR +d8200f25b fix(kms): validate onboarding domains RETAINED codex/feat-kms-historical-root-keys,codex/fix-kms-bootstrap-once,codex/fix-kms-ca-restart,codex/fix-kms-finish-response-order,codex/fix-kms-onboarding-domains,codex/fix-kms-private-key-permissions,codex/fix-kms-repeated-onboarding +a16919222 fix(kms): integrate onboarding domain tests RETAINED codex/feat-kms-historical-root-keys,codex/fix-kms-bootstrap-once,codex/fix-kms-ca-restart,codex/fix-kms-finish-response-order,codex/fix-kms-onboarding-domains,codex/fix-kms-private-key-permissions,codex/fix-kms-repeated-onboarding +7c67d46bd fix(kms): make Bootstrap one-time RETAINED codex/feat-kms-historical-root-keys,codex/fix-kms-bootstrap-once,codex/fix-kms-ca-restart,codex/fix-kms-finish-response-order,codex/fix-kms-private-key-permissions,codex/fix-kms-repeated-onboarding +daf8c1a02 fix(kms): persist private keys owner-only RETAINED codex/feat-kms-historical-root-keys,codex/fix-kms-ca-restart,codex/fix-kms-finish-response-order,codex/fix-kms-private-key-permissions,codex/fix-kms-repeated-onboarding +f9a9e560b fix(kms): import fs-err Unix extensions RETAINED codex/feat-kms-historical-root-keys,codex/fix-kms-ca-restart,codex/fix-kms-finish-response-order,codex/fix-kms-private-key-permissions,codex/fix-kms-repeated-onboarding +401c04b32 fix(guest-agent): report effective quote prefix RETAINED codex/fix-guest-agent-quote-prefix +cd64c1f1d fix(gateway): validate WireGuard public keys EXISTING_PR #839 fix/gateway-wg-public-key-validation +5ea75edba fix(gateway): tolerate empty handshake cache RETAINED codex/fix-gateway-empty-handshakes +ff9690d54 fix(gateway): align RPC and health routes RETAINED codex/fix-gateway-rpc-health-routes +785c65f60 fix(kms): return Finish response before exit RETAINED codex/feat-kms-historical-root-keys,codex/fix-kms-ca-restart,codex/fix-kms-finish-response-order,codex/fix-kms-repeated-onboarding +4ffbed5ce fix(rpc): emit empty JSON unit responses RETAINED codex/feat-kms-historical-root-keys,codex/fix-kms-ca-restart,codex/fix-kms-finish-response-order,codex/fix-kms-repeated-onboarding +e5947dc3d fix(kms): shut down after Finish response RETAINED codex/feat-kms-historical-root-keys,codex/fix-kms-ca-restart,codex/fix-kms-finish-response-order,codex/fix-kms-repeated-onboarding +64b695557 fix(gateway): reject invalid reported ports RETAINED codex/fix-gateway-reported-ports +9504ad0b6 fix(gateway): validate ZT domain inputs RETAINED codex/fix-gateway-zt-domain-crud,codex/fix-gateway-zt-domain-inputs +5a7065d5b fix(gateway): return Exit response before shutdown RETAINED codex/fix-gateway-exit-response-order +a79efd432 fix(gateway): reject zero DNS timing values RETAINED codex/fix-gateway-dns-timings +3f36afd0d fix(kms): reject production auth mock startup RETAINED codex/fix-kms-production-auth-mock +f4bc45843 fix(verifier): emit structured oneshot results RETAINED codex/fix-verifier-oneshot-output +90cf19e8e fix(util): honor random output path RETAINED codex/fix-util-random-output-path +299586b50 chore(kms): refresh auth mock lockfile RETAINED codex/fix-kms-production-auth-mock +e4d6d4a7b fix(verifier): keep oneshot stdout machine-readable RETAINED codex/fix-verifier-oneshot-output +6237ea211 fix(util): import fs err Unix options RETAINED codex/fix-util-random-output-path +4e84015ee fix(simulator): implement Nitro PCR state RETAINED codex/fix-simulator-measured-nitro-pcrs,codex/fix-simulator-nitro-pcr-state +187a98b57 fix(simulator): compile Nitro ABI coverage RETAINED codex/fix-simulator-measured-nitro-pcrs,codex/fix-simulator-nitro-pcr-state +48ba48833 test(simulator): cover SEV-SNP device boundaries TEST_ONLY move/account in test-infrastructure PR +1938b3ba7 test(simulator): cover TDX filesystem boundaries TEST_ONLY move/account in test-infrastructure PR +920133fee fix(attestation): reject trailing legacy bytes RETAINED codex/fix-attestation-trailing-bytes +2f91b0274 fix(attestation): reject trailing msgpack bytes RETAINED codex/fix-attestation-trailing-bytes +3b01e4d47 fix(attestation): validate V2 event preimages RETAINED codex/fix-attestation-v2-event-preimages +8f135661f fix(verifier): label development trust evidence REJECTED discarded; verifier trust roots already separate development and production evidence +b6aa008bf chore(attestation): update development test lockfile TEST_ONLY move/account in test-infrastructure PR +e11edcd50 fix(verifier): confine image archive extraction RETAINED codex/fix-verifier-archive-confinement +113c9547d test(verifier): use valid cache manifest digest RETAINED codex/fix-verifier-archive-confinement +ec1908a64 chore(verifier): update archive dependency lockfile RETAINED codex/fix-verifier-archive-confinement +6d23260c1 fix(verifier): version measurement cache keys REJECTED discarded; embedded cache-entry versions already enforce compatibility +b316ea22b test(dstack-mr): cover unsupported swtpm measurement TEST_ONLY move/account in test-infrastructure PR +146cb8d15 fix(guest): write generated credentials atomically RETAINED codex/fix-guest-atomic-attestation-output,codex/fix-guest-atomic-credentials,codex/fix-guest-kms-key-permissions,codex/fix-util-atomic-tpm-quotes +5f5885938 build(guest): lock CLI filesystem dependency RETAINED codex/fix-guest-atomic-attestation-output,codex/fix-guest-atomic-credentials,codex/fix-guest-kms-key-permissions,codex/fix-util-atomic-tpm-quotes +a6ee9b550 fix(guest): fail JSON vTPM attestation errors RETAINED codex/fix-guest-vtpm-json-errors +431751f41 fix(guest): write attestation outputs atomically RETAINED codex/fix-guest-atomic-attestation-output +8b0ace8dc fix(guest): protect KMS key output RETAINED codex/fix-guest-kms-key-permissions +c5b3f9c69 fix(guest): defer failed KMS measurements RETAINED codex/fix-guest-deferred-kms-measurement +18d1c16ab fix(guest): type captured KMS measurement RETAINED codex/fix-guest-deferred-kms-measurement +475a37737 fix(guest): own captured KMS measurement RETAINED codex/fix-guest-deferred-kms-measurement +0d38f2087 fix(guest): keep LUKS keys out of argv RETAINED codex/fix-guest-luks-key-argv +12f487758 fix(guest): disable active swap before replacement REJECTED discarded; normal boot starts with no active swap from the previous boot +d39a69efe fix(guest): compare resolved swap paths safely REJECTED discarded with the active-swap replacement logic +af2166b33 fix(guest): protect gateway private state RETAINED codex/fix-guest-gateway-private-state,codex/fix-guest-gateway-refresh-failover,codex/fix-guest-kms-failover-order,codex/fix-guest-local-provider-inventory +5777880c3 test(guest): import Unix permission metadata RETAINED codex/fix-guest-gateway-private-state,codex/fix-guest-gateway-refresh-failover,codex/fix-guest-kms-failover-order,codex/fix-guest-local-provider-inventory +6f7f8cecc fix(guest): bound Host API operations RETAINED codex/fix-guest-host-api-bounds +1eedd9c89 fix(supervisor): reject untrusted client sockets REJECTED discarded; normal Unix socket and directory permissions enforce the trust boundary +dd5bd0c85 build(supervisor): lock client libc dependency REJECTED discarded with concurrent UDS auto-start +367a16db2 fix(supervisor): retain socket path for validation REJECTED discarded with client-side socket path validation +6422f2efc test(supervisor): add socket fixture dependency TEST_ONLY move/account in test-infrastructure PR +3ccfc4394 test(supervisor): expose trusted socket rejection TEST_ONLY move/account in test-infrastructure PR +f10543399 test(supervisor): secure trusted socket directory TEST_ONLY move/account in test-infrastructure PR +fa58967e9 fix(guest-agent): reject inverted certificate validity RETAINED codex/fix-guest-agent-cert-validity +d02b42fcd fix(simulator): model measured Nitro enclave PCRs RETAINED codex/fix-simulator-measured-nitro-pcrs +333270fe8 test(attestation): verify simulated trust policy TEST_ONLY move/account in test-infrastructure PR +adb26ea51 test(attestation): generate signed SNP policy mutations TEST_ONLY move/account in test-infrastructure PR +f8d5b4d46 test(attestation): add cloud TPM mutation matrix TEST_ONLY move/account in test-infrastructure PR +882953826 fix(attestation): extract TPM matrix errors explicitly TEST_ONLY move/account in test-infrastructure PR +eba48b2da test(attestation): generate Nitro document matrix TEST_ONLY move/account in test-infrastructure PR +32abcf206 feat(supervisor): expose trusted UDS auto-start REJECTED discarded; the supported deployment has one VMM owner per runtime directory +b3376f90a fix(supervisor): serialize UDS auto-start REJECTED discarded with concurrent UDS auto-start +2f0ff74be fix(supervisor): remove duplicate startup lock helper REJECTED discarded with concurrent UDS auto-start +64dc5549d fix(supervisor): restrict auto-start socket permissions REJECTED discarded with concurrent UDS auto-start +634b3c502 fix(supervisor): keep client JSON output clean RETAINED codex/fix-supervisor-client-json-output +624acbbee fix(supervisor): reply before graceful shutdown RETAINED codex/fix-supervisor-shutdown-response +b81b946de fix(simulator): publish TPM resource manager device RETAINED codex/fix-simulator-tpm-resource-manager +0b4ba7576 fix(simulator): reject occupied mountpoints RETAINED codex/fix-simulator-occupied-mountpoints +989b0a3a2 fix(simulator): await NitroTPM resource manager REVERTED net-zero historical pair +e60f1295d "Revert ""fix(simulator): await NitroTPM resource manager""" REVERTED net-zero historical pair +106675ecd fix(attest): commit event log after measurement RETAINED codex/fix-attest-event-log-order +d0df60ecc fix(util): reject oversized quote input RETAINED codex/fix-util-quote-size-limit +53f415ec7 fix(util): honor quote report sys config RETAINED codex/fix-util-quote-report-config +297c0e202 fix(cert): reject mismatched CA key RETAINED codex/fix-ra-tls-ca-key-match +e6a2577b2 fix(util): stage certificate and key together REJECTED discarded; sequential renames cannot atomically publish a two-file pair +c4c09f466 test(simulator): provision legacy EK certificate TEST_ONLY move/account in test-infrastructure PR +3c2a86b16 test(simulator): serve TPM collateral in guest TEST_ONLY move/account in test-infrastructure PR +f727de4c9 test(simulator): publish guest TPM trust root TEST_ONLY move/account in test-infrastructure PR +bf74f2db4 fix(util): write TPM quotes atomically RETAINED codex/fix-util-atomic-tpm-quotes +c6038d6ca feat(vmm-cli): separate environment encryption KMS URL RETAINED codex/fix-vmm-cli-encryption-kms-url +9018b2e5e fix(test): sign seeded simulator attestations RETAINED codex/feat-simulator-seeded-attestation,codex/fix-mock-attestation-dcap-collateral,codex/fix-ra-tls-app-extensions,codex/fix-ra-tls-security-profile,codex/fix-simulator-legacy-attestation +990ea9c01 test(simulator): import TDX evidence helpers RETAINED codex/feat-simulator-seeded-attestation,codex/fix-mock-attestation-dcap-collateral,codex/fix-ra-tls-app-extensions,codex/fix-ra-tls-security-profile,codex/fix-simulator-legacy-attestation +6f159a327 build(simulator): lock attestation dependencies RETAINED codex/feat-simulator-seeded-attestation,codex/fix-mock-attestation-dcap-collateral,codex/fix-ra-tls-app-extensions,codex/fix-ra-tls-security-profile,codex/fix-simulator-legacy-attestation +37df16725 feat(simulator): trust explicit guest attestation roots MANUAL codex/feat-simulator-seeded-attestation +024ec4816 fix(vmm): import simulator seed error macro RETAINED codex/feat-simulator-seeded-attestation,codex/fix-mock-attestation-dcap-collateral,codex/fix-ra-tls-app-extensions,codex/fix-ra-tls-security-profile,codex/fix-simulator-legacy-attestation +8415bc8f4 fix(simulator): make seeded TDX PKI deterministic RETAINED codex/feat-simulator-seeded-attestation,codex/fix-mock-attestation-dcap-collateral,codex/fix-ra-tls-app-extensions,codex/fix-ra-tls-security-profile,codex/fix-simulator-legacy-attestation +e0068492c fix(simulator): share exact TDX trust root RETAINED codex/feat-simulator-seeded-attestation,codex/fix-mock-attestation-dcap-collateral,codex/fix-ra-tls-app-extensions,codex/fix-ra-tls-security-profile,codex/fix-simulator-legacy-attestation +04efa1b27 fix(simulator): deterministically sign seeded TDX PKI RETAINED codex/feat-simulator-seeded-attestation,codex/fix-mock-attestation-dcap-collateral,codex/fix-ra-tls-app-extensions,codex/fix-ra-tls-security-profile,codex/fix-simulator-legacy-attestation +7743dc3db test(simulator): cover seeded TDX process parity RETAINED codex/feat-simulator-seeded-attestation,codex/fix-mock-attestation-dcap-collateral,codex/fix-ra-tls-app-extensions,codex/fix-ra-tls-security-profile,codex/fix-simulator-legacy-attestation +5860d0037 refactor(simulator): drop retained TDX certificate key RETAINED codex/feat-simulator-seeded-attestation,codex/fix-mock-attestation-dcap-collateral,codex/fix-ra-tls-app-extensions,codex/fix-ra-tls-security-profile,codex/fix-simulator-legacy-attestation +74dccff50 fix(os): omit nondeterministic package logs RETAINED codex/fix-os-deterministic-package-logs,codex/fix-os-guest-ssh-host-keys +b32e73d92 fix(os): generate SSH host keys per guest RETAINED codex/fix-os-guest-ssh-host-keys +6e5872a6b fix(mock-attestation): serve legacy DCAP paths RETAINED codex/fix-mock-attestation-dcap-collateral +5434007d0 fix(mock-attestation): model the PCK CA chain RETAINED codex/fix-mock-attestation-dcap-collateral +88c7e75fe fix(mock-attestation): serve the PCK issuer CRL RETAINED codex/fix-mock-attestation-dcap-collateral +dfd787a8b feat(vmm): add configuration validation command RETAINED codex/feat-vmm-config-validation +ea7df8e4f fix(vmm): enforce configured ID pool bounds RETAINED codex/fix-vmm-id-pool-bounds +f66520780 test(vmm): cover ID pool concurrency and reconstruction RETAINED codex/fix-vmm-id-pool-bounds +57e5a947e fix(vmm): confine image artifacts to image root RETAINED codex/fix-vmm-image-artifact-confinement +1c3484071 test(vmm): cover image metadata trust boundaries RETAINED codex/fix-vmm-image-artifact-confinement +82e577885 "Revert ""test(vmm): cover image metadata trust boundaries""" RETAINED codex/fix-vmm-image-artifact-confinement +4b46ac606 test(vmm): cover image metadata trust boundaries RETAINED codex/fix-vmm-image-artifact-confinement +aab82dcb1 fix(vmm): retain image boundary diagnostics RETAINED codex/fix-vmm-image-artifact-confinement +f90878de7 fix(vmm): enforce image artifact confinement RETAINED codex/fix-vmm-image-artifact-confinement +ccc6c8108 test(vmm): cover MR configuration derivation matrix TEST_ONLY move/account in test-infrastructure PR +c4f3c28e6 test(vmm): cover VM info projection boundaries TEST_ONLY move/account in test-infrastructure PR +8461ee2a9 fix(testing): construct complete networking fixture TEST_ONLY move/account in test-infrastructure PR +2524bc484 fix(vmm): publish host-share disks atomically RETAINED codex/fix-vmm-atomic-host-share-disk +24016d336 test(vmm): cover host-share disk boundaries RETAINED codex/fix-vmm-atomic-host-share-disk +b04025fcb fix(vmm): reborrow temporary host-share image RETAINED codex/fix-vmm-atomic-host-share-disk +29665021e fix(testing): retain owned image borrows RETAINED codex/fix-vmm-atomic-host-share-disk +f4933688b test(vmm): cover launcher readiness and cleanup RETAINED codex/fix-vmm-one-shot-failures +fe88d2c29 fix(vmm): return one-shot launch failures RETAINED codex/fix-vmm-one-shot-failures +39c278192 fix(vmm): declare host-share tempfile dependency RETAINED codex/fix-vmm-atomic-host-share-disk +c6a4f0ef2 test(vmm): cover TDX variant compatibility recovery TEST_ONLY move/account in test-infrastructure PR +9aa3d4a17 test(vmm): cover verity volume validation and launch TEST_ONLY move/account in test-infrastructure PR +22810a97c test(vmm): exercise lease-owned network lifecycle TEST_ONLY move/account in test-infrastructure PR +4fd1c97cd test(vmm): correct prefixed network MAC TEST_ONLY move/account in test-infrastructure PR +c4be6776a test(vmm): restore custom network fixture state TEST_ONLY move/account in test-infrastructure PR +cb23f6898 fix(vmm): verify registry layer integrity RETAINED codex/fix-vmm-registry-layer-integrity +c767fbc15 fix(vmm): confine registry pull tags RETAINED codex/fix-vmm-registry-layer-integrity +fc83dec8d fix(vmm): reject skipped registry archive entries RETAINED codex/fix-vmm-registry-layer-integrity +25178c1d9 test(vmm): cover QEMU platform command matrix TEST_ONLY move/account in test-infrastructure PR +8481ae09d test(vmm): verify platform command stability TEST_ONLY move/account in test-infrastructure PR +dff2ce167 test(vmm): fix platform matrix literals TEST_ONLY move/account in test-infrastructure PR +2375e01b9 fix(vmm): bound automatic restart retries MANUAL codex/fix-vmm-restart-policy +f7002375a fix(vmm): validate automatic restart timing RETAINED codex/fix-vmm-restart-policy,codex/fix-vmm-serial-log-cap +d67e4710a test(vmm): exercise automatic restart policy matrix RETAINED codex/fix-vmm-restart-policy,codex/fix-vmm-serial-log-cap +2fb012215 fix(vmm): confine console log requests RETAINED codex/fix-vmm-console-log-confinement +8feb8bda3 test(vmm): cover serial rotation boundaries RETAINED codex/fix-vmm-serial-log-cap +0aaef6fdf test(vmm): read serial default through config RETAINED codex/fix-vmm-serial-log-cap +7bbee3ee5 fix(vmm): retain serial boot delimiter at cap RETAINED codex/fix-vmm-serial-log-cap +6115071d5 test(attestation): reject mutated simulator evidence TEST_ONLY move/account in test-infrastructure PR +27511d608 fix(test): isolate GCP simulator collateral port TEST_ONLY move/account in test-infrastructure PR +805cc28aa test(kms): generate key-bound attested CSR TEST_ONLY move/account in test-infrastructure PR +e04a67bbe fix(test): request CSR evidence from fixture agent TEST_ONLY move/account in test-infrastructure PR +e5c186129 chore(deps): record SignCert fixture dependencies TEST_ONLY move/account in test-infrastructure PR +df0d9c59e fix(kms): reject repeated onboarding RETAINED codex/feat-kms-historical-root-keys,codex/fix-kms-ca-restart,codex/fix-kms-repeated-onboarding +61951fc7c fix(guest): escape Prometheus label values RETAINED codex/fix-guest-prometheus-labels +d27991512 fix(gateway): apply CAA records to all domains RETAINED codex/fix-gateway-caa-reconciliation +47fac084f fix(gateway): serialize CAA reconciliation RETAINED codex/fix-gateway-caa-reconciliation +58dae006b fix(gateway): encrypt persisted DNS credentials REJECTED discarded; Gateway storage is already inside the CVM trust boundary and admin-token-derived encryption creates unsafe key coupling +4b1162e5e fix(gateway): validate DNS credential inputs REJECTED discarded; the additional constraints do not justify changing existing credential semantics +f7851c483 build(gateway): lock credential encryption dependency REJECTED discarded with DNS credential envelope encryption +22c6e32b9 fix(gateway): normalize ZT domain CRUD keys RETAINED codex/fix-gateway-zt-domain-crud +1c76ea0c9 fix(certbot): preserve unrelated CAA records RETAINED codex/fix-certbot-caa-preservation +2b2040abd fix(gateway): reject mismatched certificate keys RETAINED codex/feat-gateway-exact-sni-certificates,codex/fix-gateway-cert-key-match,codex/fix-gateway-corrupt-acme-credentials,codex/fix-gateway-expired-cert-reload +8448933bd test(gateway): retain cert on mismatched hot reload RETAINED codex/feat-gateway-exact-sni-certificates,codex/fix-gateway-cert-key-match,codex/fix-gateway-corrupt-acme-credentials,codex/fix-gateway-expired-cert-reload +eaad78d3e fix(gateway): reject expired certificate reloads RETAINED codex/feat-gateway-exact-sni-certificates,codex/fix-gateway-corrupt-acme-credentials,codex/fix-gateway-expired-cert-reload +5ef18a3bf test(gateway): retain cert on expired hot reload RETAINED codex/feat-gateway-exact-sni-certificates,codex/fix-gateway-corrupt-acme-credentials,codex/fix-gateway-expired-cert-reload +988195605 feat(gateway): support exact SNI certificates RETAINED codex/feat-gateway-exact-sni-certificates,codex/fix-gateway-corrupt-acme-credentials +a4d174238 test(gateway): cover SNI precedence and atomic reload RETAINED codex/feat-gateway-exact-sni-certificates,codex/fix-gateway-corrupt-acme-credentials +525053131 test(gateway): retain cert on corrupt hot reload RETAINED codex/feat-gateway-exact-sni-certificates,codex/fix-gateway-corrupt-acme-credentials +cba333e8c fix(gateway): fail closed on corrupt ACME credentials RETAINED codex/fix-gateway-corrupt-acme-credentials +80f375c67 test(gateway): cover corrupt ACME credential handling RETAINED codex/fix-gateway-corrupt-acme-credentials +d5da61a66 refactor(gateway): isolate legacy port-policy parsing RETAINED codex/refactor-gateway-port-policy +f4556c626 test(gateway): cover legacy port-policy compatibility RETAINED codex/refactor-gateway-port-policy +f970d0fbf test(gateway): make policy fetch failures diagnosable RETAINED codex/refactor-gateway-port-policy +0638e0123 fix(gateway): reject registration identity collisions RETAINED codex/fix-gateway-registration-collisions +c495f192b fix(gateway): validate peer synchronization URLs RETAINED codex/fix-gateway-app-info-peer-identity,codex/fix-gateway-peer-sync-urls +a3bd7c2f2 fix(gateway): accept app-info peer identities RETAINED codex/fix-gateway-app-info-peer-identity +2d7db7d2f fix(gateway): decode app-info peer identity RETAINED codex/fix-gateway-app-info-peer-identity +e80abbbfb test(gateway): cover dashboard model invariants TEST_ONLY move/account in test-infrastructure PR +4459de80b fix(test): match Rinja HTML escaping TEST_ONLY move/account in test-infrastructure PR +a2c781b96 test(gateway): exercise concurrent counter guards TEST_ONLY move/account in test-infrastructure PR +763eb4e14 test(gateway): expose port policy decision matrix TEST_ONLY move/account in test-infrastructure PR +4ecf7b772 test(gateway): cover local TLS stream boundaries TEST_ONLY move/account in test-infrastructure PR +2f66441f5 test(gateway): make RPC route separation explicit REVERTED net-zero historical pair +4b588c521 fix(gateway): retain Rocket main entrypoint REVERTED net-zero historical pair +3c770d0bb fix(test): compare RPC route paths REVERTED net-zero historical pair +50f4bc1a0 "Revert ""fix(test): compare RPC route paths""" REVERTED net-zero historical pair +4cac05854 "Revert ""fix(gateway): retain Rocket main entrypoint""" REVERTED net-zero historical pair +dfe49ef18 "Revert ""test(gateway): make RPC route separation explicit""" REVERTED net-zero historical pair +aa66f6713 fix(gateway): publish debug keys safely RETAINED codex/fix-gateway-debug-key-publication +83df122a6 fix(test): consume debug key workers RETAINED codex/fix-gateway-debug-key-publication +d40f42ede fix(gateway): write TLS material privately MANUAL codex/fix-gateway-private-tls-material +a9766ad42 fix(gateway): remove stale proxy writer import RETAINED codex/fix-gateway-private-tls-material +70a71cba4 test(gateway): exercise bounded SNI host failover TEST_ONLY move/account in test-infrastructure PR +86fa05e24 fix(test): import Gateway proxy address fixture TEST_ONLY move/account in test-infrastructure PR +dec56be5b feat(gateway): configure app-address DNS server RETAINED codex/feat-gateway-app-address-dns +53d34f037 fix(gateway): use public DNS runtime provider RETAINED codex/feat-gateway-app-address-dns +4abb56304 fix(gateway): maintain healthy Top-N cache RETAINED codex/fix-gateway-top-n-cache +8f974936a fix(gateway): support workspace Rust edition RETAINED codex/fix-gateway-top-n-cache +0b751ccb8 test(gateway): cover Top-N cache lifecycle TEST_ONLY move/account in test-infrastructure PR +19257cda3 test(gateway): cover WaveKV lifecycle matrix TEST_ONLY move/account in test-infrastructure PR +4617eff90 test(certbot): enable Cloudflare client tests TEST_ONLY move/account in test-infrastructure PR +cb7ac1f57 test(certbot): exercise workdir lifecycle TEST_ONLY move/account in test-infrastructure PR +f201cc4ac fix(certbot): pace daemon and run once hook RETAINED codex/fix-certbot-daemon-lifecycle +e1c65f7df fix(certbot): stop daemon gracefully RETAINED codex/fix-certbot-daemon-lifecycle +4e3b06424 feat(certbot): configure DNS API endpoint RETAINED codex/feat-certbot-dns-api-endpoint +1e1c497b6 fix(certbot): default DNS endpoint settings RETAINED codex/feat-certbot-dns-api-endpoint +7b0700aa3 fix(mkosi): enable memory cgroup controller RETAINED codex/fix-mkosi-memory-cgroup +088bdc1b4 fix(simulator): preserve legacy certificate attestation RETAINED codex/fix-simulator-legacy-attestation +6f7eec0ac fix(simulator): preserve legacy attest responses RETAINED codex/fix-simulator-legacy-attestation +7188be852 feat(kms): support unquoted compatibility RPC certificates SUPERSEDED #830 upstream implementation +f6112e858 fix(ra-rpc): accept empty JSON unit responses RETAINED codex/fix-ra-rpc-empty-json +4f13d087e fix(http-client): accept empty JSON unit responses RETAINED codex/fix-http-client-empty-json +53161a902 fix(os): enforce artifact manifest schema RETAINED codex/fix-os-artifact-manifest-schema +6b9b20afd test(verifier): cover image download security matrix TEST_ONLY move/account in test-infrastructure PR +abfb8a33a style(verifier): format image download matrix TEST_ONLY move/account in test-infrastructure PR +df83ed979 fix(verifier): import string formatting trait in test TEST_ONLY move/account in test-infrastructure PR +c2eb2a627 fix(verifier): configure image download matrix correctly TEST_ONLY move/account in test-infrastructure PR +c6f02bdea fix(test): isolate malicious image destination TEST_ONLY move/account in test-infrastructure PR +f5b0a1d60 test(attestation): cover TDX collateral and TCB matrix TEST_ONLY move/account in test-infrastructure PR +28a28530c test(attestation): cover TDX V2 event log matrix TEST_ONLY move/account in test-infrastructure PR +cca794807 fix(test): compile TDX V2 event log matrix TEST_ONLY move/account in test-infrastructure PR +0e2e8212c refactor(verifier): make image strategies exhaustive RETAINED codex/refactor-verifier-image-strategies,codex/refactor-verifier-tcb-policy +f71d959fa test(verifier): cover six-platform image strategies RETAINED codex/refactor-verifier-image-strategies,codex/refactor-verifier-tcb-policy +38623631d test(verifier): cover GCP and Nitro image bindings RETAINED codex/refactor-verifier-image-strategies,codex/refactor-verifier-tcb-policy +2c3ae5452 test(measurement): cover ACPI and swtpm policy matrix TEST_ONLY move/account in test-infrastructure PR +68828bf5d fix(test): inspect ACPI version policy errors TEST_ONLY move/account in test-infrastructure PR +bf98ad09e fix(test): inspect nested QEMU version error TEST_ONLY move/account in test-infrastructure PR +e29289da0 test(verifier): accept matching swtpm lite evidence TEST_ONLY move/account in test-infrastructure PR +cfd829c45 fix(ra-tls): validate certificate security profile RETAINED codex/fix-ra-tls-app-extensions,codex/fix-ra-tls-security-profile +800f1f251 fix(test): satisfy RA certificate SAN profile RETAINED codex/fix-ra-tls-app-extensions,codex/fix-ra-tls-security-profile +2a030801a fix(ra-tls): bind certificate app extensions RETAINED codex/fix-ra-tls-app-extensions +7a398d07e test(ra-tls): cover certificate mutation matrix RETAINED codex/fix-ra-tls-app-extensions +ddc4b10eb test(ra-tls): complete certificate mutation coverage RETAINED codex/fix-ra-tls-app-extensions +9780f90b5 fix(test): mutate legacy-compatible quote corpus RETAINED codex/fix-ra-tls-app-extensions +cf3328530 fix(test): keep RA mutation corpus on V0 wire format RETAINED codex/fix-ra-tls-app-extensions +a19c42dd6 refactor(verifier): make TCB policy sources exhaustive RETAINED codex/refactor-verifier-tcb-policy +9eb1794db fix(measurement): accept historical image versions RETAINED codex/fix-measurement-historical-images +9a71c55db fix(verifier): validate configuration precedence MANUAL codex/fix-verifier-config-precedence +ba5b88557 fix(verifier): escape image template diagnostic RETAINED codex/fix-verifier-config-precedence,codex/fix-verifier-service-config +da9062e8a fix(verifier): separate service and Rocket config RETAINED codex/fix-verifier-service-config +d9fbe7dba fix(test): use valid unsupported image URL RETAINED codex/fix-verifier-service-config +6eaaa672b test(kms): cover application key hierarchy signatures TEST_ONLY move/account in test-infrastructure PR +bea8fe244 fix(kms): preserve CA certificates across restart MANUAL codex/fix-kms-ca-restart +25ad5523d feat(kms): return configured historical root keys RETAINED codex/feat-kms-historical-root-keys +caa1785d5 test(kms): cover historical root key inventory RETAINED codex/feat-kms-historical-root-keys +2f428fb88 docs(kms): define cold backup recovery procedure RETAINED codex/feat-kms-historical-root-keys +d3b7b5c09 test(kms): generate legacy and current CSR fixtures TEST_ONLY move/account in test-infrastructure PR +c90b1ef0b fix(test): encode legacy KMS CSR fixture TEST_ONLY move/account in test-infrastructure PR +994c5c2c6 fix(build): synchronize mock attestation lock entry TEST_ONLY move/account in test-infrastructure PR +3013ebad7 fix(test): use CSR canonical encoding API TEST_ONLY move/account in test-infrastructure PR +0881c3818 fix(test): retain attestation for legacy CSR TEST_ONLY move/account in test-infrastructure PR +81c019a42 fix(kms): synchronize authorization Bun locks RETAINED codex/feat-kms-auth-policy-audit,codex/feat-kms-finalized-ethereum-auth,codex/fix-kms-auth-boot-schema,codex/fix-kms-auth-endpoint-redaction,codex/fix-kms-auth-locks,codex/fix-kms-node-auth-safety +9eb3a351c fix(kms): bound authorization boot schema RETAINED codex/feat-kms-auth-policy-audit,codex/feat-kms-finalized-ethereum-auth,codex/fix-kms-auth-boot-schema,codex/fix-kms-auth-endpoint-redaction,codex/fix-kms-node-auth-safety +4b651e2ad fix(kms): preserve unprefixed auth measurements RETAINED codex/feat-kms-auth-policy-audit,codex/feat-kms-finalized-ethereum-auth,codex/fix-kms-auth-boot-schema,codex/fix-kms-auth-endpoint-redaction,codex/fix-kms-node-auth-safety +06e7c0b1c fix(kms): redact authorization backend endpoint RETAINED codex/feat-kms-auth-policy-audit,codex/feat-kms-finalized-ethereum-auth,codex/fix-kms-auth-endpoint-redaction,codex/fix-kms-node-auth-safety +e36e38c66 fix(kms): align Node authorization safety RETAINED codex/feat-kms-auth-policy-audit,codex/feat-kms-finalized-ethereum-auth,codex/fix-kms-node-auth-safety +c1f33e683 fix(test): isolate Node authorization mocks RETAINED codex/feat-kms-auth-policy-audit,codex/feat-kms-finalized-ethereum-auth,codex/fix-kms-node-auth-safety +fed5f4570 fix(kms): align Node authorization build entrypoint RETAINED codex/feat-kms-auth-policy-audit,codex/feat-kms-finalized-ethereum-auth,codex/fix-kms-node-auth-safety +409240b7b fix(kms): align authorization validation errors RETAINED codex/feat-kms-auth-policy-audit,codex/feat-kms-finalized-ethereum-auth,codex/fix-kms-node-auth-safety +b84a275d5 fix(kms): narrow authorization validation errors RETAINED codex/feat-kms-auth-policy-audit,codex/feat-kms-finalized-ethereum-auth,codex/fix-kms-node-auth-safety +0c95f4a5e fix(kms): resolve test upgrade artifacts RETAINED codex/feat-kms-auth-policy-audit,codex/feat-kms-finalized-ethereum-auth,codex/fix-kms-node-auth-safety +7618996cb fix(kms): align authorization container runtime RETAINED codex/feat-kms-auth-policy-audit,codex/feat-kms-finalized-ethereum-auth,codex/fix-kms-node-auth-safety +4f6a4b84f fix(kms): make certificate log writes atomic RETAINED codex/fix-kms-certificate-logs +b8cb53add fix(kms): enable certificate log module RETAINED codex/fix-kms-certificate-logs +a3362c3ea fix(kms): parse certificate log names correctly RETAINED codex/fix-kms-certificate-logs +264a46a3d fix(kms): commit certificate logs atomically RETAINED codex/fix-kms-certificate-logs +553a1f19e chore(kms): remove stale certificate log import RETAINED codex/fix-kms-certificate-logs +b21e12ead feat(kms): expose startup health endpoint MANUAL codex/feat-kms-startup-health +3cda53791 feat(dstack-mr): restore measurement diagnosis RETAINED codex/feat-dstack-mr-diagnosis +ab4670f84 fix(dstack-mr): align diagnosis with current OVMF RETAINED codex/feat-dstack-mr-diagnosis +3326cb87f feat(dstack-mr): locate divergent RTMR events RETAINED codex/feat-dstack-mr-diagnosis +c6b39faac test(verifier): cover cache upgrade boundaries TEST_ONLY move/account in test-infrastructure PR +2e2bf783a test(verifier): compare serialized cache measurements TEST_ONLY move/account in test-infrastructure PR +d72963927 test(kms): cover Ethereum authorization freshness RETAINED codex/feat-kms-auth-policy-audit,codex/feat-kms-finalized-ethereum-auth +d2dd5d868 test(kms): define Ethereum finalized snapshots RETAINED codex/feat-kms-auth-policy-audit,codex/feat-kms-finalized-ethereum-auth +f3dabbe00 feat(kms): authorize from finalized Ethereum snapshots RETAINED codex/feat-kms-auth-policy-audit,codex/feat-kms-finalized-ethereum-auth +f548c2d41 fix(kms): assert redacted backend diagnostics RETAINED codex/feat-kms-auth-policy-audit,codex/feat-kms-finalized-ethereum-auth +f9f2d7937 feat(kms): emit authorization policy audit events RETAINED codex/feat-kms-auth-policy-audit +c839596e6 test(kms): reconstruct policy from audit events RETAINED codex/feat-kms-auth-policy-audit +f85469609 test(kms): prove authorization decisions are uncached RETAINED codex/feat-kms-auth-policy-audit +8962ce289 fix(test): own captured authorization headers RETAINED codex/feat-kms-auth-policy-audit +392c32fef test(guest): define configuration entry matrix TEST_ONLY move/account in test-infrastructure PR +203e136c6 fix(mkosi): install Sysbox rsync dependency RETAINED codex/fix-mkosi-sysbox-rsync +639bb13d0 fix(guest): force refresh after missing handshake RETAINED codex/fix-guest-missing-handshake-refresh +7e281627e fix(guest): make KMS failover ordering testable RETAINED codex/fix-guest-gateway-refresh-failover,codex/fix-guest-kms-failover-order,codex/fix-guest-local-provider-inventory +3bb241a46 style(guest): format KMS failover tests RETAINED codex/fix-guest-gateway-refresh-failover,codex/fix-guest-kms-failover-order,codex/fix-guest-local-provider-inventory +ba5583ad6 fix(guest): isolate local key providers from KMS inventory RETAINED codex/fix-guest-gateway-refresh-failover,codex/fix-guest-local-provider-inventory +78a6786a1 style(guest): format provider route test RETAINED codex/fix-guest-gateway-refresh-failover,codex/fix-guest-local-provider-inventory +300da2a1f fix(guest): make gateway refresh failover atomic RETAINED codex/fix-guest-gateway-refresh-failover +19af570e0 style(guest): format gateway refresh tests RETAINED codex/fix-guest-gateway-refresh-failover +7a1f885a1 fix(guest): retain gateway keys across failover attempts RETAINED codex/fix-guest-gateway-refresh-failover +ce59b0ba7 style(guest): format gateway failover closure RETAINED codex/fix-guest-gateway-refresh-failover +716e7772b test(integration): exercise verifier evidence compatibility TEST_ONLY move/account in test-infrastructure PR +17731c107 fix(test): diagnose evidence compatibility failures TEST_ONLY move/account in test-infrastructure PR +c21218fb0 feat(gateway): allow static proxy domain from app config RETAINED codex/feat-gateway-static-proxy-domain +ec1d30598 fix(guest): decouple Gateway outage from app boot RETAINED codex/fix-guest-gateway-outage-boot +35f84738f fix(vmm): avoid double-reserving reloaded VM CIDs RETAINED codex/fix-vmm-reloaded-cids +43f669717 style(rust): format product fixes TEST_ONLY move/account in test-infrastructure PR +574c0d364 fix(gateway): restore upstream build compatibility MANUAL codex/fix-gateway-private-tls-material diff --git a/test-suites/audit/core-components-product-pr-inventory.tsv b/test-suites/audit/core-components-product-pr-inventory.tsv new file mode 100644 index 000000000..95400bffe --- /dev/null +++ b/test-suites/audit/core-components-product-pr-inventory.tsv @@ -0,0 +1,132 @@ +pr state title head base dependency url +842 MERGED fix(os): omit built-in FUSE module package codex/fix-os-fuse-package master https://github.com/Dstack-TEE/dstack/pull/842 +843 MERGED fix(os): fail multi-flavor builds on first error codex/fix-os-multiflavor-failure master https://github.com/Dstack-TEE/dstack/pull/843 +844 MERGED fix(simulator): support current FUSE soname codex/fix-simulator-fuse-soname master https://github.com/Dstack-TEE/dstack/pull/844 +846 OPEN fix(simulator): wait for GCP vTPM readiness codex/fix-simulator-gcp-vtpm-readiness master https://github.com/Dstack-TEE/dstack/pull/846 +847 OPEN fix(os): install TPM device TCTI for simulator codex/fix-os-simulator-tpm-tcti master https://github.com/Dstack-TEE/dstack/pull/847 +848 OPEN fix(simulator): expose GCP TPM event log codex/fix-simulator-gcp-event-log master https://github.com/Dstack-TEE/dstack/pull/848 +849 OPEN fix(simulator): report live NitroTPM PCR capabilities codex/fix-simulator-nitrotpm-pcrs master https://github.com/Dstack-TEE/dstack/pull/849 +850 OPEN fix(vmm): generate simulated SEV-SNP mr_config codex/fix-vmm-simulated-snp-mr-config master https://github.com/Dstack-TEE/dstack/pull/850 +851 OPEN fix(vmm): pass cloud image measurements to guests codex/fix-vmm-cloud-image-measurements master https://github.com/Dstack-TEE/dstack/pull/851 +852 OPEN fix(simulator): retry interrupted vTPM reads codex/fix-simulator-interrupted-vtpm-read master https://github.com/Dstack-TEE/dstack/pull/852 +853 OPEN fix(simulator): stabilize NSM device readiness codex/fix-simulator-nsm-readiness master https://github.com/Dstack-TEE/dstack/pull/853 +854 OPEN fix(simulator): identify simulated Nitro Enclave guests codex/fix-nitro-enclave-platform-detection master https://github.com/Dstack-TEE/dstack/pull/854 +855 OPEN fix(guest): report ZFS pool creation errors codex/fix-guest-zfs-pool-errors master https://github.com/Dstack-TEE/dstack/pull/855 +856 OPEN fix(mkosi): install and stage the volume helper codex/fix-mkosi-volume-helper master https://github.com/Dstack-TEE/dstack/pull/856 +857 OPEN fix(mkosi): enable Docker IPv4 NAT modules codex/fix-mkosi-docker-ipv4-nat master https://github.com/Dstack-TEE/dstack/pull/857 +858 OPEN fix(vmm): reap VM launcher children on SvStop codex/fix-vmm-svstop-child-reaping master https://github.com/Dstack-TEE/dstack/pull/858 +859 OPEN fix(vmm): resize uninitialized stopped VM manifests codex/fix-vmm-stopped-manifest-resize master https://github.com/Dstack-TEE/dstack/pull/859 +860 OPEN [STACKED on #855] fix(guest): disable the ZFS pool root mountpoint codex/fix-guest-zfs-root-mountpoint codex/fix-guest-zfs-pool-errors #855 https://github.com/Dstack-TEE/dstack/pull/860 +861 OPEN fix(vmm): keep Host API on the private vsock listener codex/fix-vmm-private-host-api master https://github.com/Dstack-TEE/dstack/pull/861 +862 OPEN fix(vmm): reject conflicting host port mappings codex/fix-vmm-host-port-conflicts master https://github.com/Dstack-TEE/dstack/pull/862 +863 OPEN fix(kms): validate onboarding domains codex/fix-kms-onboarding-domains master https://github.com/Dstack-TEE/dstack/pull/863 +864 OPEN fix(guest-agent): report the effective quote prefix codex/fix-guest-agent-quote-prefix master https://github.com/Dstack-TEE/dstack/pull/864 +865 OPEN fix(gateway): tolerate an empty handshake cache codex/fix-gateway-empty-handshakes master https://github.com/Dstack-TEE/dstack/pull/865 +866 OPEN fix(gateway): align RPC and health routes codex/fix-gateway-rpc-health-routes master https://github.com/Dstack-TEE/dstack/pull/866 +867 OPEN fix(gateway): reject invalid reported ports codex/fix-gateway-reported-ports master https://github.com/Dstack-TEE/dstack/pull/867 +868 OPEN fix(gateway): validate ZT domain inputs codex/fix-gateway-zt-domain-inputs master https://github.com/Dstack-TEE/dstack/pull/868 +869 OPEN fix(gateway): return Exit response before shutdown codex/fix-gateway-exit-response-order master https://github.com/Dstack-TEE/dstack/pull/869 +870 OPEN fix(gateway): reject zero DNS timing values codex/fix-gateway-dns-timings master https://github.com/Dstack-TEE/dstack/pull/870 +871 OPEN fix(kms): reject production auth mock startup codex/fix-kms-production-auth-mock master https://github.com/Dstack-TEE/dstack/pull/871 +872 OPEN [STACKED on #859] fix(vmm): reject empty and zero ResizeVm updates codex/fix-vmm-resize-validation codex/fix-vmm-stopped-manifest-resize #859 https://github.com/Dstack-TEE/dstack/pull/872 +873 OPEN [STACKED on #863] fix(kms): make Bootstrap one-time codex/fix-kms-bootstrap-once codex/fix-kms-onboarding-domains #863 https://github.com/Dstack-TEE/dstack/pull/873 +874 OPEN [STACKED on #873] fix(kms): persist private keys owner-only codex/fix-kms-private-key-permissions codex/fix-kms-bootstrap-once #873 https://github.com/Dstack-TEE/dstack/pull/874 +875 OPEN [STACKED on #874] fix(kms): return Finish response before shutdown codex/fix-kms-finish-response-order codex/fix-kms-private-key-permissions #874 https://github.com/Dstack-TEE/dstack/pull/875 +876 OPEN fix(verifier): keep oneshot output machine-readable codex/fix-verifier-oneshot-output master https://github.com/Dstack-TEE/dstack/pull/876 +877 OPEN fix(util): honor the requested random output path codex/fix-util-random-output-path master https://github.com/Dstack-TEE/dstack/pull/877 +878 OPEN fix(attestation): reject trailing encoded bytes codex/fix-attestation-trailing-bytes master https://github.com/Dstack-TEE/dstack/pull/878 +879 OPEN fix(attestation): validate V2 event preimages codex/fix-attestation-v2-event-preimages master https://github.com/Dstack-TEE/dstack/pull/879 +880 OPEN test(attestation): verify simulator trust-root isolation codex/fix-verifier-development-trust-label master https://github.com/Dstack-TEE/dstack/pull/880 +881 OPEN fix(verifier): confine image archive extraction codex/fix-verifier-archive-confinement master https://github.com/Dstack-TEE/dstack/pull/881 +882 OPEN test(verifier): cover measurement cache compatibility codex/fix-verifier-cache-key-version master https://github.com/Dstack-TEE/dstack/pull/882 +883 OPEN fix(guest): write generated credentials atomically codex/fix-guest-atomic-credentials master https://github.com/Dstack-TEE/dstack/pull/883 +884 OPEN fix(guest): fail malformed JSON vTPM attestations codex/fix-guest-vtpm-json-errors master https://github.com/Dstack-TEE/dstack/pull/884 +885 OPEN fix(guest): defer failed KMS measurements codex/fix-guest-deferred-kms-measurement master https://github.com/Dstack-TEE/dstack/pull/885 +886 OPEN fix(guest): keep LUKS keys out of process arguments codex/fix-guest-luks-key-argv master https://github.com/Dstack-TEE/dstack/pull/886 +887 CLOSED fix(guest): disable active swap before replacement codex/fix-guest-swap-replacement master https://github.com/Dstack-TEE/dstack/pull/887 +888 OPEN [STACKED on #853] fix(simulator): implement Nitro PCR state codex/fix-simulator-nitro-pcr-state codex/fix-simulator-nsm-readiness #853 https://github.com/Dstack-TEE/dstack/pull/888 +889 OPEN [STACKED on #883] fix(guest): write attestation outputs atomically codex/fix-guest-atomic-attestation-output codex/fix-guest-atomic-credentials #883 https://github.com/Dstack-TEE/dstack/pull/889 +890 OPEN [STACKED on #883] fix(guest): protect KMS key output codex/fix-guest-kms-key-permissions codex/fix-guest-atomic-credentials #883 https://github.com/Dstack-TEE/dstack/pull/890 +891 OPEN fix(guest): protect gateway private state codex/fix-guest-gateway-private-state master https://github.com/Dstack-TEE/dstack/pull/891 +892 OPEN fix(guest): bound Host API operations codex/fix-guest-host-api-bounds master https://github.com/Dstack-TEE/dstack/pull/892 +893 CLOSED fix(supervisor): reject untrusted client sockets codex/fix-supervisor-trusted-client-sockets master https://github.com/Dstack-TEE/dstack/pull/893 +894 OPEN fix(guest-agent): reject inverted certificate validity codex/fix-guest-agent-cert-validity master https://github.com/Dstack-TEE/dstack/pull/894 +895 OPEN fix(attest): commit event log after measurement codex/fix-attest-event-log-order master https://github.com/Dstack-TEE/dstack/pull/895 +896 OPEN fix(util): reject oversized quote input codex/fix-util-quote-size-limit master https://github.com/Dstack-TEE/dstack/pull/896 +897 OPEN fix(util): honor quote report system configuration codex/fix-util-quote-report-config master https://github.com/Dstack-TEE/dstack/pull/897 +898 OPEN fix(cert): reject mismatched CA keys codex/fix-ra-tls-ca-key-match master https://github.com/Dstack-TEE/dstack/pull/898 +899 OPEN feat(vmm-cli): separate environment encryption KMS URL codex/fix-vmm-cli-encryption-kms-url master https://github.com/Dstack-TEE/dstack/pull/899 +900 OPEN fix(os): omit nondeterministic package logs codex/fix-os-deterministic-package-logs master https://github.com/Dstack-TEE/dstack/pull/900 +901 OPEN fix(simulator): reject occupied mountpoints codex/fix-simulator-occupied-mountpoints master https://github.com/Dstack-TEE/dstack/pull/901 +902 CLOSED fix(util): stage certificate and key together codex/fix-util-atomic-cert-key master https://github.com/Dstack-TEE/dstack/pull/902 +903 OPEN [STACKED on #900] fix(os): generate SSH host keys per guest codex/fix-os-guest-ssh-host-keys codex/fix-os-deterministic-package-logs #900 https://github.com/Dstack-TEE/dstack/pull/903 +904 OPEN [STACKED on #888] fix(simulator): model measured Nitro enclave PCRs codex/fix-simulator-measured-nitro-pcrs codex/fix-simulator-nitro-pcr-state #888 https://github.com/Dstack-TEE/dstack/pull/904 +905 CLOSED feat(supervisor): expose concurrent UDS auto-start codex/feat-supervisor-trusted-uds-autostart master https://github.com/Dstack-TEE/dstack/pull/905 +906 OPEN feat(vmm): add a configuration validation command codex/feat-vmm-config-validation master https://github.com/Dstack-TEE/dstack/pull/906 +907 OPEN fix(vmm): enforce configured ID pool bounds codex/fix-vmm-id-pool-bounds master https://github.com/Dstack-TEE/dstack/pull/907 +908 OPEN fix(vmm): publish host-share disks atomically codex/fix-vmm-atomic-host-share-disk master https://github.com/Dstack-TEE/dstack/pull/908 +909 OPEN fix(vmm): return one-shot launch failures codex/fix-vmm-one-shot-failures master https://github.com/Dstack-TEE/dstack/pull/909 +910 OPEN fix(vmm): verify registry layer integrity codex/fix-vmm-registry-layer-integrity master https://github.com/Dstack-TEE/dstack/pull/910 +911 OPEN fix(vmm): confine console log requests codex/fix-vmm-console-log-confinement master https://github.com/Dstack-TEE/dstack/pull/911 +912 OPEN [STACKED on #851] fix(vmm): confine image artifacts to the image root codex/fix-vmm-image-artifact-confinement codex/fix-vmm-cloud-image-measurements #851 https://github.com/Dstack-TEE/dstack/pull/912 +913 OPEN fix(guest): escape Prometheus label values codex/fix-guest-prometheus-labels master https://github.com/Dstack-TEE/dstack/pull/913 +914 OPEN fix(gateway): reconcile CAA records for every domain codex/fix-gateway-caa-reconciliation master https://github.com/Dstack-TEE/dstack/pull/914 +915 CLOSED fix(gateway): encrypt and validate persisted DNS credentials codex/fix-gateway-dns-credential-storage master https://github.com/Dstack-TEE/dstack/pull/915 +916 OPEN fix(certbot): preserve unrelated CAA records codex/fix-certbot-caa-preservation master https://github.com/Dstack-TEE/dstack/pull/916 +917 OPEN fix(gateway): reject mismatched certificate keys codex/fix-gateway-cert-key-match master https://github.com/Dstack-TEE/dstack/pull/917 +918 OPEN refactor(gateway): isolate legacy port-policy parsing codex/refactor-gateway-port-policy master https://github.com/Dstack-TEE/dstack/pull/918 +919 OPEN fix(gateway): reject registration identity collisions codex/fix-gateway-registration-collisions master https://github.com/Dstack-TEE/dstack/pull/919 +920 OPEN fix(gateway): publish debug keys safely codex/fix-gateway-debug-key-publication master https://github.com/Dstack-TEE/dstack/pull/920 +921 OPEN fix(gateway): write TLS material privately codex/fix-gateway-private-tls-material master https://github.com/Dstack-TEE/dstack/pull/921 +922 OPEN feat(gateway): configure app-address DNS resolution codex/feat-gateway-app-address-dns master https://github.com/Dstack-TEE/dstack/pull/922 +923 OPEN fix(gateway): maintain a healthy Top-N cache codex/fix-gateway-top-n-cache master https://github.com/Dstack-TEE/dstack/pull/923 +924 OPEN fix(certbot): pace and stop the daemon cleanly codex/fix-certbot-daemon-lifecycle master https://github.com/Dstack-TEE/dstack/pull/924 +925 OPEN feat(certbot): configure the DNS API endpoint codex/feat-certbot-dns-api-endpoint master https://github.com/Dstack-TEE/dstack/pull/925 +926 OPEN fix(mkosi): enable the memory cgroup controller codex/fix-mkosi-memory-cgroup master https://github.com/Dstack-TEE/dstack/pull/926 +927 OPEN fix(ra-rpc): accept empty JSON unit responses codex/fix-ra-rpc-empty-json master https://github.com/Dstack-TEE/dstack/pull/927 +928 OPEN fix(http-client): accept empty JSON unit responses codex/fix-http-client-empty-json master https://github.com/Dstack-TEE/dstack/pull/928 +929 OPEN fix(os): enforce the artifact manifest schema codex/fix-os-artifact-manifest-schema master https://github.com/Dstack-TEE/dstack/pull/929 +930 OPEN [STACKED on #875] fix(kms): reject repeated onboarding codex/fix-kms-repeated-onboarding codex/fix-kms-finish-response-order #875 https://github.com/Dstack-TEE/dstack/pull/930 +931 OPEN [STACKED on #917] fix(gateway): reject expired certificate reloads codex/fix-gateway-expired-cert-reload codex/fix-gateway-cert-key-match #917 https://github.com/Dstack-TEE/dstack/pull/931 +932 OPEN [STACKED on #915] fix(gateway): validate peer synchronization URLs codex/fix-gateway-peer-sync-urls codex/fix-gateway-dns-credential-storage #915 https://github.com/Dstack-TEE/dstack/pull/932 +933 OPEN [STACKED on #868] fix(gateway): normalize ZT domain CRUD keys codex/fix-gateway-zt-domain-crud codex/fix-gateway-zt-domain-inputs #868 https://github.com/Dstack-TEE/dstack/pull/933 +934 OPEN [STACKED on #931] feat(gateway): support exact SNI certificates codex/feat-gateway-exact-sni-certificates codex/fix-gateway-expired-cert-reload #931 https://github.com/Dstack-TEE/dstack/pull/934 +935 OPEN [STACKED on #934] fix(gateway): fail closed on corrupt ACME credentials codex/fix-gateway-corrupt-acme-credentials codex/feat-gateway-exact-sni-certificates #934 https://github.com/Dstack-TEE/dstack/pull/935 +936 OPEN [STACKED on #932] fix(gateway): accept app-info peer identities codex/fix-gateway-app-info-peer-identity codex/fix-gateway-peer-sync-urls #932 https://github.com/Dstack-TEE/dstack/pull/936 +937 OPEN refactor(verifier): make image strategies exhaustive codex/refactor-verifier-image-strategies master https://github.com/Dstack-TEE/dstack/pull/937 +938 OPEN [STACKED on #964] fix(ra-tls): validate the certificate security profile codex/fix-ra-tls-security-profile codex/feat-simulator-seeded-attestation #964 https://github.com/Dstack-TEE/dstack/pull/938 +939 OPEN fix(measurement): accept historical image versions codex/fix-measurement-historical-images master https://github.com/Dstack-TEE/dstack/pull/939 +940 OPEN fix(verifier): validate configuration precedence codex/fix-verifier-config-precedence master https://github.com/Dstack-TEE/dstack/pull/940 +941 OPEN fix(kms): synchronize authorization Bun locks codex/fix-kms-auth-locks master https://github.com/Dstack-TEE/dstack/pull/941 +942 OPEN fix(kms): commit certificate logs atomically codex/fix-kms-certificate-logs master https://github.com/Dstack-TEE/dstack/pull/942 +943 OPEN feat(dstack-mr): restore measurement diagnosis codex/feat-dstack-mr-diagnosis master https://github.com/Dstack-TEE/dstack/pull/943 +944 OPEN fix(mkosi): install the Sysbox rsync dependency codex/fix-mkosi-sysbox-rsync master https://github.com/Dstack-TEE/dstack/pull/944 +945 OPEN fix(guest): force refresh after a missing handshake codex/fix-guest-missing-handshake-refresh master https://github.com/Dstack-TEE/dstack/pull/945 +946 OPEN [STACKED on #891] fix(guest): make KMS failover ordering deterministic codex/fix-guest-kms-failover-order codex/fix-guest-gateway-private-state #891 https://github.com/Dstack-TEE/dstack/pull/946 +947 OPEN feat(gateway): allow a static proxy domain from app config codex/feat-gateway-static-proxy-domain master https://github.com/Dstack-TEE/dstack/pull/947 +948 OPEN fix(guest): decouple Gateway outage from app boot codex/fix-guest-gateway-outage-boot master https://github.com/Dstack-TEE/dstack/pull/948 +949 OPEN fix(vmm): avoid double-reserving reloaded VM CIDs codex/fix-vmm-reloaded-cids master https://github.com/Dstack-TEE/dstack/pull/949 +950 OPEN [STACKED on #937] refactor(verifier): make TCB policy sources exhaustive codex/refactor-verifier-tcb-policy codex/refactor-verifier-image-strategies #937 https://github.com/Dstack-TEE/dstack/pull/950 +951 OPEN [STACKED on #940] fix(verifier): separate service and Rocket configuration codex/fix-verifier-service-config codex/fix-verifier-config-precedence #940 https://github.com/Dstack-TEE/dstack/pull/951 +952 OPEN [STACKED on #930] fix(kms): preserve CA certificates across restart codex/fix-kms-ca-restart codex/fix-kms-repeated-onboarding #930 https://github.com/Dstack-TEE/dstack/pull/952 +953 OPEN [STACKED on #941] fix(kms): bound the authorization boot schema codex/fix-kms-auth-boot-schema codex/fix-kms-auth-locks #941 https://github.com/Dstack-TEE/dstack/pull/953 +954 OPEN [STACKED on #946] fix(guest): isolate local key providers from KMS inventory codex/fix-guest-local-provider-inventory codex/fix-guest-kms-failover-order #946 https://github.com/Dstack-TEE/dstack/pull/954 +955 OPEN [STACKED on #952] feat(kms): return configured historical root keys codex/feat-kms-historical-root-keys codex/fix-kms-ca-restart #952 https://github.com/Dstack-TEE/dstack/pull/955 +956 OPEN [STACKED on #953] fix(kms): redact the authorization backend endpoint codex/fix-kms-auth-endpoint-redaction codex/fix-kms-auth-boot-schema #953 https://github.com/Dstack-TEE/dstack/pull/956 +957 OPEN [STACKED on #956] fix(kms): align Node authorization safety codex/fix-kms-node-auth-safety codex/fix-kms-auth-endpoint-redaction #956 https://github.com/Dstack-TEE/dstack/pull/957 +958 OPEN [STACKED on #957] feat(kms): authorize from finalized Ethereum snapshots codex/feat-kms-finalized-ethereum-auth codex/fix-kms-node-auth-safety #957 https://github.com/Dstack-TEE/dstack/pull/958 +959 OPEN [STACKED on #958] feat(kms): emit authorization policy audit events codex/feat-kms-auth-policy-audit codex/feat-kms-finalized-ethereum-auth #958 https://github.com/Dstack-TEE/dstack/pull/959 +960 OPEN [STACKED on #851] fix(vmm): align simulated NitroTPM measurement codex/fix-vmm-simulated-nitrotpm-measurement codex/fix-vmm-cloud-image-measurements #851 https://github.com/Dstack-TEE/dstack/pull/960 +961 OPEN [STACKED on #883] fix(util): write TPM quotes atomically codex/fix-util-atomic-tpm-quotes codex/fix-guest-atomic-credentials #883 https://github.com/Dstack-TEE/dstack/pull/961 +962 OPEN [STACKED on #954] fix(guest): make gateway refresh failover atomic codex/fix-guest-gateway-refresh-failover codex/fix-guest-local-provider-inventory #954 https://github.com/Dstack-TEE/dstack/pull/962 +963 OPEN feat(kms): expose a startup health endpoint codex/feat-kms-startup-health master https://github.com/Dstack-TEE/dstack/pull/963 +964 OPEN [STACKED on #880] feat(simulator): add deterministic seeded attestations codex/feat-simulator-seeded-attestation codex/fix-verifier-development-trust-label #880 https://github.com/Dstack-TEE/dstack/pull/964 +965 OPEN [STACKED on #964] fix(mock-attestation): model complete DCAP collateral codex/fix-mock-attestation-dcap-collateral codex/feat-simulator-seeded-attestation #964 https://github.com/Dstack-TEE/dstack/pull/965 +966 OPEN [STACKED on #964] fix(simulator): preserve legacy attestation responses codex/fix-simulator-legacy-attestation codex/feat-simulator-seeded-attestation #964 https://github.com/Dstack-TEE/dstack/pull/966 +967 OPEN [STACKED on #938] fix(ra-tls): bind certificate app extensions codex/fix-ra-tls-app-extensions codex/fix-ra-tls-security-profile #938 https://github.com/Dstack-TEE/dstack/pull/967 +968 OPEN fix(vmm): bound automatic restart retries codex/fix-vmm-restart-policy master https://github.com/Dstack-TEE/dstack/pull/968 +969 OPEN [STACKED on #845] fix(simulator): publish the TPM resource-manager device codex/fix-simulator-tpm-resource-manager codex/fix-simulator-udev-tpm-race #845 https://github.com/Dstack-TEE/dstack/pull/969 +970 OPEN [STACKED on #968] fix(vmm): retain the serial boot delimiter at the cap codex/fix-vmm-serial-log-cap codex/fix-vmm-restart-policy #968 https://github.com/Dstack-TEE/dstack/pull/970 +976 OPEN fix(simulator): provide TDX configfs without a kernel provider codex/fix-simulator-tdx-configfs-shadow master https://github.com/Dstack-TEE/dstack/pull/976 +994 OPEN fix(supervisor): keep client JSON output clean codex/fix-supervisor-client-json-output master https://github.com/Dstack-TEE/dstack/pull/994 +995 OPEN fix(supervisor): reply before graceful shutdown codex/fix-supervisor-shutdown-response master https://github.com/Dstack-TEE/dstack/pull/995 diff --git a/test-suites/audit/core-components-product-pr-split-audit.md b/test-suites/audit/core-components-product-pr-split-audit.md new file mode 100644 index 000000000..d95d9851d --- /dev/null +++ b/test-suites/audit/core-components-product-pr-split-audit.md @@ -0,0 +1,45 @@ +# Core component product PR split audit + +This document records the completion gates for replacing monolithic product PR #840. + +## Inventory + +The authoritative split-PR inventory is +[`core-components-product-pr-inventory.tsv`](core-components-product-pr-inventory.tsv). +It lists 131 replacement product PRs (#842 through #970 excluding rejected #845, plus #976, #994, and #995), including each head branch, GitHub base, +and explicit dependency for every stacked PR. + +The per-commit inventory is +[`core-components-product-pr-accounting.tsv`](core-components-product-pr-accounting.tsv). +It classifies all 332 historical commits from #840. + +## Mechanical gates + +The final audit applies these gates: + +1. Every replacement PR from #842 through #970 except rejected #845, plus #976, exists; the inventory records whether it is open or merged. +2. Every accounting target branch exists remotely and has an open PR. +3. Every direct PR is based on `master`. +4. Every non-`master` PR title starts with `[STACKED on #NNN]`, where `#NNN` + is the PR owning its actual GitHub base branch. +5. Every stacked PR body names the same parent and warns about merge order. +6. The declared base is an ancestor of every split head. +7. Every split product diff is non-empty, passes `git diff --check`, and excludes + `REUSE.toml`, `docs/test-plans/**`, `docs/testing/**`, and `test-suites/core-components/runner/**`. +8. Every `RETAINED` accounting row has an exact stable patch-ID match in the + delta of one of its declared target PRs. +9. Every `MANUAL` target is compiled against its declared PR base and reviewed + as an upstream-adapted preservation of the named behavior. +10. Rewritten dependency chains are compiled again after their final ancestry + changes. + +## Coverage model + +Historical commits with no intended standalone product delta are explicitly +classified as `TEST_ONLY`, `REVERTED`, `REJECTED`, `EXISTING_PR`, or `SUPERSEDED`; they are +not silently omitted. `RETAINED` and `MANUAL` rows map to the split product PRs. +The existing independent WireGuard fix is #839, and compatibility RPC behavior +is superseded by merged upstream PR #830. + +PR #841 is based directly on `master` and contains only test infrastructure, +evidence, documentation, and these accounting artifacts. diff --git a/test-suites/audit/core-components-retest-watchlist.md b/test-suites/audit/core-components-retest-watchlist.md new file mode 100644 index 000000000..c45eeca29 --- /dev/null +++ b/test-suites/audit/core-components-retest-watchlist.md @@ -0,0 +1,83 @@ +# Core component retest watchlist + +This document records review corrections and conditions that must be checked during the next core-component retest. It is a watchlist, not a replacement for the case specifications or result evidence. + +## Simulator changes requiring focused retest + +### PR #844: FUSE shared-library SONAME compatibility + +PR #844 must remain limited to the Nitro NSM CUSE loader. + +Retest requirements: + +- Verify startup when only `libfuse3.so.4` is installed. +- Verify the compatibility fallback when only `libfuse3.so.3` is installed. +- Verify startup fails with a clear dynamic-library error when neither SONAME is available. +- Confirm the PR does not change TDX configfs handling, mount behavior, Cargo dependencies, or TPM device creation. +- Record the library selected at runtime and the resulting `/dev/nsm` readiness evidence. + +### PR #976: TDX configfs without a kernel TSM provider + +PR #976 owns the TDX configfs fallback that was removed from #844. + +Retest requirements: + +- Run in a development guest where configfs is mounted but no kernel TSM provider has registered `/sys/kernel/config/tsm`. +- Confirm creation of `/sys/kernel/config/tsm/report` initially fails with `EPERM` or `EACCES` and triggers the intended tmpfs shadow path. +- Verify the simulator exposes the expected TSM report ABI at the standard path after fallback. +- Verify errors other than `EPERM` or `EACCES` remain fatal. +- Verify a custom simulator mountpoint does not trigger the configfs shadow. +- Check that shadowing `/sys/kernel/config` does not unexpectedly break another configfs consumer in the development guest. +- Confirm mount cleanup and repeated-start behavior; no stale tmpfs/FUSE mount may survive the case lease. +- Confirm `tdx.rs` contains no raw `unsafe` mount or UID/GID operation. + +### Rejected PR #845: TPM device-node race tolerance + +PR #845 was rejected and must not be included in a candidate build. The original strict behavior on `master` is intentional. + +Required invariants: + +- The selected simulator platform determines which device ABI is created. +- `dstack-gcp-tdx` creates the GCP vTPM path. +- `dstack-aws-nitro-tpm` creates the Nitro TPM path. +- Non-TPM simulator modes do not create a TPM device. +- A pre-existing `/dev/tpm0` or `/dev/tpmrm0` causes startup to fail. +- Failure of the selected mode's `mknod` operation, including `EEXIST`, remains fatal. +- The simulator must not adopt an existing node based only on path existence. +- Do not add or expect a `create_tpm_device_node` configuration field. + +## Cases to rerun + +| Case | Focus | Required observations | +|---|---|---| +| `TC-GOS-SETUP-015` | TPM command proxy and lifecycle | Platform-selected creation, strict conflict failure, PCR/quote/random operations, dependency failure, restart, and exact device/process cleanup | +| `TC-GOS-SETUP-017` | Five-platform simulator lifecycle | Correct platform-to-device mapping, GCP and Nitro device ABI readiness, failure isolation, repeated start, and cleanup | +| `TC-GOS-SETUP-022` | vTPM CLI integration | `/dev/tpm0` and `/dev/tpmrm0` usability, quote verification, fault recovery, restart behavior, and cleanup | +| `TC-GOS-SETUP-016` | Nitro NSM request ABI | `.so.4` and `.so.3` CUSE loader coverage, NSM ioctl behavior, malformed requests, and `/dev/nsm` cleanup | +| TDX simulator row in `TC-GOS-SETUP-017` | TSM filesystem fallback | No-provider configfs failure, tmpfs fallback, report generation, repeated start, and mount cleanup | + +## Environment controls + +- Use the candidate mkosi development image; do not test Yocto or mkosi build correctness as part of these cases. +- Capture the effective `TeeVariant`, simulator configuration, kernel modules, configfs mounts, and device nodes before startup. +- Record whether udev/devtmpfs is running, but do not treat it as authority to change simulator device ownership. +- Remove `/dev/tpm0`, `/dev/tpmrm0`, `/dev/nsm`, simulator FUSE mounts, tmpfs shadows, swtpm processes, and `tpm_vtpm_proxy` state during case cleanup. +- Before each TPM row, prove that no physical or stale TPM node is present. +- Do not suppress a node-creation conflict to make a lifecycle case pass; record the failure and investigate ownership/configuration instead. + +## Evidence to retain + +For every affected case, retain: + +- candidate commit and PR head; +- effective simulator platform and redacted configuration; +- relevant `/sys/class/tpm*` and `/sys/kernel/config/tsm` state; +- device major/minor values and file types; +- mount table entries before, during, and after execution; +- simulator exit status and bounded logs; +- explicit cleanup evidence; +- PASS/FAIL/BLOCKED classification with a one-sentence reason. + +## Completion gate + +The retest is not complete until all affected cases have fresh candidate evidence and no result relies on the rejected #845 tolerance behavior. Hardware-only limitations remain BLOCKED only when the case genuinely requires unavailable hardware; simulator setup, fixture, script, documentation, or product failures are not environmental blockers. diff --git a/test-suites/cases/01-guest-os/01-rpc-tappd/metadata.json b/test-suites/cases/01-guest-os/01-rpc-tappd/metadata.json new file mode 100644 index 000000000..08dbe464f --- /dev/null +++ b/test-suites/cases/01-guest-os/01-rpc-tappd/metadata.json @@ -0,0 +1,4 @@ +{ + "id": "section-guest-os-rpc-tappd", + "title": "Tappd RPC" +} diff --git a/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-001/case.md b/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-001/case.md new file mode 100644 index 000000000..beab2c918 --- /dev/null +++ b/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-001/case.md @@ -0,0 +1,72 @@ + + + +# TC-GOS-TAPPD-001: Tappd.DeriveKey + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-tappd-001](../../../../catalog/feature-audit.md#req-gos-tappd-001) +- Risks: [risk-gos-tappd-001](../../../../catalog/feature-audit.md#risk-gos-tappd-001) +- Source: `dstack/guest-agent/rpc/proto/agent_rpc.proto:15` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Tappd.DeriveKey` takes `DeriveKeyArgs` (`path: string`, `subject: string`, `alt_names: string`, `usage_ra_tls: bool`, `usage_server_auth: bool`, `usage_client_auth: bool`, `random_seed: bool`) and returns `GetTlsKeyResponse` (`key: string`, `certificate_chain: string`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Tappd.DeriveKey`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Tappd.DeriveKey` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for tappd.derivekey. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Tappd.DeriveKey` with a valid `DeriveKeyArgs` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `GetTlsKeyResponse` with every documented field and exhibits the documented `DeriveKey` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-001/metadata.json b/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-001/metadata.json new file mode 100644 index 000000000..40b465311 --- /dev/null +++ b/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-001/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-tappd-001", + "title": "Tappd.DeriveKey", + "priority": "P1", + "requirements": [ + "req-gos-tappd-001" + ], + "risks": [ + "risk-gos-tappd-001" + ], + "tags": [ + "guest", + "tappd-rpc" + ], + "fixture": { + "profile": "no-tee-dev", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "Tappd.DeriveKey" + ], + "execution": { + "entrypoint": "shared/automation/passed-rpc-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-002/case.md b/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-002/case.md new file mode 100644 index 000000000..01f331bd6 --- /dev/null +++ b/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-002/case.md @@ -0,0 +1,72 @@ + + + +# TC-GOS-TAPPD-002: Tappd.DeriveK256Key + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-tappd-002](../../../../catalog/feature-audit.md#req-gos-tappd-002) +- Risks: [risk-gos-tappd-002](../../../../catalog/feature-audit.md#risk-gos-tappd-002) +- Source: `dstack/guest-agent/rpc/proto/agent_rpc.proto:18` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Tappd.DeriveK256Key` takes `GetKeyArgs` (`path: string`, `purpose: string`, `algorithm: string`) and returns `DeriveK256KeyResponse` (`k256_key: bytes`, `k256_signature_chain: bytes`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Tappd.DeriveK256Key`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Tappd.DeriveK256Key` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for tappd.derivek256key. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Tappd.DeriveK256Key` with a valid `GetKeyArgs` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `DeriveK256KeyResponse` with every documented field and exhibits the documented `DeriveK256Key` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-002/metadata.json b/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-002/metadata.json new file mode 100644 index 000000000..6fb4de0ac --- /dev/null +++ b/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-002/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-tappd-002", + "title": "Tappd.DeriveK256Key", + "priority": "P1", + "requirements": [ + "req-gos-tappd-002" + ], + "risks": [ + "risk-gos-tappd-002" + ], + "tags": [ + "guest", + "tappd-rpc" + ], + "fixture": { + "profile": "no-tee-dev", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "Tappd.DeriveK256Key" + ], + "execution": { + "entrypoint": "shared/automation/passed-rpc-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-003/case.md b/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-003/case.md new file mode 100644 index 000000000..5bdb8eeca --- /dev/null +++ b/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-003/case.md @@ -0,0 +1,72 @@ + + + +# TC-GOS-TAPPD-003: Tappd.TdxQuote + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-tappd-003](../../../../catalog/feature-audit.md#req-gos-tappd-003) +- Risks: [risk-gos-tappd-003](../../../../catalog/feature-audit.md#risk-gos-tappd-003) +- Source: `dstack/guest-agent/rpc/proto/agent_rpc.proto:21` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Tappd.TdxQuote` takes `TdxQuoteArgs` (`report_data: bytes`, `hash_algorithm: string`, `prefix: string`) and returns `TdxQuoteResponse` (`quote: bytes`, `event_log: string`, `hash_algorithm: string`, `prefix: string`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Tappd.TdxQuote`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Tappd.TdxQuote` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for tappd.tdxquote. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Tappd.TdxQuote` with a valid `TdxQuoteArgs` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `TdxQuoteResponse` with every documented field and exhibits the documented `TdxQuote` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-003/metadata.json b/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-003/metadata.json new file mode 100644 index 000000000..5323c58a0 --- /dev/null +++ b/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-003/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-tappd-003", + "title": "Tappd.TdxQuote", + "priority": "P1", + "requirements": [ + "req-gos-tappd-003" + ], + "risks": [ + "risk-gos-tappd-003" + ], + "tags": [ + "guest", + "tappd-rpc" + ], + "fixture": { + "profile": "no-tee-dev", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "Tappd.TdxQuote" + ], + "execution": { + "entrypoint": "shared/automation/passed-rpc-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-004/case.md b/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-004/case.md new file mode 100644 index 000000000..5e3f7cb8c --- /dev/null +++ b/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-004/case.md @@ -0,0 +1,72 @@ + + + +# TC-GOS-TAPPD-004: Tappd.RawQuote + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-tappd-004](../../../../catalog/feature-audit.md#req-gos-tappd-004) +- Risks: [risk-gos-tappd-004](../../../../catalog/feature-audit.md#risk-gos-tappd-004) +- Source: `dstack/guest-agent/rpc/proto/agent_rpc.proto:28` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Tappd.RawQuote` takes `RawQuoteArgs` (`report_data: bytes`) and returns `TdxQuoteResponse` (`quote: bytes`, `event_log: string`, `hash_algorithm: string`, `prefix: string`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Tappd.RawQuote`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Tappd.RawQuote` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for tappd.rawquote. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Tappd.RawQuote` with a valid `RawQuoteArgs` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `TdxQuoteResponse` with every documented field and exhibits the documented `RawQuote` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-004/metadata.json b/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-004/metadata.json new file mode 100644 index 000000000..d08e2df27 --- /dev/null +++ b/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-004/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-tappd-004", + "title": "Tappd.RawQuote", + "priority": "P1", + "requirements": [ + "req-gos-tappd-004" + ], + "risks": [ + "risk-gos-tappd-004" + ], + "tags": [ + "guest", + "tappd-rpc" + ], + "fixture": { + "profile": "no-tee-dev", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "Tappd.RawQuote" + ], + "execution": { + "entrypoint": "shared/automation/passed-rpc-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-005/case.md b/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-005/case.md new file mode 100644 index 000000000..93a72c1ba --- /dev/null +++ b/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-005/case.md @@ -0,0 +1,73 @@ + + + +# TC-GOS-TAPPD-005: Tappd.Info + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-tappd-005](../../../../catalog/feature-audit.md#req-gos-tappd-005) +- Risks: [risk-gos-tappd-005](../../../../catalog/feature-audit.md#risk-gos-tappd-005) +- Source: `dstack/guest-agent/rpc/proto/agent_rpc.proto:31` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Tappd.Info` takes `google.protobuf.Empty` (no fields) and returns `AppInfo` (`app_id: bytes`, `instance_id: bytes`, `app_cert: string`, `tcb_info: string`, `app_name: string`, `device_id: bytes`, `mr_aggregated: bytes`, `os_image_hash: bytes`, `key_provider_info: string`, `compose_hash: bytes`, `vm_config: string`, `cloud_vendor: string`, `cloud_product: string`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- Empty-input transport semantics: `prpc-build` intentionally generates a zero-argument handler for `google.protobuf.Empty` and does not decode the request body. Exercise empty and extraneous/malformed bodies as body-ignored compatibility inputs, and use an invalid pRPC route for the negative transport check; GET is an explicitly supported JSON transport and is not a negative case. Do not expect malformed body rejection from an Empty-input handler. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Tappd.Info`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Tappd.Info` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. `google.protobuf.Empty` has no request fields. Exercise the documented empty JSON/protobuf encodings plus extraneous and malformed body bytes, which the generated zero-argument handler intentionally ignores, and validate every response field, nested message field, and presence bit. Use an invalid pRPC route for negative transport framing; GET is supported by `ra-rpc`. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for tappd.info. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Tappd.Info` with a valid `google.protobuf.Empty` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send extraneous and malformed request-body bytes to confirm Empty-input body-ignore semantics, exercise an invalid pRPC route, and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `AppInfo` with every documented field and exhibits the documented `Info` state and side effects; extraneous or malformed bodies are ignored without changing the response or state, invalid routing returns a structured transport error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with an invalid route or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid routing or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-005/metadata.json b/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-005/metadata.json new file mode 100644 index 000000000..1174b6606 --- /dev/null +++ b/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-005/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-tappd-005", + "title": "Tappd.Info", + "priority": "P1", + "requirements": [ + "req-gos-tappd-005" + ], + "risks": [ + "risk-gos-tappd-005" + ], + "tags": [ + "guest", + "tappd-rpc" + ], + "fixture": { + "profile": "no-tee-dev", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "Tappd.Info" + ], + "execution": { + "entrypoint": "shared/automation/passed-rpc-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-006/case.md b/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-006/case.md new file mode 100644 index 000000000..98099a168 --- /dev/null +++ b/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-006/case.md @@ -0,0 +1,73 @@ + + + +# TC-GOS-TAPPD-006: Tappd.Version + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-tappd-006](../../../../catalog/feature-audit.md#req-gos-tappd-006) +- Risks: [risk-gos-tappd-006](../../../../catalog/feature-audit.md#risk-gos-tappd-006) +- Source: `dstack/guest-agent/rpc/proto/agent_rpc.proto:34` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Tappd.Version` takes `google.protobuf.Empty` (no fields) and returns `WorkerVersion` (`version: string`, `rev: string`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- Empty-input transport semantics: `prpc-build` intentionally generates a zero-argument handler for `google.protobuf.Empty` and does not decode the request body. Exercise empty and extraneous/malformed bodies as body-ignored compatibility inputs, and use an invalid pRPC route for the negative transport check; GET is an explicitly supported JSON transport and is not a negative case. Do not expect malformed body rejection from an Empty-input handler. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Tappd.Version`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Tappd.Version` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. `google.protobuf.Empty` has no request fields. Exercise the documented empty JSON/protobuf encodings plus extraneous and malformed body bytes, which the generated zero-argument handler intentionally ignores, and validate every response field, nested message field, and presence bit. Use an invalid pRPC route for negative transport framing; GET is supported by `ra-rpc`. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for tappd.version. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Tappd.Version` with a valid `google.protobuf.Empty` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send extraneous and malformed request-body bytes to confirm Empty-input body-ignore semantics, exercise an invalid pRPC route, and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `WorkerVersion` with every documented field and exhibits the documented `Version` state and side effects; extraneous or malformed bodies are ignored without changing the response or state, invalid routing returns a structured transport error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with an invalid route or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid routing or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-006/metadata.json b/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-006/metadata.json new file mode 100644 index 000000000..cd18fc063 --- /dev/null +++ b/test-suites/cases/01-guest-os/01-rpc-tappd/tc-gos-tappd-006/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-tappd-006", + "title": "Tappd.Version", + "priority": "P1", + "requirements": [ + "req-gos-tappd-006" + ], + "risks": [ + "risk-gos-tappd-006" + ], + "tags": [ + "guest", + "tappd-rpc" + ], + "fixture": { + "profile": "no-tee-dev", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "Tappd.Version" + ], + "execution": { + "entrypoint": "shared/automation/passed-rpc-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/01-guest-os/02-rpc-dstackguest/metadata.json b/test-suites/cases/01-guest-os/02-rpc-dstackguest/metadata.json new file mode 100644 index 000000000..d21bbe0cc --- /dev/null +++ b/test-suites/cases/01-guest-os/02-rpc-dstackguest/metadata.json @@ -0,0 +1,4 @@ +{ + "id": "section-guest-os-rpc-dstackguest", + "title": "DstackGuest RPC" +} diff --git a/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-001/case.md b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-001/case.md new file mode 100644 index 000000000..6ed1da728 --- /dev/null +++ b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-001/case.md @@ -0,0 +1,72 @@ + + + +# TC-GOS-DSTACKGUEST-001: DstackGuest.GetTlsKey + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-dstackguest-001](../../../../catalog/feature-audit.md#req-gos-dstackguest-001) +- Risks: [risk-gos-dstackguest-001](../../../../catalog/feature-audit.md#risk-gos-dstackguest-001) +- Source: `dstack/guest-agent/rpc/proto/agent_rpc.proto:41` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `DstackGuest.GetTlsKey` takes `GetTlsKeyArgs` (`subject: string`, `alt_names: string`, `usage_ra_tls: bool`, `usage_server_auth: bool`, `usage_client_auth: bool`, `not_before: uint64`, `not_after: uint64`, `with_app_info: bool`) and returns `GetTlsKeyResponse` (`key: string`, `certificate_chain: string`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `DstackGuest.GetTlsKey`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `DstackGuest.GetTlsKey` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for dstackguest.gettlskey. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `DstackGuest.GetTlsKey` with a valid `GetTlsKeyArgs` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `GetTlsKeyResponse` with every documented field and exhibits the documented `GetTlsKey` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-001/metadata.json b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-001/metadata.json new file mode 100644 index 000000000..5d6700857 --- /dev/null +++ b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-001/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-dstackguest-001", + "title": "DstackGuest.GetTlsKey", + "priority": "P1", + "requirements": [ + "req-gos-dstackguest-001" + ], + "risks": [ + "risk-gos-dstackguest-001" + ], + "tags": [ + "guest", + "dstackguest-rpc" + ], + "fixture": { + "profile": "no-tee-dev", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "DstackGuest.GetTlsKey" + ], + "execution": { + "entrypoint": "shared/automation/passed-rpc-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-002/case.md b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-002/case.md new file mode 100644 index 000000000..6e497c9d8 --- /dev/null +++ b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-002/case.md @@ -0,0 +1,72 @@ + + + +# TC-GOS-DSTACKGUEST-002: DstackGuest.GetKey + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-dstackguest-002](../../../../catalog/feature-audit.md#req-gos-dstackguest-002) +- Risks: [risk-gos-dstackguest-002](../../../../catalog/feature-audit.md#risk-gos-dstackguest-002) +- Source: `dstack/guest-agent/rpc/proto/agent_rpc.proto:44` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `DstackGuest.GetKey` takes `GetKeyArgs` (`path: string`, `purpose: string`, `algorithm: string`) and returns `GetKeyResponse` (`key: bytes`, `signature_chain: bytes`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `DstackGuest.GetKey`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `DstackGuest.GetKey` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for dstackguest.getkey. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `DstackGuest.GetKey` with a valid `GetKeyArgs` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `GetKeyResponse` with every documented field and exhibits the documented `GetKey` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-002/metadata.json b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-002/metadata.json new file mode 100644 index 000000000..12e44f094 --- /dev/null +++ b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-002/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-dstackguest-002", + "title": "DstackGuest.GetKey", + "priority": "P1", + "requirements": [ + "req-gos-dstackguest-002" + ], + "risks": [ + "risk-gos-dstackguest-002" + ], + "tags": [ + "guest", + "dstackguest-rpc" + ], + "fixture": { + "profile": "no-tee-dev", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "DstackGuest.GetKey" + ], + "execution": { + "entrypoint": "shared/automation/passed-rpc-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-003/case.md b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-003/case.md new file mode 100644 index 000000000..fdb2a8216 --- /dev/null +++ b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-003/case.md @@ -0,0 +1,72 @@ + + + +# TC-GOS-DSTACKGUEST-003: DstackGuest.GetQuote + +## Metadata + +- Priority: P0 +- Type: Functional, API, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-dstackguest-003](../../../../catalog/feature-audit.md#req-gos-dstackguest-003) +- Risks: [risk-gos-dstackguest-003](../../../../catalog/feature-audit.md#risk-gos-dstackguest-003) +- Source: `dstack/guest-agent/rpc/proto/agent_rpc.proto:47` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `DstackGuest.GetQuote` takes `RawQuoteArgs` (`report_data: bytes`) and returns `GetQuoteResponse` (`quote: bytes`, `event_log: string`, `report_data: bytes`, `vm_config: string`, `attestation: bytes`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `DstackGuest.GetQuote`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `DstackGuest.GetQuote` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for dstackguest.getquote. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `DstackGuest.GetQuote` with a valid `RawQuoteArgs` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `GetQuoteResponse` with every documented field and exhibits the documented `GetQuote` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-003/metadata.json b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-003/metadata.json new file mode 100644 index 000000000..cdadc5831 --- /dev/null +++ b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-003/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-dstackguest-003", + "title": "DstackGuest.GetQuote", + "priority": "P0", + "requirements": [ + "req-gos-dstackguest-003" + ], + "risks": [ + "risk-gos-dstackguest-003" + ], + "tags": [ + "guest", + "dstackguest-rpc" + ], + "fixture": { + "profile": "no-tee-dev", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "DstackGuest.GetQuote" + ], + "execution": { + "entrypoint": "shared/automation/passed-rpc-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-004/case.md b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-004/case.md new file mode 100644 index 000000000..439ac0673 --- /dev/null +++ b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-004/case.md @@ -0,0 +1,72 @@ + + + +# TC-GOS-DSTACKGUEST-004: DstackGuest.Attest + +## Metadata + +- Priority: P0 +- Type: Functional, API, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-dstackguest-004](../../../../catalog/feature-audit.md#req-gos-dstackguest-004) +- Risks: [risk-gos-dstackguest-004](../../../../catalog/feature-audit.md#risk-gos-dstackguest-004) +- Source: `dstack/guest-agent/rpc/proto/agent_rpc.proto:51` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `DstackGuest.Attest` takes `RawQuoteArgs` (`report_data: bytes`) and returns `AttestResponse` (`attestation: bytes`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `DstackGuest.Attest`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `DstackGuest.Attest` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for dstackguest.attest. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `DstackGuest.Attest` with a valid `RawQuoteArgs` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `AttestResponse` with every documented field and exhibits the documented `Attest` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-004/metadata.json b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-004/metadata.json new file mode 100644 index 000000000..55528422a --- /dev/null +++ b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-004/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-dstackguest-004", + "title": "DstackGuest.Attest", + "priority": "P0", + "requirements": [ + "req-gos-dstackguest-004" + ], + "risks": [ + "risk-gos-dstackguest-004" + ], + "tags": [ + "guest", + "dstackguest-rpc" + ], + "fixture": { + "profile": "no-tee-dev", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "DstackGuest.Attest" + ], + "execution": { + "entrypoint": "shared/automation/passed-rpc-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-005/case.md b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-005/case.md new file mode 100644 index 000000000..fa074e1b6 --- /dev/null +++ b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-005/case.md @@ -0,0 +1,73 @@ + + + +# TC-GOS-DSTACKGUEST-005: DstackGuest.Info + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-dstackguest-005](../../../../catalog/feature-audit.md#req-gos-dstackguest-005) +- Risks: [risk-gos-dstackguest-005](../../../../catalog/feature-audit.md#risk-gos-dstackguest-005) +- Source: `dstack/guest-agent/rpc/proto/agent_rpc.proto:54` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `DstackGuest.Info` takes `google.protobuf.Empty` (no fields) and returns `AppInfo` (`app_id: bytes`, `instance_id: bytes`, `app_cert: string`, `tcb_info: string`, `app_name: string`, `device_id: bytes`, `mr_aggregated: bytes`, `os_image_hash: bytes`, `key_provider_info: string`, `compose_hash: bytes`, `vm_config: string`, `cloud_vendor: string`, `cloud_product: string`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- Empty-input transport semantics: `prpc-build` intentionally generates a zero-argument handler for `google.protobuf.Empty` and does not decode the request body. Exercise empty and extraneous/malformed bodies as body-ignored compatibility inputs, and use an invalid pRPC route for the negative transport check; GET is an explicitly supported JSON transport and is not a negative case. Do not expect malformed body rejection from an Empty-input handler. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `DstackGuest.Info`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `DstackGuest.Info` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. `google.protobuf.Empty` has no request fields. Exercise the documented empty JSON/protobuf encodings plus extraneous and malformed body bytes, which the generated zero-argument handler intentionally ignores, and validate every response field, nested message field, and presence bit. Use an invalid pRPC route for negative transport framing; GET is supported by `ra-rpc`. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for dstackguest.info. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `DstackGuest.Info` with a valid `google.protobuf.Empty` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send extraneous and malformed request-body bytes to confirm Empty-input body-ignore semantics, exercise an invalid pRPC route, and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `AppInfo` with every documented field and exhibits the documented `Info` state and side effects; extraneous or malformed bodies are ignored without changing the response or state, invalid routing returns a structured transport error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with an invalid route or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid routing or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-005/metadata.json b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-005/metadata.json new file mode 100644 index 000000000..dcfc2ca61 --- /dev/null +++ b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-005/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-dstackguest-005", + "title": "DstackGuest.Info", + "priority": "P1", + "requirements": [ + "req-gos-dstackguest-005" + ], + "risks": [ + "risk-gos-dstackguest-005" + ], + "tags": [ + "guest", + "dstackguest-rpc" + ], + "fixture": { + "profile": "no-tee-dev", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "DstackGuest.Info" + ], + "execution": { + "entrypoint": "shared/automation/passed-rpc-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-006/case.md b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-006/case.md new file mode 100644 index 000000000..993152242 --- /dev/null +++ b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-006/case.md @@ -0,0 +1,73 @@ + + + +# TC-GOS-DSTACKGUEST-006: DstackGuest.GpuInfo + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-dstackguest-006](../../../../catalog/feature-audit.md#req-gos-dstackguest-006) +- Risks: [risk-gos-dstackguest-006](../../../../catalog/feature-audit.md#risk-gos-dstackguest-006) +- Source: `dstack/guest-agent/rpc/proto/agent_rpc.proto:57` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `DstackGuest.GpuInfo` takes `google.protobuf.Empty` (no fields) and returns `GpuInfoResponse` (`attestation: string`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- Empty-input transport semantics: `prpc-build` intentionally generates a zero-argument handler for `google.protobuf.Empty` and does not decode the request body. Exercise empty and extraneous/malformed bodies as body-ignored compatibility inputs, and use an invalid pRPC route for the negative transport check; GET is an explicitly supported JSON transport and is not a negative case. Do not expect malformed body rejection from an Empty-input handler. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `DstackGuest.GpuInfo`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `DstackGuest.GpuInfo` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. `google.protobuf.Empty` has no request fields. Exercise the documented empty JSON/protobuf encodings plus extraneous and malformed body bytes, which the generated zero-argument handler intentionally ignores, and validate every response field, nested message field, and presence bit. Use an invalid pRPC route for negative transport framing; GET is supported by `ra-rpc`. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for dstackguest.gpuinfo. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `DstackGuest.GpuInfo` with a valid `google.protobuf.Empty` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send extraneous and malformed request-body bytes to confirm Empty-input body-ignore semantics, exercise an invalid pRPC route, and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `GpuInfoResponse` with every documented field and exhibits the documented `GpuInfo` state and side effects; extraneous or malformed bodies are ignored without changing the response or state, invalid routing returns a structured transport error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with an invalid route or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid routing or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-006/metadata.json b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-006/metadata.json new file mode 100644 index 000000000..0a7259e6d --- /dev/null +++ b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-006/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-dstackguest-006", + "title": "DstackGuest.GpuInfo", + "priority": "P1", + "requirements": [ + "req-gos-dstackguest-006" + ], + "risks": [ + "risk-gos-dstackguest-006" + ], + "tags": [ + "guest", + "dstackguest-rpc" + ], + "fixture": { + "profile": "no-tee-dev", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "DstackGuest.GpuInfo" + ], + "execution": { + "entrypoint": "shared/automation/passed-rpc-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-007/case.md b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-007/case.md new file mode 100644 index 000000000..7fa2afab5 --- /dev/null +++ b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-007/case.md @@ -0,0 +1,73 @@ + + + +# TC-GOS-DSTACKGUEST-007: DstackGuest.Sign + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-dstackguest-007](../../../../catalog/feature-audit.md#req-gos-dstackguest-007) +- Risks: [risk-gos-dstackguest-007](../../../../catalog/feature-audit.md#risk-gos-dstackguest-007) +- Source: `dstack/guest-agent/rpc/proto/agent_rpc.proto:60` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `DstackGuest.Sign` takes `SignRequest` (`algorithm: string`, `data: bytes`) and returns `SignResponse` (`signature: bytes`, `signature_chain: bytes`, `public_key: bytes`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- Signing algorithm semantics: accepted values are `ed25519`, `secp256k1`, its `k256` alias, and `secp256k1_prehashed`; empty/other values fail. Prehashed input must be exactly 32 bytes. Ed25519 and secp256k1 signatures are 64 bytes, their public keys are respectively 32 and compressed 33 bytes, and the returned signature chain has three entries. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `DstackGuest.Sign`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `DstackGuest.Sign` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for dstackguest.sign. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `DstackGuest.Sign` with a valid `SignRequest` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `SignResponse` with every documented field and exhibits the documented `Sign` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-007/metadata.json b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-007/metadata.json new file mode 100644 index 000000000..b06133ec2 --- /dev/null +++ b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-007/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-dstackguest-007", + "title": "DstackGuest.Sign", + "priority": "P1", + "requirements": [ + "req-gos-dstackguest-007" + ], + "risks": [ + "risk-gos-dstackguest-007" + ], + "tags": [ + "guest", + "dstackguest-rpc" + ], + "fixture": { + "profile": "no-tee-dev", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "DstackGuest.Sign" + ], + "execution": { + "entrypoint": "shared/automation/passed-rpc-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-008/case.md b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-008/case.md new file mode 100644 index 000000000..f46fc0867 --- /dev/null +++ b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-008/case.md @@ -0,0 +1,73 @@ + + + +# TC-GOS-DSTACKGUEST-008: DstackGuest.Verify + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-dstackguest-008](../../../../catalog/feature-audit.md#req-gos-dstackguest-008) +- Risks: [risk-gos-dstackguest-008](../../../../catalog/feature-audit.md#risk-gos-dstackguest-008) +- Source: `dstack/guest-agent/rpc/proto/agent_rpc.proto:63` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `DstackGuest.Verify` takes `VerifyRequest` (`algorithm: string`, `data: bytes`, `signature: bytes`, `public_key: bytes`) and returns `VerifyResponse` (`valid: bool`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- Verification algorithm semantics: accepted values are `ed25519`, `secp256k1`, its `k256` alias, and `secp256k1_prehashed`; empty/other values fail. Prehashed input must be exactly 32 bytes. Verify valid signatures for each family and require `valid: false` for a well-formed but mismatched signature/data pair; malformed key/signature encodings return structured errors. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `DstackGuest.Verify`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `DstackGuest.Verify` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for dstackguest.verify. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `DstackGuest.Verify` with a valid `VerifyRequest` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `VerifyResponse` with every documented field and exhibits the documented `Verify` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-008/metadata.json b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-008/metadata.json new file mode 100644 index 000000000..c99ac5afd --- /dev/null +++ b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-008/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-dstackguest-008", + "title": "DstackGuest.Verify", + "priority": "P1", + "requirements": [ + "req-gos-dstackguest-008" + ], + "risks": [ + "risk-gos-dstackguest-008" + ], + "tags": [ + "guest", + "dstackguest-rpc" + ], + "fixture": { + "profile": "no-tee-dev", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "DstackGuest.Verify" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-008/run.py b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-008/run.py new file mode 100755 index 000000000..b1678e2cb --- /dev/null +++ b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-008/run.py @@ -0,0 +1,551 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 +"""Deterministic DstackGuest.Verify signature-verification contract regression. + +`Verify` is a pure function over material that only the guest can produce, so +the harness first calls `DstackGuest.Sign` to obtain a genuine signature and +public key for the lease-owned app key, then verifies that pair, then proves a +tampered signature and a tampered message are rejected. Hard-coding a +signature would bind the case to one lease's derived key. +""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import subprocess +import sys +import tempfile +from typing import Any + +CASE = "tc-gos-dstackguest-008" +SERVICE = "DstackGuest" +# `k256` aliases `secp256k1`; `secp256k1_prehashed` requires exactly 32 bytes, +# which the lease-derived probe message always satisfies. +ALGORITHMS = ("ed25519", "secp256k1", "k256", "secp256k1_prehashed") + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as output: + json.dump(value, output, ensure_ascii=False, indent=2, sort_keys=True) + output.write("\n") + temporary = pathlib.Path(output.name) + temporary.replace(path) + + +def varint(value: int) -> bytes: + """Encode an integer as a protobuf varint.""" + output = bytearray() + while value > 0x7F: + output.append((value & 0x7F) | 0x80) + value >>= 7 + output.append(value) + return bytes(output) + + +def length_delimited(number: int, raw: bytes) -> bytes: + """Encode one length-delimited protobuf field.""" + return varint((number << 3) | 2) + varint(len(raw)) + raw + + +def decode_wire(data: bytes) -> dict[int, list[Any]]: + """Decode a bounded protobuf response into field-number buckets.""" + values: dict[int, list[Any]] = {} + offset = 0 + while offset < len(data): + key = shift = 0 + while True: + byte = data[offset] + offset += 1 + key |= (byte & 0x7F) << shift + if byte < 0x80: + break + shift += 7 + number, wire = key >> 3, key & 7 + if wire == 0: + value = shift = 0 + while True: + byte = data[offset] + offset += 1 + value |= (byte & 0x7F) << shift + if byte < 0x80: + break + shift += 7 + elif wire == 2: + length = shift = 0 + while True: + byte = data[offset] + offset += 1 + length |= (byte & 0x7F) << shift + if byte < 0x80: + break + shift += 7 + value = data[offset : offset + length] + offset += length + else: + raise AssertionError(f"unsupported response wire type {wire}") + values.setdefault(number, []).append(value) + return values + + +def call(socket: str, route: str, content_type: str, body: bytes) -> tuple[int, bytes]: + """Call one unix-socket pRPC endpoint.""" + marker = b"\nDSTACK_HTTP_STATUS:" + process = subprocess.run( + [ + "curl", + "--silent", + "--show-error", + "--unix-socket", + socket, + "--request", + "POST", + "--header", + f"Content-Type: {content_type}", + "--data-binary", + "@-", + "--write-out", + marker.decode() + "%{http_code}", + "http://localhost" + route, + ], + input=body, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + timeout=30, + check=False, + ) + if process.returncode: + raise RuntimeError(process.stderr.decode(errors="replace")[-1000:]) + response, code = process.stdout.rsplit(marker, 1) + return int(code), response + + +def inventory_entry(root: pathlib.Path, service: str, method: str) -> dict[str, Any]: + """Load the authoritative API inventory entry for one method.""" + document = json.loads((root / "catalog" / "api-inventory.json").read_text()) + matches: list[dict[str, Any]] = [] + + def walk(value: Any) -> None: + if isinstance(value, dict): + if value.get("service") == service and value.get("method") == method: + matches.append(value) + for child in value.values(): + walk(child) + elif isinstance(value, list): + for child in value: + walk(child) + + walk(document) + if len(matches) != 1: + raise AssertionError(f"expected one inventory entry for {service}.{method}") + return matches[0] + + +def read_varint(data: bytes, offset: int) -> tuple[int, int]: + """Read a protobuf varint from a buffer.""" + value = shift = 0 + while True: + byte = data[offset] + offset += 1 + value |= (byte & 0x7F) << shift + if byte < 0x80: + return value, offset + shift += 7 + + +def structured_error(body: bytes) -> str: + """Return the structured error of a rejected pRPC response. + + A rejection is framed in the representation of its request: a JSON request + is answered with an `error` member, while a binary request is answered with + a protobuf message whose field 1 carries the message. + """ + if body[:1] == b"\x0a": + length, offset = read_varint(body, 1) + text = body[offset : offset + length].decode(errors="replace") + if text: + return text + try: + value = json.loads(body) + except json.JSONDecodeError as error: + raise AssertionError("rejection was not structured JSON or protobuf") from error + message = value.get("error") + if not isinstance(message, str) or not message: + raise AssertionError("rejection omitted a structured error") + return message + + +def flip_first_byte(raw: bytes) -> bytes: + """Return the value with its first byte flipped.""" + mutated = bytearray(raw) + mutated[0] ^= 0x01 + return bytes(mutated) + + +class Verifier: + """Drive Sign/Verify over both pRPC representations for one fixture.""" + + def __init__(self, socket: str, route: str, fields: list[dict[str, Any]]) -> None: + """Bind the harness to the lease-owned socket and Verify route.""" + self.socket = socket + self.route = route + self.numbers = {field["name"]: int(field["number"]) for field in fields} + + def sign(self, sign_route: str, algorithm: str, data: bytes) -> tuple[bytes, bytes]: + """Produce a genuine signature and public key for one algorithm.""" + payload = {"algorithm": algorithm, "data": data.hex()} + code, body = call( + self.socket, sign_route, "application/json", json.dumps(payload).encode() + ) + if code != 200: + raise AssertionError(f"Sign({algorithm}) returned HTTP {code}") + value = json.loads(body) + for name in ("signature", "signature_chain", "public_key"): + if name not in value: + raise AssertionError(f"Sign({algorithm}) omitted {name}") + return bytes.fromhex(value["signature"]), bytes.fromhex(value["public_key"]) + + def verify_json(self, payload: dict[str, Any]) -> tuple[int, bytes]: + """Send one JSON Verify request.""" + return call( + self.socket, self.route, "application/json", json.dumps(payload).encode() + ) + + def verify_protobuf(self, payload: dict[str, Any], extra: bytes = b"") -> bytes: + """Send one binary protobuf Verify request and return the raw response.""" + body = b"" + for name in ("algorithm", "data", "signature", "public_key"): + raw = payload[name] + body += length_delimited( + self.numbers[name], raw.encode() if isinstance(raw, str) else raw + ) + code, response = call( + self.socket, self.route, "application/octet-stream", body + extra + ) + if code != 200: + raise AssertionError(f"protobuf Verify returned HTTP {code}") + return response + + def valid_flag(self, response: bytes) -> bool: + """Read the `valid` flag from a protobuf VerifyResponse.""" + wire = decode_wire(response) + # proto3 omits a false bool, so an empty body is a well-formed false. + return bool(wire.get(1, [0])[0]) + + +def request_payload(algorithm: str, data: bytes, signature: bytes, key: bytes) -> dict: + """Build a JSON-shaped Verify payload with hex-encoded byte fields.""" + return { + "algorithm": algorithm, + "data": data.hex(), + "signature": signature.hex(), + "public_key": key.hex(), + } + + +def roundtrip(client: Verifier, sign_route: str, algorithm: str, data: bytes) -> dict: + """Sign with one algorithm, then verify genuine and tampered material.""" + signature, key = client.sign(sign_route, algorithm, data) + payload = request_payload(algorithm, data, signature, key) + code, body = client.verify_json(payload) + if code != 200 or json.loads(body).get("valid") is not True: + raise AssertionError(f"{algorithm}: genuine signature was not accepted") + if not client.valid_flag( + client.verify_protobuf( + { + "algorithm": algorithm, + "data": data, + "signature": signature, + "public_key": key, + } + ) + ): + raise AssertionError( + f"{algorithm}: protobuf verification of a genuine " + "signature returned valid=false" + ) + tampered = request_payload(algorithm, data, flip_first_byte(signature), key) + tampered_code, tampered_body = client.verify_json(tampered) + if tampered_code != 200 or json.loads(tampered_body).get("valid") is not False: + raise AssertionError(f"{algorithm}: a tampered signature was not rejected") + if client.valid_flag( + client.verify_protobuf( + { + "algorithm": algorithm, + "data": data, + "signature": flip_first_byte(signature), + "public_key": key, + } + ) + ): + raise AssertionError( + f"{algorithm}: protobuf verification accepted a tampered signature" + ) + altered = request_payload(algorithm, flip_first_byte(data), signature, key) + altered_code, altered_body = client.verify_json(altered) + if altered_code != 200 or json.loads(altered_body).get("valid") is not False: + raise AssertionError(f"{algorithm}: a tampered message was not rejected") + return { + "signature_bytes": len(signature), + "public_key_bytes": len(key), + "public_key_sha256": hashlib.sha256(key).hexdigest(), + "genuine_json_valid": True, + "genuine_protobuf_valid": True, + "tampered_signature_valid": False, + "tampered_message_valid": False, + } + + +def negatives( + client: Verifier, algorithm: str, data: bytes, signature: bytes, key: bytes +) -> dict[str, Any]: + """Exercise the rejection contract of Verify.""" + base = request_payload(algorithm, data, signature, key) + observed: dict[str, Any] = {} + for name, payload in ( + ("unsupported_algorithm", {**base, "algorithm": "dstack-test-unsupported"}), + ("absent_fields", {}), + ("malformed_signature", {**base, "signature": "aabb"}), + ("malformed_public_key", {**base, "public_key": "aabb"}), + ("schema_invalid_algorithm", {**base, "algorithm": 123}), + ): + code, body = client.verify_json(payload) + if code < 400: + raise AssertionError(f"{name} was accepted with HTTP {code}") + observed[name] = {"http": code, "error": structured_error(body)} + code, body = call(client.socket, client.route, "application/json", b'{"algorithm":') + if code < 400: + raise AssertionError("malformed JSON framing was accepted") + observed["malformed_json"] = {"http": code, "error": structured_error(body)} + code, body = call( + client.socket, client.route, "application/octet-stream", b"\x0a\xff" + ) + if code < 400: + raise AssertionError("malformed protobuf framing was accepted") + observed["malformed_protobuf"] = {"http": code, "error": structured_error(body)} + code, body = call( + client.socket, client.route + "-invalid", "application/json", b"{}" + ) + if code < 400: + raise AssertionError("an unknown route was accepted") + observed["invalid_route"] = {"http": code, "error": structured_error(body)} + return observed + + +def log_observation(path: str) -> dict[str, Any]: + """Summarise the tail of the lease-owned simulator log without content.""" + log = pathlib.Path(path) + if not log.is_file(): + return {"available": False} + lines = log.read_text(encoding="utf-8", errors="replace").splitlines()[-200:] + lowered = [line.lower() for line in lines] + return { + "available": True, + "observed_lines": len(lines), + "panic_lines": sum(1 for line in lowered if "panic" in line), + "error_lines": sum(1 for line in lowered if "error" in line), + "sha256": hashlib.sha256("\n".join(lines).encode()).hexdigest(), + } + + +def main() -> int: + """Run the DstackGuest.Verify regression.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE: + raise RuntimeError(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + plan_root = pathlib.Path(os.environ["DSTACK_TEST_PLAN_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + steps: list[dict[str, str]] = [] + failures: list[str] = [] + evidence: dict[str, Any] = {"case_id": case_id, "environment": "SIMULATION"} + try: + print(f"STEP {case_id}-step-01 START", flush=True) + values = manifest["values"] + service = values["services"][SERVICE] + socket = str(service["socket"]) + if not pathlib.Path(socket).is_socket(): + raise AssertionError(f"fixture socket is not available: {socket}") + verify_entry = inventory_entry(plan_root, SERVICE, "Verify") + sign_entry = inventory_entry(plan_root, SERVICE, "Sign") + route = str(service["route"]).replace("", "Verify") + sign_route = str(service["route"]).replace("", "Sign") + client = Verifier(socket, route, verify_entry["request_fields"]) + evidence["fixture"] = { + "profile": manifest["profile"], + "lease_id": manifest["lease_id"], + "socket_available": True, + "verify_request_fields": [ + field["name"] for field in verify_entry["request_fields"] + ], + "verify_response_fields": [ + field["name"] for field in verify_entry["response_fields"] + ], + "sign_response_fields": [ + field["name"] for field in sign_entry["response_fields"] + ], + } + steps.append( + { + "id": f"{case_id}-step-01", + "status": "PASS", + "observed": "The lease-owned simulator socket, Sign route and indexed " + "Verify contract were available with no run-scoped persistent object.", + } + ) + print( + f"EVIDENCE {case_id}-step-01 - Proves the isolated guest listener and the " + "indexed Verify contract were ready.", + flush=True, + ) + print(json.dumps(evidence["fixture"], sort_keys=True), flush=True) + print(f"STEP {case_id}-step-01 END - PASS", flush=True) + + print(f"STEP {case_id}-step-02 START", flush=True) + # Run-scoped, non-production probe message. 32 bytes satisfies the + # secp256k1_prehashed length constraint recorded in api-inventory.json. + data = hashlib.sha256(str(manifest["lease_id"]).encode()).digest() + evidence["algorithms"] = { + algorithm: roundtrip(client, sign_route, algorithm, data) + for algorithm in ALGORITHMS + } + signature, key = client.sign(sign_route, "secp256k1", data) + alias = request_payload("k256", data, signature, key) + alias_code, alias_body = client.verify_json(alias) + if alias_code != 200 or json.loads(alias_body).get("valid") is not True: + raise AssertionError("k256 did not accept a secp256k1 signature") + evidence["alias_k256_accepts_secp256k1"] = True + ed_signature, ed_key = client.sign(sign_route, "ed25519", data) + unknown = { + **request_payload("ed25519", data, ed_signature, ed_key), + f"unknown_{manifest['lease_id']}": 1, + } + unknown_code, unknown_body = client.verify_json(unknown) + if unknown_code != 200 or json.loads(unknown_body).get("valid") is not True: + raise AssertionError("an unknown JSON member changed the Verify result") + if not client.valid_flag( + client.verify_protobuf( + { + "algorithm": "ed25519", + "data": data, + "signature": ed_signature, + "public_key": ed_key, + }, + extra=length_delimited(9, b"unknown"), + ) + ): + raise AssertionError("an unknown protobuf field changed the Verify result") + evidence["unknown_fields_ignored"] = True + evidence["negatives"] = negatives(client, "ed25519", data, ed_signature, ed_key) + steps.append( + { + "id": f"{case_id}-step-02", + "status": "PASS", + "observed": "Every indexed algorithm verified its own Sign output over " + "JSON and protobuf, tampered signatures and messages returned " + "valid=false, and unsupported or malformed input returned structured " + "errors.", + } + ) + print( + f"EVIDENCE {case_id}-step-02 - Proves genuine/tampered verification " + "outcomes and structured rejection across both representations.", + flush=True, + ) + print(json.dumps(evidence["negatives"], sort_keys=True), flush=True) + print(f"STEP {case_id}-step-02 END - PASS", flush=True) + + print(f"STEP {case_id}-step-03 START", flush=True) + payload = request_payload("ed25519", data, ed_signature, ed_key) + first_code, first_body = client.verify_json(payload) + repeat_code, repeat_body = client.verify_json(payload) + if first_code != 200 or repeat_code != 200: + raise AssertionError("Verify was unavailable after invalid input") + # Verify is a pure function of its request, so repeated identical + # requests must be byte-identical: no timestamp or live state is + # carried in VerifyResponse. + if first_body != repeat_body: + raise AssertionError("repeated identical Verify responses differed") + evidence["repeat"] = { + "http": repeat_code, + "byte_identical": True, + "sha256": hashlib.sha256(repeat_body).hexdigest(), + "sensitive_response_persisted": False, + } + evidence["simulator_log"] = log_observation(str(manifest["values"]["log"])) + if evidence["simulator_log"].get("panic_lines"): + raise AssertionError("the simulator log recorded a panic") + steps.append( + { + "id": f"{case_id}-step-03", + "status": "PASS", + "observed": "Repeated identical Verify calls were byte-identical and " + "stateless, the listener survived every rejection, and bounded " + "simulator diagnostics recorded no panic.", + } + ) + print( + f"EVIDENCE {case_id}-step-03 - Proves stateless idempotent repeats, " + "post-error availability and clean bounded diagnostics.", + flush=True, + ) + print(json.dumps(evidence["repeat"], sort_keys=True), flush=True) + print(f"STEP {case_id}-step-03 END - PASS", flush=True) + except Exception as error: # noqa: BLE001 - recorded as a case failure + failures.append(f"{type(error).__name__}: {error}") + completed = {step["id"] for step in steps} + for number in range(1, 4): + step_id = f"{case_id}-step-{number:02d}" + if step_id not in completed: + steps.append( + {"id": step_id, "status": "FAIL", "observed": failures[-1]} + ) + print(failures[-1], file=sys.stderr, flush=True) + + status = "PASS" if not failures else "FAIL" + evidence["status"] = status + evidence["failure"] = failures[0] if failures else None + artifact = { + "name": "DstackGuest.Verify contract matrix", + "path": "artifacts/dstackguest-verify-matrix.json", + "step_id": f"{case_id}-step-02", + "description": "Per-algorithm genuine and tampered verification outcomes, " + "structured rejection statuses, repeat determinism, and bounded simulator " + "diagnostics. Signature material stays in memory; only lengths and public " + "key hashes are recorded.", + } + atomic_json(result_dir / artifact["path"], evidence) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": "DstackGuest.Verify accepted every genuine Sign output over " + "JSON and protobuf, rejected tampered signatures and messages, and " + "returned structured errors for unsupported and malformed input." + if status == "PASS" + else failures[0], + "steps": steps, + "artifacts": [artifact], + "remarks": "SIMULATION: this confirms the Verify RPC contract, not " + "physical TEE trust properties. Verify is stateless, so the case leaves " + "no run-scoped object behind.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-009/case.md b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-009/case.md new file mode 100644 index 000000000..cb215660b --- /dev/null +++ b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-009/case.md @@ -0,0 +1,73 @@ + + + +# TC-GOS-DSTACKGUEST-009: DstackGuest.Version + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-dstackguest-009](../../../../catalog/feature-audit.md#req-gos-dstackguest-009) +- Risks: [risk-gos-dstackguest-009](../../../../catalog/feature-audit.md#risk-gos-dstackguest-009) +- Source: `dstack/guest-agent/rpc/proto/agent_rpc.proto:66` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `DstackGuest.Version` takes `google.protobuf.Empty` (no fields) and returns `WorkerVersion` (`version: string`, `rev: string`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- Empty-input transport semantics: `prpc-build` intentionally generates a zero-argument handler for `google.protobuf.Empty` and does not decode the request body. Exercise empty and extraneous/malformed bodies as body-ignored compatibility inputs, and use an invalid pRPC route for the negative transport check; GET is an explicitly supported JSON transport and is not a negative case. Do not expect malformed body rejection from an Empty-input handler. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `DstackGuest.Version`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `DstackGuest.Version` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. `google.protobuf.Empty` has no request fields. Exercise the documented empty JSON/protobuf encodings plus extraneous and malformed body bytes, which the generated zero-argument handler intentionally ignores, and validate every response field, nested message field, and presence bit. Use an invalid pRPC route for negative transport framing; GET is supported by `ra-rpc`. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for dstackguest.version. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `DstackGuest.Version` with a valid `google.protobuf.Empty` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send extraneous and malformed request-body bytes to confirm Empty-input body-ignore semantics, exercise an invalid pRPC route, and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `WorkerVersion` with every documented field and exhibits the documented `Version` state and side effects; extraneous or malformed bodies are ignored without changing the response or state, invalid routing returns a structured transport error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with an invalid route or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid routing or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-009/metadata.json b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-009/metadata.json new file mode 100644 index 000000000..e3e977526 --- /dev/null +++ b/test-suites/cases/01-guest-os/02-rpc-dstackguest/tc-gos-dstackguest-009/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-dstackguest-009", + "title": "DstackGuest.Version", + "priority": "P1", + "requirements": [ + "req-gos-dstackguest-009" + ], + "risks": [ + "risk-gos-dstackguest-009" + ], + "tags": [ + "guest", + "dstackguest-rpc" + ], + "fixture": { + "profile": "no-tee-dev", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "DstackGuest.Version" + ], + "execution": { + "entrypoint": "shared/automation/passed-rpc-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/01-guest-os/03-rpc-worker/metadata.json b/test-suites/cases/01-guest-os/03-rpc-worker/metadata.json new file mode 100644 index 000000000..31c7bbe7e --- /dev/null +++ b/test-suites/cases/01-guest-os/03-rpc-worker/metadata.json @@ -0,0 +1,4 @@ +{ + "id": "section-guest-os-rpc-worker", + "title": "Worker RPC" +} diff --git a/test-suites/cases/01-guest-os/03-rpc-worker/tc-gos-worker-001/case.md b/test-suites/cases/01-guest-os/03-rpc-worker/tc-gos-worker-001/case.md new file mode 100644 index 000000000..397ae37ce --- /dev/null +++ b/test-suites/cases/01-guest-os/03-rpc-worker/tc-gos-worker-001/case.md @@ -0,0 +1,73 @@ + + + +# TC-GOS-WORKER-001: Worker.Info + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-worker-001](../../../../catalog/feature-audit.md#req-gos-worker-001) +- Risks: [risk-gos-worker-001](../../../../catalog/feature-audit.md#risk-gos-worker-001) +- Source: `dstack/guest-agent/rpc/proto/agent_rpc.proto:255` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Worker.Info` takes `google.protobuf.Empty` (no fields) and returns `AppInfo` (`app_id: bytes`, `instance_id: bytes`, `app_cert: string`, `tcb_info: string`, `app_name: string`, `device_id: bytes`, `mr_aggregated: bytes`, `os_image_hash: bytes`, `key_provider_info: string`, `compose_hash: bytes`, `vm_config: string`, `cloud_vendor: string`, `cloud_product: string`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- Empty-input transport semantics: `prpc-build` intentionally generates a zero-argument handler for `google.protobuf.Empty` and does not decode the request body. Exercise empty and extraneous/malformed bodies as body-ignored compatibility inputs, and use an invalid pRPC route for the negative transport check; GET is an explicitly supported JSON transport and is not a negative case. Do not expect malformed body rejection from an Empty-input handler. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Worker.Info`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Worker.Info` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. `google.protobuf.Empty` has no request fields. Exercise the documented empty JSON/protobuf encodings plus extraneous and malformed body bytes, which the generated zero-argument handler intentionally ignores, and validate every response field, nested message field, and presence bit. Use an invalid pRPC route for negative transport framing; GET is supported by `ra-rpc`. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for worker.info. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Worker.Info` with a valid `google.protobuf.Empty` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send extraneous and malformed request-body bytes to confirm Empty-input body-ignore semantics, exercise an invalid pRPC route, and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `AppInfo` with every documented field and exhibits the documented `Info` state and side effects; extraneous or malformed bodies are ignored without changing the response or state, invalid routing returns a structured transport error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with an invalid route or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid routing or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/03-rpc-worker/tc-gos-worker-001/metadata.json b/test-suites/cases/01-guest-os/03-rpc-worker/tc-gos-worker-001/metadata.json new file mode 100644 index 000000000..77588eea4 --- /dev/null +++ b/test-suites/cases/01-guest-os/03-rpc-worker/tc-gos-worker-001/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-worker-001", + "title": "Worker.Info", + "priority": "P1", + "requirements": [ + "req-gos-worker-001" + ], + "risks": [ + "risk-gos-worker-001" + ], + "tags": [ + "guest", + "worker-rpc" + ], + "fixture": { + "profile": "no-tee-dev", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "Worker.Info" + ], + "execution": { + "entrypoint": "shared/automation/passed-rpc-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/01-guest-os/03-rpc-worker/tc-gos-worker-002/case.md b/test-suites/cases/01-guest-os/03-rpc-worker/tc-gos-worker-002/case.md new file mode 100644 index 000000000..3e026d607 --- /dev/null +++ b/test-suites/cases/01-guest-os/03-rpc-worker/tc-gos-worker-002/case.md @@ -0,0 +1,73 @@ + + + +# TC-GOS-WORKER-002: Worker.Version + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-worker-002](../../../../catalog/feature-audit.md#req-gos-worker-002) +- Risks: [risk-gos-worker-002](../../../../catalog/feature-audit.md#risk-gos-worker-002) +- Source: `dstack/guest-agent/rpc/proto/agent_rpc.proto:257` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Worker.Version` takes `google.protobuf.Empty` (no fields) and returns `WorkerVersion` (`version: string`, `rev: string`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- Empty-input transport semantics: `prpc-build` intentionally generates a zero-argument handler for `google.protobuf.Empty` and does not decode the request body. Exercise empty and extraneous/malformed bodies as body-ignored compatibility inputs, and use an invalid pRPC route for the negative transport check; GET is an explicitly supported JSON transport and is not a negative case. Do not expect malformed body rejection from an Empty-input handler. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Worker.Version`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Worker.Version` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. `google.protobuf.Empty` has no request fields. Exercise the documented empty JSON/protobuf encodings plus extraneous and malformed body bytes, which the generated zero-argument handler intentionally ignores, and validate every response field, nested message field, and presence bit. Use an invalid pRPC route for negative transport framing; GET is supported by `ra-rpc`. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for worker.version. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Worker.Version` with a valid `google.protobuf.Empty` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send extraneous and malformed request-body bytes to confirm Empty-input body-ignore semantics, exercise an invalid pRPC route, and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `WorkerVersion` with every documented field and exhibits the documented `Version` state and side effects; extraneous or malformed bodies are ignored without changing the response or state, invalid routing returns a structured transport error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with an invalid route or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid routing or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/03-rpc-worker/tc-gos-worker-002/metadata.json b/test-suites/cases/01-guest-os/03-rpc-worker/tc-gos-worker-002/metadata.json new file mode 100644 index 000000000..338952d8f --- /dev/null +++ b/test-suites/cases/01-guest-os/03-rpc-worker/tc-gos-worker-002/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-worker-002", + "title": "Worker.Version", + "priority": "P1", + "requirements": [ + "req-gos-worker-002" + ], + "risks": [ + "risk-gos-worker-002" + ], + "tags": [ + "guest", + "worker-rpc" + ], + "fixture": { + "profile": "no-tee-dev", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "Worker.Version" + ], + "execution": { + "entrypoint": "shared/automation/passed-rpc-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/01-guest-os/03-rpc-worker/tc-gos-worker-003/case.md b/test-suites/cases/01-guest-os/03-rpc-worker/tc-gos-worker-003/case.md new file mode 100644 index 000000000..d50aad0aa --- /dev/null +++ b/test-suites/cases/01-guest-os/03-rpc-worker/tc-gos-worker-003/case.md @@ -0,0 +1,73 @@ + + + +# TC-GOS-WORKER-003: Worker.GetAttestationForAppKey + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-worker-003](../../../../catalog/feature-audit.md#req-gos-worker-003) +- Risks: [risk-gos-worker-003](../../../../catalog/feature-audit.md#risk-gos-worker-003) +- Source: `dstack/guest-agent/rpc/proto/agent_rpc.proto:259` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Worker.GetAttestationForAppKey` takes `GetAttestationForAppKeyRequest` (`algorithm: string`) and returns `GetQuoteResponse` (`quote: bytes`, `event_log: string`, `report_data: bytes`, `vm_config: string`, `attestation: bytes`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- Algorithm and report-data semantics: empty is unsupported; accepted explicit values are `ed25519`, `secp256k1`, its `k256` alias, and `secp256k1_prehashed`. Ed25519 report data begins `dip1::ed25519-pk:` plus URL-safe unpadded Base64 of the 32-byte public key; secp256k1 variants begin `dip1::secp256k1c-pk:` plus URL-safe unpadded Base64 of the compressed 33-byte public key, zero-padded to 64 bytes. Other algorithms return a structured error. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Worker.GetAttestationForAppKey`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Worker.GetAttestationForAppKey` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for worker.getattestationforappkey. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Worker.GetAttestationForAppKey` with a valid `GetAttestationForAppKeyRequest` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `GetQuoteResponse` with every documented field and exhibits the documented `GetAttestationForAppKey` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/03-rpc-worker/tc-gos-worker-003/metadata.json b/test-suites/cases/01-guest-os/03-rpc-worker/tc-gos-worker-003/metadata.json new file mode 100644 index 000000000..87e7af551 --- /dev/null +++ b/test-suites/cases/01-guest-os/03-rpc-worker/tc-gos-worker-003/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-worker-003", + "title": "Worker.GetAttestationForAppKey", + "priority": "P1", + "requirements": [ + "req-gos-worker-003" + ], + "risks": [ + "risk-gos-worker-003" + ], + "tags": [ + "guest", + "worker-rpc" + ], + "fixture": { + "profile": "no-tee-dev", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "Worker.GetAttestationForAppKey" + ], + "execution": { + "entrypoint": "shared/automation/passed-rpc-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/01-guest-os/04-rpc-guestapi/metadata.json b/test-suites/cases/01-guest-os/04-rpc-guestapi/metadata.json new file mode 100644 index 000000000..09fc20969 --- /dev/null +++ b/test-suites/cases/01-guest-os/04-rpc-guestapi/metadata.json @@ -0,0 +1,4 @@ +{ + "id": "section-guest-os-rpc-guestapi", + "title": "GuestApi RPC" +} diff --git a/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-001/case.md b/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-001/case.md new file mode 100644 index 000000000..31c9a3a4b --- /dev/null +++ b/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-001/case.md @@ -0,0 +1,73 @@ + + + +# TC-GOS-GUESTAPI-001: GuestApi.Info + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-guestapi-001](../../../../catalog/feature-audit.md#req-gos-guestapi-001) +- Risks: [risk-gos-guestapi-001](../../../../catalog/feature-audit.md#risk-gos-guestapi-001) +- Source: `dstack/guest-api/proto/guest_api.proto:135` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `GuestApi.Info` takes `google.protobuf.Empty` (no fields) and returns `GuestInfo` (`version: string`, `app_id: bytes`, `instance_id: bytes`, `app_cert: string`, `tcb_info: string`, `device_id: bytes`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- Empty-input transport semantics: `prpc-build` intentionally generates a zero-argument handler for `google.protobuf.Empty` and does not decode the request body. Exercise empty and extraneous/malformed bodies as body-ignored compatibility inputs, and use an invalid pRPC route for the negative transport check; GET is an explicitly supported JSON transport and is not a negative case. Do not expect malformed body rejection from an Empty-input handler. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `GuestApi.Info`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `GuestApi.Info` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. `google.protobuf.Empty` has no request fields. Exercise the documented empty JSON/protobuf encodings plus extraneous and malformed body bytes, which the generated zero-argument handler intentionally ignores, and validate every response field, nested message field, and presence bit. Use an invalid pRPC route for negative transport framing; GET is supported by `ra-rpc`. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for guestapi.info. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `GuestApi.Info` with a valid `google.protobuf.Empty` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send extraneous and malformed request-body bytes to confirm Empty-input body-ignore semantics, exercise an invalid pRPC route, and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `GuestInfo` with every documented field and exhibits the documented `Info` state and side effects; extraneous or malformed bodies are ignored without changing the response or state, invalid routing returns a structured transport error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with an invalid route or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid routing or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-001/metadata.json b/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-001/metadata.json new file mode 100644 index 000000000..158197bc2 --- /dev/null +++ b/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-001/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-guestapi-001", + "title": "GuestApi.Info", + "priority": "P1", + "requirements": [ + "req-gos-guestapi-001" + ], + "risks": [ + "risk-gos-guestapi-001" + ], + "tags": [ + "guest", + "guestapi-rpc" + ], + "fixture": { + "profile": "no-tee-dev", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "GuestApi.Info" + ], + "execution": { + "entrypoint": "shared/automation/passed-rpc-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-002/case.md b/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-002/case.md new file mode 100644 index 000000000..77578c2e1 --- /dev/null +++ b/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-002/case.md @@ -0,0 +1,73 @@ + + + +# TC-GOS-GUESTAPI-002: GuestApi.SysInfo + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-guestapi-002](../../../../catalog/feature-audit.md#req-gos-guestapi-002) +- Risks: [risk-gos-guestapi-002](../../../../catalog/feature-audit.md#risk-gos-guestapi-002) +- Source: `dstack/guest-api/proto/guest_api.proto:137` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `GuestApi.SysInfo` takes `google.protobuf.Empty` (no fields) and returns `SystemInfo` (`os_name: string`, `os_version: string`, `kernel_version: string`, `cpu_model: string`, `num_cpus: uint32`, `total_memory: uint64`, `available_memory: uint64`, `used_memory: uint64`, `free_memory: uint64`, `total_swap: uint64`, `used_swap: uint64`, `free_swap: uint64`, `uptime: uint64`, `loadavg_one: uint32`, `loadavg_five: uint32`, `loadavg_fifteen: uint32`, `disks: DiskInfo`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- Empty-input transport semantics: `prpc-build` intentionally generates a zero-argument handler for `google.protobuf.Empty` and does not decode the request body. Exercise empty and extraneous/malformed bodies as body-ignored compatibility inputs, and use an invalid pRPC route for the negative transport check; GET is an explicitly supported JSON transport and is not a negative case. Do not expect malformed body rejection from an Empty-input handler. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `GuestApi.SysInfo`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `GuestApi.SysInfo` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. `google.protobuf.Empty` has no request fields. Exercise the documented empty JSON/protobuf encodings plus extraneous and malformed body bytes, which the generated zero-argument handler intentionally ignores, and validate every response field, nested message field, and presence bit. Use an invalid pRPC route for negative transport framing; GET is supported by `ra-rpc`. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for guestapi.sysinfo. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `GuestApi.SysInfo` with a valid `google.protobuf.Empty` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send extraneous and malformed request-body bytes to confirm Empty-input body-ignore semantics, exercise an invalid pRPC route, and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `SystemInfo` with every documented field and exhibits the documented `SysInfo` state and side effects; extraneous or malformed bodies are ignored without changing the response or state, invalid routing returns a structured transport error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with an invalid route or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid routing or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-002/metadata.json b/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-002/metadata.json new file mode 100644 index 000000000..87d31b1b8 --- /dev/null +++ b/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-002/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-guestapi-002", + "title": "GuestApi.SysInfo", + "priority": "P1", + "requirements": [ + "req-gos-guestapi-002" + ], + "risks": [ + "risk-gos-guestapi-002" + ], + "tags": [ + "guest", + "guestapi-rpc" + ], + "fixture": { + "profile": "no-tee-dev", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "GuestApi.SysInfo" + ], + "execution": { + "entrypoint": "shared/automation/passed-rpc-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-003/case.md b/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-003/case.md new file mode 100644 index 000000000..74102519e --- /dev/null +++ b/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-003/case.md @@ -0,0 +1,73 @@ + + + +# TC-GOS-GUESTAPI-003: GuestApi.NetworkInfo + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-guestapi-003](../../../../catalog/feature-audit.md#req-gos-guestapi-003) +- Risks: [risk-gos-guestapi-003](../../../../catalog/feature-audit.md#risk-gos-guestapi-003) +- Source: `dstack/guest-api/proto/guest_api.proto:139` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `GuestApi.NetworkInfo` takes `google.protobuf.Empty` (no fields) and returns `NetworkInformation` (`dns_servers: string`, `gateways: Gateway`, `interfaces: Interface`, `wg_info: string`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- Empty-input transport semantics: `prpc-build` intentionally generates a zero-argument handler for `google.protobuf.Empty` and does not decode the request body. Exercise empty and extraneous/malformed bodies as body-ignored compatibility inputs, and use an invalid pRPC route for the negative transport check; GET is an explicitly supported JSON transport and is not a negative case. Do not expect malformed body rejection from an Empty-input handler. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `GuestApi.NetworkInfo`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `GuestApi.NetworkInfo` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. `google.protobuf.Empty` has no request fields. Exercise the documented empty JSON/protobuf encodings plus extraneous and malformed body bytes, which the generated zero-argument handler intentionally ignores, and validate every response field, nested message field, and presence bit. Use an invalid pRPC route for negative transport framing; GET is supported by `ra-rpc`. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for guestapi.networkinfo. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `GuestApi.NetworkInfo` with a valid `google.protobuf.Empty` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send extraneous and malformed request-body bytes to confirm Empty-input body-ignore semantics, exercise an invalid pRPC route, and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `NetworkInformation` with every documented field and exhibits the documented `NetworkInfo` state and side effects; extraneous or malformed bodies are ignored without changing the response or state, invalid routing returns a structured transport error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with an invalid route or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid routing or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-003/metadata.json b/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-003/metadata.json new file mode 100644 index 000000000..e06198e95 --- /dev/null +++ b/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-003/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-guestapi-003", + "title": "GuestApi.NetworkInfo", + "priority": "P1", + "requirements": [ + "req-gos-guestapi-003" + ], + "risks": [ + "risk-gos-guestapi-003" + ], + "tags": [ + "guest", + "guestapi-rpc" + ], + "fixture": { + "profile": "no-tee-dev", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "GuestApi.NetworkInfo" + ], + "execution": { + "entrypoint": "shared/automation/passed-rpc-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-004/case.md b/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-004/case.md new file mode 100644 index 000000000..e528e6f99 --- /dev/null +++ b/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-004/case.md @@ -0,0 +1,73 @@ + + + +# TC-GOS-GUESTAPI-004: GuestApi.ListContainers + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-guestapi-004](../../../../catalog/feature-audit.md#req-gos-guestapi-004) +- Risks: [risk-gos-guestapi-004](../../../../catalog/feature-audit.md#risk-gos-guestapi-004) +- Source: `dstack/guest-api/proto/guest_api.proto:141` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `GuestApi.ListContainers` takes `google.protobuf.Empty` (no fields) and returns `ListContainersResponse` (`containers: Container`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- Empty-input transport semantics: `prpc-build` intentionally generates a zero-argument handler for `google.protobuf.Empty` and does not decode the request body. Exercise empty and extraneous/malformed bodies as body-ignored compatibility inputs, and use an invalid pRPC route for the negative transport check; GET is an explicitly supported JSON transport and is not a negative case. Do not expect malformed body rejection from an Empty-input handler. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `GuestApi.ListContainers`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `GuestApi.ListContainers` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. `google.protobuf.Empty` has no request fields. Exercise the documented empty JSON/protobuf encodings plus extraneous and malformed body bytes, which the generated zero-argument handler intentionally ignores, and validate every response field, nested message field, and presence bit. Use an invalid pRPC route for negative transport framing; GET is supported by `ra-rpc`. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for guestapi.listcontainers. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `GuestApi.ListContainers` with a valid `google.protobuf.Empty` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send extraneous and malformed request-body bytes to confirm Empty-input body-ignore semantics, exercise an invalid pRPC route, and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `ListContainersResponse` with every documented field and exhibits the documented `ListContainers` state and side effects; extraneous or malformed bodies are ignored without changing the response or state, invalid routing returns a structured transport error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with an invalid route or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid routing or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-004/metadata.json b/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-004/metadata.json new file mode 100644 index 000000000..0e7830a33 --- /dev/null +++ b/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-004/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-guestapi-004", + "title": "GuestApi.ListContainers", + "priority": "P1", + "requirements": [ + "req-gos-guestapi-004" + ], + "risks": [ + "risk-gos-guestapi-004" + ], + "tags": [ + "guest", + "guestapi-rpc" + ], + "fixture": { + "profile": "no-tee-dev", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "GuestApi.ListContainers" + ], + "execution": { + "entrypoint": "shared/automation/passed-rpc-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-005/case.md b/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-005/case.md new file mode 100644 index 000000000..73db36100 --- /dev/null +++ b/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-005/case.md @@ -0,0 +1,74 @@ + + + +# TC-GOS-GUESTAPI-005: GuestApi.Shutdown + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-guestapi-005](../../../../catalog/feature-audit.md#req-gos-guestapi-005) +- Risks: [risk-gos-guestapi-005](../../../../catalog/feature-audit.md#risk-gos-guestapi-005) +- Source: `dstack/guest-api/proto/guest_api.proto:143` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `GuestApi.Shutdown` takes `google.protobuf.Empty` (no fields) and returns `google.protobuf.Empty` (no fields). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- Empty-input transport semantics: `prpc-build` intentionally generates a zero-argument handler for `google.protobuf.Empty` and does not decode the request body. Exercise empty and extraneous/malformed bodies as body-ignored compatibility inputs, and use an invalid pRPC route for the negative transport check; GET is an explicitly supported JSON transport and is not a negative case. Do not expect malformed body rejection from an Empty-input handler. +- Simulator shutdown safety: never invoke the host's real `systemctl`. The prepared simulator launcher supplies a case-scoped `systemctl` stub and records invocations in `systemctl.log`. At SIMULATOR level, the required side effect is exactly one recorded `poweroff` dispatch per valid call while the simulator remains available until explicit case cleanup; this does not confirm physical guest shutdown. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `GuestApi.Shutdown`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `GuestApi.Shutdown` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. `google.protobuf.Empty` has no request fields. Exercise the documented empty JSON/protobuf encodings plus extraneous and malformed body bytes, which the generated zero-argument handler intentionally ignores, and validate every response field, nested message field, and presence bit. Use an invalid pRPC route for negative transport framing; GET is supported by `ra-rpc`. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for guestapi.shutdown. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `GuestApi.Shutdown` with a valid `google.protobuf.Empty` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send extraneous and malformed request-body bytes to confirm Empty-input body-ignore semantics, exercise an invalid pRPC route, and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `google.protobuf.Empty` with every documented field and exhibits the documented `Shutdown` state and side effects; extraneous or malformed bodies are ignored without changing the response or state, invalid routing returns a structured transport error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with an invalid route or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid routing or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-005/metadata.json b/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-005/metadata.json new file mode 100644 index 000000000..77cd95751 --- /dev/null +++ b/test-suites/cases/01-guest-os/04-rpc-guestapi/tc-gos-guestapi-005/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-guestapi-005", + "title": "GuestApi.Shutdown", + "priority": "P1", + "requirements": [ + "req-gos-guestapi-005" + ], + "risks": [ + "risk-gos-guestapi-005" + ], + "tags": [ + "guest", + "guestapi-rpc" + ], + "fixture": { + "profile": "no-tee-dev", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "GuestApi.Shutdown" + ], + "execution": { + "entrypoint": "shared/automation/passed-rpc-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/metadata.json b/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/metadata.json new file mode 100644 index 000000000..181df6fa9 --- /dev/null +++ b/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/metadata.json @@ -0,0 +1,4 @@ +{ + "id": "section-guest-os-rpc-proxiedguestapi", + "title": "ProxiedGuestApi RPC" +} diff --git a/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-001/case.md b/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-001/case.md new file mode 100644 index 000000000..0a06c7893 --- /dev/null +++ b/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-001/case.md @@ -0,0 +1,72 @@ + + + +# TC-GOS-PROXIEDGUESTAPI-001: ProxiedGuestApi.Info + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-gos-proxiedguestapi-001](../../../../catalog/feature-audit.md#req-gos-proxiedguestapi-001) +- Risks: [risk-gos-proxiedguestapi-001](../../../../catalog/feature-audit.md#risk-gos-proxiedguestapi-001) +- Source: `dstack/guest-api/proto/guest_api.proto:148` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `ProxiedGuestApi.Info` takes `Id` (`id: string`) and returns `GuestInfo` (`version: string`, `app_id: bytes`, `instance_id: bytes`, `app_cert: string`, `tcb_info: string`, `device_id: bytes`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- Target ownership: `ProxiedGuestApi` is implemented by the VMM and mounted under its `/guest` API surface. Do not substitute the guest-agent simulator `GuestApi` socket. Use an isolated candidate VMM endpoint plus a run-scoped VMM guest instance ID recorded in `DSTACK_TEST_RUNTIME_MANIFEST`; if either is absent, report BLOCKED rather than testing a different service. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `ProxiedGuestApi.Info`. + +## Preconditions + +1. The shared plan prerequisites are healthy, an isolated candidate VMM `ProxiedGuestApi` listener is reachable, and the run-scoped guest instance ID resolves to the intended guest. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `ProxiedGuestApi.Info` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for proxiedguestapi.info. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `ProxiedGuestApi.Info` with a valid `Id` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `GuestInfo` with every documented field and exhibits the documented `Info` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-001/metadata.json b/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-001/metadata.json new file mode 100644 index 000000000..f48ea0942 --- /dev/null +++ b/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-001/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-proxiedguestapi-001", + "title": "ProxiedGuestApi.Info", + "priority": "P1", + "requirements": [ + "req-gos-proxiedguestapi-001" + ], + "risks": [ + "risk-gos-proxiedguestapi-001" + ], + "tags": [ + "guest", + "proxiedguestapi-rpc" + ], + "fixture": { + "profile": "guest-readonly", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "ProxiedGuestApi.Info" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-001/run.py b/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-001/run.py new file mode 100755 index 000000000..403ddde6d --- /dev/null +++ b/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-001/run.py @@ -0,0 +1,191 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 +"""Deterministic ProxiedGuestApi.Info JSON/protobuf contract regression.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import tempfile +import urllib.error +import urllib.request +from typing import Any + +CASE = "tc-gos-proxiedguestapi-001" +FIELDS = {"version", "app_id", "instance_id", "app_cert", "tcb_info", "device_id"} + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as output: + json.dump(value, output, indent=2, sort_keys=True) + output.write("\n") + temporary = pathlib.Path(output.name) + temporary.replace(path) + + +def request(url: str, content_type: str, body: bytes) -> tuple[int, bytes]: + """Call one ProxiedGuestApi method.""" + req = urllib.request.Request(url, data=body, headers={"content-type": content_type}) + try: + with urllib.request.urlopen(req, timeout=30) as response: + return response.status, response.read() + except urllib.error.HTTPError as error: + return error.code, error.read() + + +def protobuf_string(value: str) -> bytes: + """Encode one field-one protobuf string request.""" + raw = value.encode() + if len(raw) >= 128: + raise ValueError("fixture VM id is unexpectedly long") + return b"\x0a" + bytes([len(raw)]) + raw + + +def protobuf_fields(raw: bytes) -> set[int]: + """Return length-delimited field numbers from a bounded response.""" + fields: set[int] = set() + offset = 0 + while offset < len(raw): + tag = raw[offset] + offset += 1 + field, wire = tag >> 3, tag & 7 + if wire != 2 or field < 1: + raise ValueError("unexpected protobuf wire encoding") + length = 0 + shift = 0 + while True: + byte = raw[offset] + offset += 1 + length |= (byte & 0x7F) << shift + if byte < 128: + break + shift += 7 + if shift > 28: + raise ValueError("protobuf length overflow") + offset += length + if offset > len(raw): + raise ValueError("truncated protobuf field") + fields.add(field) + return fields + + +def main() -> int: + """Run the promoted regression.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + if case_id != CASE: + raise RuntimeError(f"unsupported case: {case_id}") + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + service = manifest["values"]["services"]["ProxiedGuestApi"] + vm_id = str(service["id"]) + url = str(service["url"]).format(method="Info") + failures: list[str] = [] + evidence: dict[str, Any] = {} + steps: list[dict[str, str]] = [] + try: + print(f"STEP {case_id}-step-01 START", flush=True) + if not vm_id or not url.startswith("http://127.0.0.1:"): + raise AssertionError( + "fixture did not provide an isolated ProxiedGuestApi target" + ) + steps.append( + { + "id": f"{case_id}-step-01", + "status": "PASS", + "observed": "Resolved the lease-owned VM and isolated ProxiedGuestApi endpoint.", + } + ) + print(f"STEP {case_id}-step-01 END - PASS", flush=True) + + print(f"STEP {case_id}-step-02 START", flush=True) + payload = json.dumps({"id": vm_id}, separators=(",", ":")).encode() + json_code, json_raw = request(url, "application/json", payload) + repeated_code, repeated_raw = request(url, "application/json", payload) + response = json.loads(json_raw) + if json_code != 200 or repeated_code != 200 or set(response) != FIELDS: + raise AssertionError("JSON GuestInfo schema was incomplete") + if json_raw != repeated_raw: + raise AssertionError("repeated GuestInfo response was unstable") + proto_code, proto_raw = request( + url, "application/octet-stream", protobuf_string(vm_id) + ) + if proto_code != 200 or protobuf_fields(proto_raw) != set(range(1, 7)): + raise AssertionError("protobuf GuestInfo schema was incomplete") + invalid_code, _ = request(url, "application/json", b'{"id":"invalid"}') + malformed_code, _ = request(url, "application/octet-stream", b"\x0a\xff") + if invalid_code < 400 or malformed_code < 400: + raise AssertionError("invalid ProxiedGuestApi.Info input was accepted") + evidence["matrix"] = { + "json_status": json_code, + "json_fields": sorted(response), + "repeat_status": repeated_code, + "repeat_sha256_equal": hashlib.sha256(json_raw).digest() + == hashlib.sha256(repeated_raw).digest(), + "protobuf_status": proto_code, + "protobuf_fields": sorted(protobuf_fields(proto_raw)), + "invalid_status": invalid_code, + "malformed_status": malformed_code, + "sensitive_values_persisted": False, + } + steps.append( + { + "id": f"{case_id}-step-02", + "status": "PASS", + "observed": "JSON and protobuf returned fields 1-6; invalid requests failed closed.", + } + ) + print(f"STEP {case_id}-step-02 END - PASS", flush=True) + + print(f"STEP {case_id}-step-03 START", flush=True) + steps.append( + { + "id": f"{case_id}-step-03", + "status": "PASS", + "observed": "Repeated read-only Info calls were byte-stable and created no mutable state.", + } + ) + print(f"STEP {case_id}-step-03 END - PASS", flush=True) + except Exception as error: + failures.append(f"{type(error).__name__}: {error}") + for number in range(1, 4): + sid = f"{case_id}-step-{number:02d}" + if not any(step["id"] == sid for step in steps): + steps.append({"id": sid, "status": "FAIL", "observed": failures[-1]}) + artifact = { + "name": "ProxiedGuestApi.Info contract matrix", + "path": "artifacts/proxiedguestapi-info-matrix.json", + "step_id": f"{case_id}-step-02", + "description": "Bounded status, schema, determinism, invalid-input, and no-secret assertions for JSON and protobuf Info calls.", + } + atomic_json(result_dir / artifact["path"], evidence) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + status = "PASS" if not failures else "FAIL" + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": "ProxiedGuestApi.Info deterministic JSON/protobuf regression passed." + if status == "PASS" + else failures[0], + "steps": steps, + "artifacts": [artifact], + "remarks": "Read-only lease-scoped calls; response contents are not persisted.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-002/case.md b/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-002/case.md new file mode 100644 index 000000000..523c2ea28 --- /dev/null +++ b/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-002/case.md @@ -0,0 +1,72 @@ + + + +# TC-GOS-PROXIEDGUESTAPI-002: ProxiedGuestApi.SysInfo + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-gos-proxiedguestapi-002](../../../../catalog/feature-audit.md#req-gos-proxiedguestapi-002) +- Risks: [risk-gos-proxiedguestapi-002](../../../../catalog/feature-audit.md#risk-gos-proxiedguestapi-002) +- Source: `dstack/guest-api/proto/guest_api.proto:149` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `ProxiedGuestApi.SysInfo` takes `Id` (`id: string`) and returns `SystemInfo` (`os_name: string`, `os_version: string`, `kernel_version: string`, `cpu_model: string`, `num_cpus: uint32`, `total_memory: uint64`, `available_memory: uint64`, `used_memory: uint64`, `free_memory: uint64`, `total_swap: uint64`, `used_swap: uint64`, `free_swap: uint64`, `uptime: uint64`, `loadavg_one: uint32`, `loadavg_five: uint32`, `loadavg_fifteen: uint32`, `disks: DiskInfo`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- Target ownership: `ProxiedGuestApi` is implemented by the VMM and mounted under its `/guest` API surface. Do not substitute the guest-agent simulator `GuestApi` socket. Use an isolated candidate VMM endpoint plus a run-scoped VMM guest instance ID recorded in `DSTACK_TEST_RUNTIME_MANIFEST`; if either is absent, report BLOCKED rather than testing a different service. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `ProxiedGuestApi.SysInfo`. + +## Preconditions + +1. The shared plan prerequisites are healthy, an isolated candidate VMM `ProxiedGuestApi` listener is reachable, and the run-scoped guest instance ID resolves to the intended guest. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `ProxiedGuestApi.SysInfo` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for proxiedguestapi.sysinfo. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `ProxiedGuestApi.SysInfo` with a valid `Id` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `SystemInfo` with every documented field and exhibits the documented `SysInfo` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-002/metadata.json b/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-002/metadata.json new file mode 100644 index 000000000..96a21ad38 --- /dev/null +++ b/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-002/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-proxiedguestapi-002", + "title": "ProxiedGuestApi.SysInfo", + "priority": "P1", + "requirements": [ + "req-gos-proxiedguestapi-002" + ], + "risks": [ + "risk-gos-proxiedguestapi-002" + ], + "tags": [ + "guest", + "proxiedguestapi-rpc" + ], + "fixture": { + "profile": "guest-readonly", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "ProxiedGuestApi.SysInfo" + ], + "execution": { + "entrypoint": "shared/automation/passed-gos-proxiedguestapi-read-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-003/case.md b/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-003/case.md new file mode 100644 index 000000000..52d692303 --- /dev/null +++ b/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-003/case.md @@ -0,0 +1,72 @@ + + + +# TC-GOS-PROXIEDGUESTAPI-003: ProxiedGuestApi.NetworkInfo + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-gos-proxiedguestapi-003](../../../../catalog/feature-audit.md#req-gos-proxiedguestapi-003) +- Risks: [risk-gos-proxiedguestapi-003](../../../../catalog/feature-audit.md#risk-gos-proxiedguestapi-003) +- Source: `dstack/guest-api/proto/guest_api.proto:150` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `ProxiedGuestApi.NetworkInfo` takes `Id` (`id: string`) and returns `NetworkInformation` (`dns_servers: string`, `gateways: Gateway`, `interfaces: Interface`, `wg_info: string`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- Target ownership: `ProxiedGuestApi` is implemented by the VMM and mounted under its `/guest` API surface. Do not substitute the guest-agent simulator `GuestApi` socket. Use an isolated candidate VMM endpoint plus a run-scoped VMM guest instance ID recorded in `DSTACK_TEST_RUNTIME_MANIFEST`; if either is absent, report BLOCKED rather than testing a different service. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `ProxiedGuestApi.NetworkInfo`. + +## Preconditions + +1. The shared plan prerequisites are healthy, an isolated candidate VMM `ProxiedGuestApi` listener is reachable, and the run-scoped guest instance ID resolves to the intended guest. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `ProxiedGuestApi.NetworkInfo` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for proxiedguestapi.networkinfo. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `ProxiedGuestApi.NetworkInfo` with a valid `Id` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `NetworkInformation` with every documented field and exhibits the documented `NetworkInfo` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-003/metadata.json b/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-003/metadata.json new file mode 100644 index 000000000..4e0c82c15 --- /dev/null +++ b/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-003/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-proxiedguestapi-003", + "title": "ProxiedGuestApi.NetworkInfo", + "priority": "P1", + "requirements": [ + "req-gos-proxiedguestapi-003" + ], + "risks": [ + "risk-gos-proxiedguestapi-003" + ], + "tags": [ + "guest", + "proxiedguestapi-rpc" + ], + "fixture": { + "profile": "network-lifecycle", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "ProxiedGuestApi.NetworkInfo" + ], + "execution": { + "entrypoint": "shared/automation/passed-gos-proxiedguestapi-read-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-004/case.md b/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-004/case.md new file mode 100644 index 000000000..314a73c7f --- /dev/null +++ b/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-004/case.md @@ -0,0 +1,72 @@ + + + +# TC-GOS-PROXIEDGUESTAPI-004: ProxiedGuestApi.ListContainers + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-gos-proxiedguestapi-004](../../../../catalog/feature-audit.md#req-gos-proxiedguestapi-004) +- Risks: [risk-gos-proxiedguestapi-004](../../../../catalog/feature-audit.md#risk-gos-proxiedguestapi-004) +- Source: `dstack/guest-api/proto/guest_api.proto:151` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `ProxiedGuestApi.ListContainers` takes `Id` (`id: string`) and returns `ListContainersResponse` (`containers: Container`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- Target ownership: `ProxiedGuestApi` is implemented by the VMM and mounted under its `/guest` API surface. Do not substitute the guest-agent simulator `GuestApi` socket. Use an isolated candidate VMM endpoint plus a run-scoped VMM guest instance ID recorded in `DSTACK_TEST_RUNTIME_MANIFEST`; if either is absent, report BLOCKED rather than testing a different service. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `ProxiedGuestApi.ListContainers`. + +## Preconditions + +1. The shared plan prerequisites are healthy, an isolated candidate VMM `ProxiedGuestApi` listener is reachable, and the run-scoped guest instance ID resolves to the intended guest. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `ProxiedGuestApi.ListContainers` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for proxiedguestapi.listcontainers. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `ProxiedGuestApi.ListContainers` with a valid `Id` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `ListContainersResponse` with every documented field and exhibits the documented `ListContainers` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-004/metadata.json b/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-004/metadata.json new file mode 100644 index 000000000..6359e614b --- /dev/null +++ b/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-004/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-proxiedguestapi-004", + "title": "ProxiedGuestApi.ListContainers", + "priority": "P1", + "requirements": [ + "req-gos-proxiedguestapi-004" + ], + "risks": [ + "risk-gos-proxiedguestapi-004" + ], + "tags": [ + "guest", + "proxiedguestapi-rpc" + ], + "fixture": { + "profile": "container-observability", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "ProxiedGuestApi.ListContainers" + ], + "execution": { + "entrypoint": "shared/automation/passed-gos-proxiedguestapi-read-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-005/case.md b/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-005/case.md new file mode 100644 index 000000000..ca7e3a9c6 --- /dev/null +++ b/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-005/case.md @@ -0,0 +1,72 @@ + + + +# TC-GOS-PROXIEDGUESTAPI-005: ProxiedGuestApi.Shutdown + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-gos-proxiedguestapi-005](../../../../catalog/feature-audit.md#req-gos-proxiedguestapi-005) +- Risks: [risk-gos-proxiedguestapi-005](../../../../catalog/feature-audit.md#risk-gos-proxiedguestapi-005) +- Source: `dstack/guest-api/proto/guest_api.proto:152` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `ProxiedGuestApi.Shutdown` takes `Id` (`id: string`) and returns `google.protobuf.Empty` (no fields). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- Target ownership: `ProxiedGuestApi` is implemented by the VMM and mounted under its `/guest` API surface. Do not substitute the guest-agent simulator `GuestApi` socket. Use an isolated candidate VMM endpoint plus a run-scoped VMM guest instance ID recorded in `DSTACK_TEST_RUNTIME_MANIFEST`; if either is absent, report BLOCKED rather than testing a different service. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `ProxiedGuestApi.Shutdown`. + +## Preconditions + +1. The shared plan prerequisites are healthy, an isolated candidate VMM `ProxiedGuestApi` listener is reachable, and the run-scoped guest instance ID resolves to the intended guest. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `ProxiedGuestApi.Shutdown` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for proxiedguestapi.shutdown. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `ProxiedGuestApi.Shutdown` with a valid `Id` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `google.protobuf.Empty` with every documented field and exhibits the documented `Shutdown` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-005/metadata.json b/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-005/metadata.json new file mode 100644 index 000000000..4706f0fa7 --- /dev/null +++ b/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-005/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-proxiedguestapi-005", + "title": "ProxiedGuestApi.Shutdown", + "priority": "P1", + "requirements": [ + "req-gos-proxiedguestapi-005" + ], + "risks": [ + "risk-gos-proxiedguestapi-005" + ], + "tags": [ + "guest", + "proxiedguestapi-rpc" + ], + "fixture": { + "profile": "guest-lifecycle", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "ProxiedGuestApi.Shutdown" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-005/run.py b/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-005/run.py new file mode 100755 index 000000000..85fbd3c3e --- /dev/null +++ b/test-suites/cases/01-guest-os/05-rpc-proxiedguestapi/tc-gos-proxiedguestapi-005/run.py @@ -0,0 +1,426 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 +"""ProxiedGuestApi.Shutdown terminal state-transition regression. + +`Shutdown` is the one ProxiedGuestApi method that is not idempotent: it powers +the lease-owned guest off, after which the proxy can no longer reach it. The +table-driven RPC harness calls each method three times and requires all three +to succeed, which cannot model this, so the case owns a harness that drives the +transition once and asserts the state either side of it. + +Every rejection path runs before the transition, so a rejected request is shown +not to disturb a running guest. The success path is exercised once, over the +binary representation; the JSON representation is exercised on the same handler +through a well-formed request whose VM id the lease does not own, which proves +the JSON body decoded and reached the handler rather than the codec. A second +successful Shutdown is impossible by construction: the guest is gone. +""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import subprocess +import sys +import tempfile +import time +import urllib.error +import urllib.request +from typing import Any + +CASE = "tc-gos-proxiedguestapi-005" +METHOD = "Shutdown" +UNKNOWN_VM_ID = "00000000-0000-4000-8000-000000000000" +STOP_TIMEOUT_SECONDS = 90 + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as output: + json.dump(value, output, ensure_ascii=False, indent=2, sort_keys=True) + output.write("\n") + temporary = pathlib.Path(output.name) + temporary.replace(path) + + +def request(url: str, content_type: str, body: bytes) -> tuple[int, bytes]: + """Call one ProxiedGuestApi method over the lease-owned VMM endpoint.""" + call = urllib.request.Request( + url, data=body, headers={"content-type": content_type} + ) + try: + with urllib.request.urlopen(call, timeout=60) as response: + return response.status, response.read() + except urllib.error.HTTPError as error: + return error.code, error.read() + + +def varint(value: int) -> bytes: + """Encode an integer as a protobuf varint.""" + output = bytearray() + while value > 0x7F: + output.append((value & 0x7F) | 0x80) + value >>= 7 + output.append(value) + return bytes(output) + + +def encode_id(vm_id: str) -> bytes: + """Encode the one-field `Id` protobuf request.""" + raw = vm_id.encode() + return b"\x0a" + varint(len(raw)) + raw + + +def read_varint(data: bytes, offset: int) -> tuple[int, int]: + """Read a protobuf varint from a buffer.""" + value = shift = 0 + while True: + byte = data[offset] + offset += 1 + value |= (byte & 0x7F) << shift + if byte < 0x80: + return value, offset + shift += 7 + + +def structured_error(body: bytes) -> str: + """Return the structured error of a rejected pRPC response. + + A rejection is framed in the representation of its request: a JSON request + is answered with an `error` member, while a binary request is answered with + a protobuf message whose field 1 carries the message. + """ + if body[:1] == b"\x0a": + length, offset = read_varint(body, 1) + text = body[offset : offset + length].decode(errors="replace") + if text: + return text + try: + value = json.loads(body) + except json.JSONDecodeError as error: + raise AssertionError("rejection was not structured JSON or protobuf") from error + message = value.get("error") + if not isinstance(message, str) or not message: + raise AssertionError("rejection omitted a structured error") + return message + + +def run_cli(argv: list[str]) -> tuple[int, str]: + """Run a lease-owned VMM CLI command.""" + process = subprocess.run( + argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=60, check=False + ) + return process.returncode, process.stdout.decode(errors="replace") + + +def vm_state(argv: list[str]) -> dict[str, Any]: + """Return the lease-owned VM state reported by the candidate VMM.""" + code, text = run_cli(argv) + if code != 0: + raise AssertionError("the lease-owned VMM did not report VM state") + value = json.loads(text) + if not isinstance(value, dict): + raise AssertionError("the lease-owned VMM returned non-object VM state") + return value + + +def inventory(root: pathlib.Path) -> dict[str, Any]: + """Return the indexed ProxiedGuestApi.Shutdown contract.""" + document = json.loads((root / "catalog" / "api-inventory.json").read_text()) + matches = [ + entry + for entry in document["components"]["guest-os"]["rpc_methods"] + if entry.get("service") == "ProxiedGuestApi" and entry.get("method") == METHOD + ] + if len(matches) != 1: + raise AssertionError(f"expected one inventory entry for {METHOD}") + return matches[0] + + +def log_observation(path: str) -> dict[str, Any]: + """Summarise the lease-owned guest boot log without persisting content.""" + log = pathlib.Path(path) + if not log.is_file(): + return {"available": False} + lines = log.read_text(encoding="utf-8", errors="replace").splitlines()[-200:] + return { + "available": True, + "observed_lines": len(lines), + "panic_lines": sum(1 for line in lines if "panic" in line.lower()), + "sha256": hashlib.sha256("\n".join(lines).encode()).hexdigest(), + "content_persisted": False, + } + + +def main() -> int: + """Run the ProxiedGuestApi.Shutdown regression.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE: + raise RuntimeError(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + plan_root = pathlib.Path(os.environ["DSTACK_TEST_PLAN_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + steps: list[dict[str, str]] = [] + failures: list[str] = [] + evidence: dict[str, Any] = { + "case_id": case_id, + "environment": "HARDWARE", + "service": "ProxiedGuestApi", + "method": METHOD, + } + try: + print(f"STEP {case_id}-step-01 START", flush=True) + values = manifest["values"] + service = values["services"]["ProxiedGuestApi"] + vm_id = str(service["id"]) + url = str(service["url"]).format(method=METHOD) + if not vm_id or not url.startswith("http://127.0.0.1:"): + raise AssertionError( + "fixture did not provide an isolated ProxiedGuestApi target" + ) + if not values.get("destructive_actions_allowed"): + raise AssertionError("the lease does not permit a destructive transition") + info_argv = [str(item) for item in values["vm_info_argv"]] + before = vm_state(info_argv) + if before.get("status") != "running" or before.get("boot_progress") != "done": + raise AssertionError(f"lease guest is not ready: {before.get('status')}") + identity_code, identity_body = request( + str(service["url"]).format(method="Info"), + "application/json", + json.dumps({"id": vm_id}, separators=(",", ":")).encode(), + ) + if identity_code != 200: + raise AssertionError(f"ProxiedGuestApi.Info returned {identity_code}") + observed_instance = str(json.loads(identity_body).get("instance_id", "")) + if observed_instance.lower() != str(values["instance_id"]).lower(): + raise AssertionError("the run-scoped VM id resolved to another guest") + entry = inventory(plan_root) + if entry["response_fields"]: + raise AssertionError("indexed Shutdown response is no longer Empty") + evidence["prerequisite"] = { + "profile": manifest["profile"], + "lease_id": manifest["lease_id"], + "status": before.get("status"), + "boot_progress": before.get("boot_progress"), + "instance_id_matches_lease": True, + "indexed_request_fields": [ + field["name"] for field in entry["request_fields"] + ], + "indexed_response_fields": [], + } + steps.append( + { + "id": f"{case_id}-step-01", + "status": "PASS", + "observed": "The lease-owned VMM reported the intended guest running " + "with boot progress done, the ProxiedGuestApi listener resolved the " + "run-scoped VM id to that guest's instance id, and the indexed " + "Shutdown contract declared an empty response.", + } + ) + print( + f"EVIDENCE {case_id}-step-01 - Proves the isolated VMM listener and the " + "running run-scoped guest were the effective baseline.", + flush=True, + ) + print(json.dumps(evidence["prerequisite"], sort_keys=True), flush=True) + print(f"STEP {case_id}-step-01 END - PASS", flush=True) + + print(f"STEP {case_id}-step-02 START", flush=True) + # Every rejection runs first: a rejected Shutdown must leave the guest + # running, which can only be observed while it still is. + rejections: dict[str, Any] = {} + for name, content_type, body in ( + ("absent_id_json", "application/json", b"{}"), + ("empty_id_json", "application/json", b'{"id":""}'), + ( + "unknown_id_json", + "application/json", + f'{{"id":"{UNKNOWN_VM_ID}"}}'.encode(), + ), + ("schema_invalid_id_json", "application/json", b'{"id":123}'), + ("malformed_json", "application/json", b'{"id":'), + ( + "unknown_id_protobuf", + "application/octet-stream", + encode_id(UNKNOWN_VM_ID), + ), + ("malformed_protobuf", "application/octet-stream", b"\x0a\xff"), + ): + code, body_out = request(url, content_type, body) + if code < 400: + raise AssertionError(f"{name} was accepted with HTTP {code}") + rejections[name] = {"http": code, "error": structured_error(body_out)} + survived = vm_state(info_argv) + if survived.get("status") != "running": + raise AssertionError("a rejected Shutdown disturbed the running guest") + shutdown_code, shutdown_body = request( + url, "application/octet-stream", encode_id(vm_id) + ) + if shutdown_code != 200: + raise AssertionError(f"valid Shutdown returned HTTP {shutdown_code}") + if shutdown_body != b"": + raise AssertionError("Shutdown returned a body for google.protobuf.Empty") + evidence["transition"] = { + "rejections": rejections, + "running_after_rejections": True, + "shutdown_http": shutdown_code, + "shutdown_response_bytes": len(shutdown_body), + "success_representation": "application/octet-stream", + "json_representation_reached_handler": rejections["unknown_id_json"], + "protobuf_representation_reached_handler": rejections[ + "unknown_id_protobuf" + ], + } + steps.append( + { + "id": f"{case_id}-step-02", + "status": "PASS", + "observed": "Absent, empty, unresolvable, wrong-typed and malformed " + "requests were rejected with structured errors in both " + "representations and left the guest running; the valid binary " + "Shutdown returned HTTP 200 with an empty google.protobuf.Empty body.", + } + ) + print( + f"EVIDENCE {case_id}-step-02 - Proves the rejection contract in both " + "representations and the accepted terminal request.", + flush=True, + ) + print(json.dumps(evidence["transition"], sort_keys=True), flush=True) + print(f"STEP {case_id}-step-02 END - PASS", flush=True) + + print(f"STEP {case_id}-step-03 START", flush=True) + deadline = time.monotonic() + STOP_TIMEOUT_SECONDS + after = vm_state(info_argv) + while after.get("status") != "stopped" and time.monotonic() < deadline: + time.sleep(1) + after = vm_state(info_argv) + if after.get("status") == "running": + raise AssertionError( + f"the guest was still running {STOP_TIMEOUT_SECONDS}s after Shutdown" + ) + # The indexed contract documents no repeat semantics for Shutdown, and + # the VMM answers a repeat against a stopped guest either with a proxy + # error or with an accepted no-op depending on how far teardown has + # progressed. Record which one happened and assert only what the case + # requires: the repeat must not bring the guest back. + repeat_code, repeat_body = request( + url, "application/octet-stream", encode_id(vm_id) + ) + settled = vm_state(info_argv) + if settled.get("status") == "running": + raise AssertionError("a repeated Shutdown returned the guest to running") + list_code, list_text = run_cli( + [*[str(item) for item in values["vmm_cli_argv"]], "lsvm", "--json"] + ) + if list_code != 0: + raise AssertionError("the candidate VMM control plane became unavailable") + listed = json.loads(list_text) + rows = listed if isinstance(listed, list) else listed.get("vms", []) + owned = [row for row in rows if str(row.get("id", "")) == vm_id] + scoped_code, scoped_body = request( + url, + "application/json", + f'{{"id":"{UNKNOWN_VM_ID}"}}'.encode(), + ) + if scoped_code < 400: + raise AssertionError("an unowned VM id was accepted after the transition") + evidence["final_state"] = { + "status": after.get("status"), + "shutdown_progress": after.get("shutdown_progress"), + "boot_progress": after.get("boot_progress"), + "repeat_shutdown_http": repeat_code, + "repeat_shutdown_rejected": repeat_code >= 400, + "repeat_shutdown_error": ( + structured_error(repeat_body) if repeat_code >= 400 else None + ), + "status_after_repeat": settled.get("status"), + "lease_vm_listed": bool(owned), + "control_plane_available": True, + "unowned_id_still_rejected": { + "http": scoped_code, + "error": structured_error(scoped_body), + }, + } + evidence["diagnostics"] = log_observation(str(values["serial_log"])) + if evidence["diagnostics"].get("panic_lines"): + raise AssertionError("the lease guest boot log recorded a panic") + steps.append( + { + "id": f"{case_id}-step-03", + "status": "PASS", + "observed": "The lease-owned guest left the running state after " + "Shutdown and did not return to it when the terminal request was " + "repeated, the candidate VMM control plane stayed available and " + "still rejected an unowned VM id, and bounded guest diagnostics " + "recorded no panic.", + } + ) + print( + f"EVIDENCE {case_id}-step-03 - Proves the observed state transition, " + "the repeat outcome and retained service availability.", + flush=True, + ) + print(json.dumps(evidence["final_state"], sort_keys=True), flush=True) + print(f"STEP {case_id}-step-03 END - PASS", flush=True) + except Exception as error: + failures.append(f"{type(error).__name__}: {error}") + completed = {step["id"] for step in steps} + for number in range(1, 4): + step_id = f"{case_id}-step-{number:02d}" + if step_id not in completed: + steps.append( + {"id": step_id, "status": "FAIL", "observed": failures[-1]} + ) + print(failures[-1], file=sys.stderr, flush=True) + + status = "PASS" if not failures else "FAIL" + evidence["status"] = status + evidence["failure"] = failures[0] if failures else None + artifact = { + "name": "ProxiedGuestApi.Shutdown transition matrix", + "path": "artifacts/proxiedguestapi-shutdown-matrix.json", + "step_id": f"{case_id}-step-02", + "description": "Pre-transition rejection statuses in both representations, " + "the accepted terminal request, the observed running-to-stopped transition, " + "the recorded repeat outcome, and bounded guest diagnostics.", + } + atomic_json(result_dir / artifact["path"], evidence) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": "ProxiedGuestApi.Shutdown rejected invalid input in both " + "representations without disturbing the running guest, returned an empty " + "google.protobuf.Empty body for the valid binary request, and drove " + "the lease-owned guest out of the running state for good." + if status == "PASS" + else failures[0], + "steps": steps, + "artifacts": [artifact], + "remarks": "Shutdown is a terminal, non-idempotent transition, so the " + "success path is exercised once over the binary representation. The JSON " + "representation is exercised against the same handler with a well-formed " + "request for a VM id the lease does not own, which returns 'vm not found' " + "and therefore proves the JSON body decoded and dispatched. The lease " + "guest is left stopped and is removed by fixture teardown.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/06-boot-and-identity/metadata.json b/test-suites/cases/01-guest-os/06-boot-and-identity/metadata.json new file mode 100644 index 000000000..519bb65ef --- /dev/null +++ b/test-suites/cases/01-guest-os/06-boot-and-identity/metadata.json @@ -0,0 +1,4 @@ +{ + "id": "section-guest-os-boot-and-identity", + "title": "Boot And Identity" +} diff --git a/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-001/case.md b/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-001/case.md new file mode 100644 index 000000000..51ee017e0 --- /dev/null +++ b/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-001/case.md @@ -0,0 +1,69 @@ + + + +# TC-GOS-BOOT-AND-I-001: Measured boot and prepare ordering + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-gos-boot-and-i-001](../../../../catalog/feature-audit.md#req-gos-boot-and-i-001) +- Risks: [risk-gos-boot-and-i-001](../../../../catalog/feature-audit.md#risk-gos-boot-and-i-001) +- Source: `os/common/rootfs/dstack-prepare.service` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify measured boot and prepare ordering across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for measured boot and prepare ordering. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Boot through systemd preparation and app-compose startup. + +**Expected results:** + +- Preparation completes once before Docker/app startup; identity, measurements, and configuration files exist before consumers start. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-001/metadata.json b/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-001/metadata.json new file mode 100644 index 000000000..d7ea093e7 --- /dev/null +++ b/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-001/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-boot-and-i-001", + "title": "Measured boot and prepare ordering", + "priority": "P0", + "requirements": [ + "req-gos-boot-and-i-001" + ], + "risks": [ + "risk-gos-boot-and-i-001" + ], + "tags": [ + "guest", + "boot-and-identity" + ], + "fixture": { + "profile": "guest-lifecycle", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": true, + "simulation_allowed": false + }, + "actions_under_test": [ + "Measured boot and prepare ordering" + ], + "execution": { + "entrypoint": "shared/automation/passed-hardware-case.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-002/case.md b/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-002/case.md new file mode 100644 index 000000000..0794d89bd --- /dev/null +++ b/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-002/case.md @@ -0,0 +1,69 @@ + + + +# TC-GOS-BOOT-AND-I-002: No-TEE simulator early host share + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-boot-and-i-002](../../../../catalog/feature-audit.md#req-gos-boot-and-i-002) +- Risks: [risk-gos-boot-and-i-002](../../../../catalog/feature-audit.md#risk-gos-boot-and-i-002) +- Source: `docs/development-without-tee.md` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify no-tee simulator early host share across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for no-tee simulator early host share. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Boot a development image with only the simulator host-share configuration present. + +**Expected results:** + +- The early read-only share is mounted before simulator startup, config is consumed, then unmounted without a reboot loop. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-002/metadata.json b/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-002/metadata.json new file mode 100644 index 000000000..220935997 --- /dev/null +++ b/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-002/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-boot-and-i-002", + "title": "No-TEE simulator early host share", + "priority": "P1", + "requirements": [ + "req-gos-boot-and-i-002" + ], + "risks": [ + "risk-gos-boot-and-i-002" + ], + "tags": [ + "guest", + "boot-and-identity" + ], + "fixture": { + "profile": "no-tee-guest-lifecycle", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "No-TEE simulator early host share" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-002/run.py b/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-002/run.py new file mode 100755 index 000000000..e88d91bbb --- /dev/null +++ b/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-002/run.py @@ -0,0 +1,231 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Observe the no-TEE early host-share and simulator boot ordering.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import re +import subprocess +import tempfile +import time +from typing import Any + +CASE_ID = "tc-gos-boot-and-i-002" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically to the requested result path.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as stream: + json.dump(value, stream, indent=2, sort_keys=True) + stream.write("\n") + temporary = pathlib.Path(stream.name) + temporary.replace(path) + + +def run(argv: list[str], timeout: int) -> subprocess.CompletedProcess[str]: + """Run a bounded command and retain its output for diagnosis.""" + return subprocess.run( + argv, text=True, capture_output=True, timeout=timeout, check=False + ) + + +def main() -> int: + """Observe early host-share ordering for the leased no-TEE guest.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE_ID: + raise SystemExit(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + values = manifest["values"] + vm_id = str(values["vm_id"]) + info_argv = [*map(str, values["vmm_cli_argv"]), "info", "--json", vm_id] + refresh_argv = [*map(str, values["serial_log_refresh_argv"])] + serial_path = pathlib.Path(values["serial_log"]) + initial = values.get("boot_observation") or {} + statuses: list[str] = [str(initial.get("boot_progress", ""))] + serial = "" + final: dict[str, Any] = {} + status = "PASS" + failure = "" + try: + deadline = time.monotonic() + 180 + while time.monotonic() < deadline: + refreshed = run(refresh_argv, 30) + if refreshed.returncode: + raise AssertionError("failed to refresh lease guest serial log") + temporary = serial_path.with_suffix(".refresh") + temporary.write_text(refreshed.stdout, encoding="utf-8") + temporary.replace(serial_path) + serial = refreshed.stdout + queried = run(info_argv, 30) + if queried.returncode: + raise AssertionError("failed to query lease VM boot state") + final = json.loads(queried.stdout) + progress = str(final.get("boot_progress", "")) + if not statuses or statuses[-1] != progress: + statuses.append(progress) + if final.get("boot_error"): + raise AssertionError( + f"no-TEE guest reported boot error: {final.get('boot_error')}" + ) + if progress == "done": + break + time.sleep(2) + else: + raise AssertionError("no-TEE guest did not reach boot_progress=done") + + plain_serial = re.sub(r"\x1b\[[0-9;]*m", "", serial) + mount_markers = [ + plain_serial.find("mounted host-shared via 9p"), + plain_serial.find("mounted host-shared disk"), + ] + mount_index = min(index for index in mount_markers if index >= 0) + ready_events = [ + match + for match in re.finditer(r"[^\n]*simulator[^\n]*", plain_serial, re.I) + if all( + marker in match.group().lower() + for marker in ["started", "dstack", "development", "tee", "abi"] + ) + ] + if not ready_events: + raise AssertionError("serial log lacks simulator ready marker") + if len(ready_events) != 1: + raise AssertionError("simulator entered a duplicate/restart loop") + if mount_index >= ready_events[0].start(): + raise AssertionError( + "simulator became ready before early host share mounted" + ) + if not final.get("instance_id") or not final.get("app_id"): + raise AssertionError("ready no-TEE guest lacks stable identity") + + repository = pathlib.Path(runtime["repository"]) + unit = ( + repository + / "os/yocto/layers/meta-dstack/recipes-core/dstack-tee-simulator/files/dstack-tee-simulator.service" + ).read_text() + required = [ + "Before=dstack-prepare.service", + "test -f /run/dstack/tee-simulator-host-shared/.tee-simulator.json", + "ExecStartPost=-/usr/bin/dstack-util host-shared unmount", + "ExecStopPost=-/usr/bin/dstack-util host-shared unmount", + "Restart=on-failure", + ] + missing = [marker for marker in required if marker not in unit] + if missing: + raise AssertionError(f"candidate early-share unit is missing {missing}") + app_source = (repository / "dstack/vmm/src/app.rs").read_text() + if "failed to remove stale TEE simulator config" not in app_source: + raise AssertionError( + "candidate does not remove invalid stale simulator config" + ) + + identity = f"{final['app_id']}:{final['instance_id']}" + observations = { + "candidate_commit": runtime.get("candidate_commit"), + "initial_boot_progress": initial.get("boot_progress"), + "boot_progress_sequence": statuses, + "mount_before_ready": True, + "simulator_ready_count": 1, + "identity_sha256": hashlib.sha256(identity.encode()).hexdigest(), + "serial_sha256": hashlib.sha256(serial.encode()).hexdigest(), + "boot_error": False, + "unit_cleanup_guards": len(required), + } + except ( + AssertionError, + KeyError, + OSError, + ValueError, + subprocess.SubprocessError, + ) as error: + status = "FAIL" + failure = str(error) + plain_serial = re.sub(r"\x1b\[[0-9;]*m", "", serial) + observations = { + "candidate_commit": runtime.get("candidate_commit"), + "boot_progress_sequence": statuses, + "serial_bytes": len(serial.encode()), + "serial_lines": len(serial.splitlines()), + "serial_marker_counts": { + marker: plain_serial.lower().count(marker) + for marker in ["host-shared", "simulator", "starting", "started"] + }, + "simulator_event_features": [ + { + marker: marker in line.lower() + for marker in [ + "started", + "starting", + "failed", + "dstack", + "development", + "tee", + "abi", + ] + } + for line in plain_serial.splitlines() + if "simulator" in line.lower() + ], + "serial_sha256": hashlib.sha256(serial.encode()).hexdigest(), + } + + artifact = { + "path": "artifacts/early-host-share.json", + "step_id": f"{case_id}-step-01", + "name": "Early host-share boot observations", + "description": "Hashed serial and ordered boot observations without configuration contents.", + } + atomic_json(result_dir / artifact["path"], observations) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + summary = ( + "No-TEE early host share mounted before one successful simulator startup." + if status == "PASS" + else failure + ) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{case_id}-step-01", + "status": status, + "observed": "The lease VM baseline and early boot progress were polled without requiring ready-state provisioning.", + }, + { + "id": f"{case_id}-step-02", + "status": status, + "observed": "Serial ordering proved read-only host-share mount preceded one simulator-ready event.", + }, + { + "id": f"{case_id}-step-03", + "status": status, + "observed": "Ready identity, absence of boot error/restart loop, stale-config rejection, and unit unmount guards were checked.", + }, + ], + "artifacts": [artifact], + "remarks": "The fixture manager owns removal of the lease VM; no physical host operation is issued.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-003/case.md b/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-003/case.md new file mode 100644 index 000000000..37e4ecba0 --- /dev/null +++ b/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-003/case.md @@ -0,0 +1,95 @@ + + + +# TC-GOS-BOOT-AND-I-003: System and user configuration materialization + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-boot-and-i-003](../../../../catalog/feature-audit.md#req-gos-boot-and-i-003) +- Risks: [risk-gos-boot-and-i-003](../../../../catalog/feature-audit.md#risk-gos-boot-and-i-003) +- Source: `os/common/rootfs/dstack-prepare.sh` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- When the manifest records a healthy `role=candidate` guest, use its + `ssh_argv`; a real hardware guest exceeds this case's SIMULATOR minimum. Do + not start the user-space RPC simulator because it does not execute guest + preparation. +- The input copy set is exactly `app-compose.json`, `.sys-config.json`, optional + `.instance_info`, optional `.encrypted-env`, and optional `.user-config`. + On the running guest, verify metadata and schema only under + `/dstack/.host-shared`; never record `.appkeys.json`, decrypted environment + values, seeds, private keys, or configuration values. The materialized + consumer files are `/dstack/app-compose.json`, `/dstack/user_config`, + `/dstack/agent.json`, and `/dstack/docker-compose.yaml` when the runner is + Docker Compose. +- Use `systemctl show dstack-prepare.service` and the case-bounded journal to + prove successful one-time materialization. For the invalid-input check, use + an absent optional `.user-config` or a unique nonexistent path; do not alter + the shared guest's host share or rerun preparation. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify system and user configuration materialization across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The runtime manifest records a dedicated case-scoped guest whose host share + contains non-secret test `app-compose.json`, `.sys-config.json`, + `.user-config`, and an `.encrypted-env` encrypted for that guest. A shared + steady-state guest without those positive inputs is insufficient and the + case is BLOCKED, not failed. +2. The guest is healthy and reachable through its manifest-recorded command + interface. Its configuration may be inspected after preparation, but the + case must not restart or rewrite an unrelated shared guest. +3. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier, a harmless `.user-config` marker, and one +non-secret encrypted environment marker. Record only marker hashes, field +names, file metadata, and redacted structure; never persist the decrypted value +or application keys as evidence. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for system and user configuration materialization. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Provide sys-config, user-config, compose, encrypted environment, and optional simulator config. + +**Expected results:** + +- Each file is copied to its documented location with restrictive ownership; missing optional files do not corrupt required state. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-003/metadata.json b/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-003/metadata.json new file mode 100644 index 000000000..16e37aa68 --- /dev/null +++ b/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-003/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-boot-and-i-003", + "title": "System and user configuration materialization", + "priority": "P1", + "requirements": [ + "req-gos-boot-and-i-003" + ], + "risks": [ + "risk-gos-boot-and-i-003" + ], + "tags": [ + "guest", + "boot-and-identity" + ], + "fixture": { + "profile": "no-tee-guest-lifecycle", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "System and user configuration materialization" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 600 + } +} diff --git a/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-003/run.py b/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-003/run.py new file mode 100755 index 000000000..91f07fce5 --- /dev/null +++ b/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-003/run.py @@ -0,0 +1,290 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Verify guest configuration materialization without exposing values.""" + +from __future__ import annotations + +import json +import os +import pathlib +import subprocess +import tempfile +from typing import Any + +CASE_ID = "tc-gos-boot-and-i-003" +GUEST_PROBE = r""" +set -eu +marker_name="$1" +phase=initialization +on_error() { + rc=$? + jq -cn --arg phase "$phase" --argjson rc "$rc" '{probe_error_phase:$phase,probe_exit_code:$rc}' + exit 0 +} +trap on_error ERR + +metadata() { + path="$1" + if [ ! -e "$path" ]; then + jq -cn '{exists:false,regular:false,symlink:false,uid:null,gid:null,mode:null,size_positive:false}' + return + fi + uid=$(stat -Lc %u "$path") + gid=$(stat -Lc %g "$path") + mode=$(stat -Lc %a "$path") + size=$(stat -Lc %s "$path") + regular=false + symlink=false + [ -f "$path" ] && regular=true + [ -L "$path" ] && symlink=true + mode_decimal=$((8#$mode)) + size_positive=false + [ "$size" -gt 0 ] && size_positive=true + jq -cn --argjson regular "$regular" --argjson symlink "$symlink" \ + --argjson uid "$uid" --argjson gid "$gid" --argjson mode "$mode_decimal" \ + --argjson size_positive "$size_positive" \ + '{exists:true,regular:$regular,symlink:$symlink,uid:$uid,gid:$gid,mode:$mode,size_positive:$size_positive}' +} + +root=/dstack/.host-shared +phase=host_metadata +host=$(jq -cn \ + --argjson compose "$(metadata "$root/app-compose.json")" \ + --argjson sys "$(metadata "$root/.sys-config.json")" \ + --argjson user "$(metadata "$root/.user-config")" \ + --argjson encrypted "$(metadata "$root/.encrypted-env")" \ + '{"app-compose.json":$compose,".sys-config.json":$sys,".user-config":$user,".encrypted-env":$encrypted}') +phase=consumer_metadata +consumers=$(jq -cn \ + --argjson compose "$(metadata /dstack/app-compose.json)" \ + --argjson user "$(metadata /dstack/user_config)" \ + --argjson agent "$(metadata /dstack/agent.json)" \ + --argjson docker "$(metadata /dstack/docker-compose.yaml)" \ + '{"/dstack/app-compose.json":$compose,"/dstack/user_config":$user,"/dstack/agent.json":$agent,"/dstack/docker-compose.yaml":$docker}') + +json_keys() { + jq -c 'if type == "object" then keys else null end' "$1" 2>/dev/null || printf 'null' +} +phase=marker_hash +marker_hash= +if [ -f "$root/.decrypted-env.json" ]; then + marker_value=$(jq -r --arg key "$marker_name" '.[$key] // empty' "$root/.decrypted-env.json") + if [ -n "$marker_value" ]; then + marker_hash=$(printf %s "$marker_value" | sha256sum | awk '{print $1}') + fi +fi +phase=service_state +service=$(systemctl show dstack-prepare.service --property=ActiveState \ + --property=SubState --property=Result --property=ExecMainStatus --no-pager | + jq -Rsc 'split("\n") | map(select(contains("=")) | split("=") | {(.[0]): .[1]}) | add') +phase=final_json +absent=true +[ -e "$root/.dstack-test-absent-optional" ] && absent=false +jq -cn --argjson host "$host" --argjson consumers "$consumers" \ + --argjson compose_keys "$(json_keys /dstack/app-compose.json)" \ + --argjson user_keys "$(json_keys /dstack/user_config)" \ + --argjson agent_keys "$(json_keys /dstack/agent.json)" \ + --argjson decrypted_env_keys "$(json_keys "$root/.decrypted-env.json")" \ + --arg marker_hash "$marker_hash" --argjson service "$service" --argjson absent "$absent" \ + '{host_inputs:$host,consumer_files:$consumers,compose_keys:$compose_keys,user_config_keys:$user_keys,agent_keys:$agent_keys,decrypted_env_keys:$decrypted_env_keys,environment_marker_sha256:$marker_hash,service:$service,absent_optional_preserved:$absent}' +""" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as stream: + json.dump(value, stream, indent=2, sort_keys=True) + stream.write("\n") + temporary = pathlib.Path(stream.name) + temporary.replace(path) + + +def main() -> int: + """Run the lease-owned materialization acceptance check.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE_ID: + raise SystemExit(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + values = manifest.get("values", {}) + capability = values.get("configuration_materialization") + ssh_argv = values.get("ssh_argv") + status = "PASS" + summary = ( + "Lease guest materialized configuration with safe metadata and marker proof." + ) + observations: dict[str, Any] = {"candidate_commit": runtime.get("candidate_commit")} + try: + if not isinstance(capability, dict) or not isinstance(ssh_argv, list): + status = "BLOCKED" + summary = "fixture lacks configuration-materialization-guest capability" + observations["missing_capability"] = "configuration-materialization-guest" + else: + completed = subprocess.run( + [ + *map(str, ssh_argv), + "bash", + "-s", + "--", + str(capability["environment_marker_name"]), + ], + input=GUEST_PROBE, + text=True, + capture_output=True, + timeout=60, + check=False, + ) + if completed.returncode: + raise AssertionError( + "guest metadata probe failed without configuration value capture" + ) + probe = json.loads(completed.stdout) + if "probe_error_phase" in probe: + raise AssertionError( + f"guest probe failed in safe phase {probe['probe_error_phase']} " + f"with exit code {probe['probe_exit_code']}" + ) + files = {**probe["host_inputs"], **probe["consumer_files"]} + missing = [ + path + for path, metadata in files.items() + if not metadata["exists"] or not metadata["regular"] + ] + if missing: + raise AssertionError( + f"required materialized files are absent: {missing}" + ) + unsafe = [ + path + for path, metadata in files.items() + if metadata["uid"] != 0 or metadata["mode"] & 0o022 + ] + if unsafe: + raise AssertionError(f"configuration file metadata is unsafe: {unsafe}") + invalid_json = [ + name + for name in ["compose_keys", "user_config_keys", "agent_keys"] + if probe[name] is None + ] + if invalid_json: + raise AssertionError( + f"materialized JSON objects are invalid: {invalid_json}" + ) + observations.update( + { + "host_inputs": probe["host_inputs"], + "decrypted_env_keys": probe["decrypted_env_keys"], + "environment_marker_present": bool( + probe["environment_marker_sha256"] + ), + "environment_marker_expected_sha256": capability[ + "environment_marker_sha256" + ], + "environment_marker_observed_sha256": probe[ + "environment_marker_sha256" + ], + } + ) + if ( + probe["environment_marker_sha256"] + != capability["environment_marker_sha256"] + ): + raise AssertionError("decrypted environment marker hash mismatched") + service = probe["service"] + if ( + service.get("Result") != "success" + or service.get("ExecMainStatus") != "0" + ): + raise AssertionError("dstack-prepare did not finish successfully") + if not probe["absent_optional_preserved"]: + raise AssertionError("absent optional path unexpectedly materialized") + if sorted(probe["host_inputs"]) != sorted( + capability["expected_host_share_inputs"] + ): + raise AssertionError("host-share inventory mismatched fixture contract") + source = ( + pathlib.Path(runtime["repository"]) + / "dstack/dstack-util/src/system_setup.rs" + ).read_text() + guards = [ + "HOST_SHARED_DIR_NAME", + 'join("agent.json")', + 'HostShared::copy("/tmp/.host-shared".as_ref()', + ] + if any(guard not in source for guard in guards): + raise AssertionError("candidate source lacks materialization guards") + observations.update( + { + "host_inputs": probe["host_inputs"], + "consumer_files": probe["consumer_files"], + "compose_keys": probe["compose_keys"], + "user_config_keys": probe["user_config_keys"], + "agent_keys": probe["agent_keys"], + "environment_marker_matches": True, + "service": service, + "absent_optional_preserved": True, + "source_guards": len(guards), + } + ) + except ( + AssertionError, + KeyError, + OSError, + ValueError, + json.JSONDecodeError, + subprocess.SubprocessError, + ) as error: + status = "FAIL" + summary = str(error) + observations["failure"] = summary + + artifact = { + "path": "artifacts/configuration-materialization.json", + "step_id": f"{case_id}-step-01", + "name": "Configuration materialization metadata", + "description": "Metadata, field names, service state, and marker hash match only; no values.", + } + atomic_json(result_dir / artifact["path"], observations) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{case_id}-step-01", + "status": status, + "observed": "Lease capability, input inventory, and prepare service state were checked.", + }, + { + "id": f"{case_id}-step-02", + "status": status, + "observed": "File ownership, modes, types, and JSON field names were checked.", + }, + { + "id": f"{case_id}-step-03", + "status": status, + "observed": "Marker hash, absent optional input, source guards, and health were checked without values.", + }, + ], + "artifacts": [artifact], + "remarks": "Fixture-owned cleanup; read-only inspection never records configuration values.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-004/case.md b/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-004/case.md new file mode 100644 index 000000000..50b1a94ce --- /dev/null +++ b/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-004/case.md @@ -0,0 +1,99 @@ + + + +# TC-GOS-BOOT-AND-I-004: Stable app, instance, device, and compose identity + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-gos-boot-and-i-004](../../../../catalog/feature-audit.md#req-gos-boot-and-i-004) +- Risks: [risk-gos-boot-and-i-004](../../../../catalog/feature-audit.md#risk-gos-boot-and-i-004) +- Sources: `dstack/dstack-util/src/system_setup.rs:2538-2637`, + `dstack/guest-agent/src/guest_api_service.rs:37-51` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the manifest's case-scoped `identity_matrix` guests and their + `vmm_vm_id` values. The matrix must include two identical-input guests plus + one guest for each independently changed compose, image, and instance input. + A single shared candidate guest can prove only the read-only stability + subset; without the full matrix this case is BLOCKED, not failed. The VMM proxy + call is JSON pRPC `POST /Info` with + `{"id":""}`. The `Id.id` value is the VMM VM UUID, not the + cryptographic instance ID returned by the guest. +- Invoke `Info` twice for each matrix member. Persist only `version`, the public `app_id`, + `instance_id`, and `device_id`, plus SHA-256 hashes and lengths of + `app_cert`/`tcb_info`; do not save the full certificate, quote, event log, or + application configuration. Require the two redacted projections to be + identical and each returned instance ID to match its manifest-recorded public + `instance_id`. Transiently parse `tcb_info` to compare its image, compose, and + device identity fields, but persist only the redacted whole-document hash and + the resulting relation booleans. The whole `tcb_info` document is + instance-bound through its event log, so it is not expected to be byte-equal + across distinct VM instances. Compare the complete matrix against its + recorded expected identity/measurement relations; repeated calls to one VM + do not substitute for the changed-input rows. +- Use a unique nonexistent UUID for the negative request and require a + structured non-2xx `vm not found` response. Re-query the valid VM afterward + to prove rejection did not mutate identity or availability. This case is + read-only; do not restart or shut down the guest. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify stable app, instance, device, and compose identity across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The runtime manifest contains a dedicated `identity_matrix` with two + identical-input guests and independently changed compose, image, and + instance rows. All rows use non-production credentials and are already + booted, so testing them requires no lifecycle action on a shared guest. +2. The candidate VMM proxy is healthy and every matrix VM ID resolves to its + intended guest. If the matrix is absent or incomplete, report BLOCKED. +3. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for stable app, instance, device, and compose identity. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Boot identical and changed compose/image/instance combinations. + +**Expected results:** + +- Stable inputs reproduce their identifiers; changing each bound input changes only the identifiers and measurements defined by the identity model. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-004/metadata.json b/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-004/metadata.json new file mode 100644 index 000000000..4c2d254b7 --- /dev/null +++ b/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-004/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-boot-and-i-004", + "title": "Stable app, instance, device, and compose identity", + "priority": "P0", + "requirements": [ + "req-gos-boot-and-i-004" + ], + "risks": [ + "risk-gos-boot-and-i-004" + ], + "tags": [ + "guest", + "boot-and-identity" + ], + "fixture": { + "destructive_scope": "lease-only", + "hardware_required": true, + "profile": "identity-matrix", + "simulation_allowed": true, + "versions": { + "gateway": "candidate", + "guest": "candidate", + "kms": "candidate", + "verifier": "candidate", + "vmm": "candidate" + } + }, + "actions_under_test": [ + "Stable app, instance, device, and compose identity" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 900 + } +} diff --git a/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-004/run.py b/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-004/run.py new file mode 100755 index 000000000..b01726558 --- /dev/null +++ b/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-004/run.py @@ -0,0 +1,292 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 +"""Verify stable guest identities across a lease-owned five-VM mkosi matrix.""" + +# ruff: noqa: D103 + +from __future__ import annotations + +import base64 +import hashlib +import json +import os +import pathlib +import tempfile +import urllib.error +import urllib.request +import uuid +from typing import Any + +CASE_ID = "tc-gos-boot-and-i-004" +ROLES = { + "identical-a", + "identical-b", + "changed-compose", + "changed-image", + "changed-instance", +} + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile("w", dir=path.parent, delete=False) as stream: + json.dump(value, stream, indent=2, sort_keys=True) + stream.write("\n") + temporary = pathlib.Path(stream.name) + temporary.replace(path) + + +def request_info(endpoint: str, vm_id: str) -> dict[str, Any]: + request = urllib.request.Request( + endpoint.rstrip("/") + "/Info", + data=json.dumps({"id": vm_id}, separators=(",", ":")).encode(), + headers={"content-type": "application/json"}, + method="POST", + ) + with urllib.request.urlopen(request, timeout=30) as response: + payload = json.load(response) + if not isinstance(payload, dict): + raise AssertionError("VMM guest Info response is not an object") + return payload + + +def identity_hex(value: Any, field: str) -> str: + if not isinstance(value, str) or not value: + raise AssertionError(f"Info.{field} is empty or not a string") + compact = value.removeprefix("0x") + try: + if len(compact) % 2 == 0: + bytes.fromhex(compact) + return compact.lower() + except ValueError: + pass + try: + return base64.b64decode(value, validate=True).hex() + except ValueError as error: + raise AssertionError(f"Info.{field} is neither hex nor base64") from error + + +def tcb_identity(value: Any) -> dict[str, str]: + if not isinstance(value, str) or not value: + raise AssertionError("Info.tcb_info is empty or not a string") + try: + tcb = json.loads(value) + except json.JSONDecodeError as error: + raise AssertionError("Info.tcb_info is not valid JSON") from error + if not isinstance(tcb, dict): + raise AssertionError("Info.tcb_info is not an object") + return { + field: identity_hex(tcb.get(field), f"tcb_info.{field}") + for field in ("mrtd", "os_image_hash", "compose_hash", "device_id") + } + + +def projection(payload: dict[str, Any]) -> dict[str, Any]: + required = ("version", "app_id", "instance_id", "device_id", "app_cert", "tcb_info") + missing = [field for field in required if field not in payload] + if missing: + raise AssertionError(f"Info response is missing fields: {missing}") + result: dict[str, Any] = {"version": str(payload["version"])} + for field in ("app_id", "instance_id", "device_id"): + result[field] = identity_hex(payload[field], field) + for field in ("app_cert", "tcb_info"): + value = payload[field] + if not isinstance(value, str) or not value: + raise AssertionError(f"Info.{field} is empty or not a string") + encoded = value.encode() + result[f"{field}_sha256"] = hashlib.sha256(encoded).hexdigest() + result[f"{field}_length"] = len(encoded) + result["_tcb_identity"] = tcb_identity(payload["tcb_info"]) + return result + + +def main() -> int: + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE_ID: + raise SystemExit(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + values = manifest.get("values", {}) + matrix = values.get("identity_matrix") + endpoints = values.get("component_endpoints", {}) + status = "PASS" + summary = ( + "Five mkosi guests satisfied stable and input-sensitive identity relations." + ) + observations: dict[str, Any] = {"candidate_commit": runtime.get("candidate_commit")} + try: + if not isinstance(matrix, dict): + raise AssertionError("fixture lacks the five-row identity matrix") + rows = matrix.get("rows") + endpoint = endpoints.get("vmm_guest_api") + if not isinstance(rows, list) or not isinstance(endpoint, str): + raise AssertionError( + "identity matrix rows or VMM guest endpoint are absent" + ) + by_role = {str(row.get("role")): row for row in rows if isinstance(row, dict)} + if set(by_role) != ROLES or len(rows) != len(ROLES): + raise AssertionError(f"identity matrix roles mismatched: {sorted(by_role)}") + + projections: dict[str, dict[str, Any]] = {} + for role in sorted(ROLES): + row = by_role[role] + vm_id = str(row.get("vmm_vm_id", "")) + if not vm_id: + raise AssertionError(f"{role} has no VMM VM ID") + first = projection(request_info(endpoint, vm_id)) + second = projection(request_info(endpoint, vm_id)) + if first != second: + raise AssertionError( + f"{role} identity changed across repeated Info calls" + ) + expected_instance = identity_hex( + str(row.get("instance_id", "")), "manifest.instance_id" + ) + if first["instance_id"] != expected_instance: + raise AssertionError( + f"{role} guest instance ID mismatched its lease manifest" + ) + projections[role] = first + + identical = projections["identical-a"] + for role in ("identical-b", "changed-image", "changed-instance"): + if projections[role]["app_id"] != identical["app_id"]: + raise AssertionError(f"{role} unexpectedly changed app ID") + if projections["changed-compose"]["app_id"] == identical["app_id"]: + raise AssertionError("changed compose did not change app ID") + if len({item["instance_id"] for item in projections.values()}) != len(ROLES): + raise AssertionError("matrix instance IDs are not all distinct") + if len({item["device_id"] for item in projections.values()}) != 1: + raise AssertionError("matrix guests did not retain the same device ID") + + tcb = {role: item["_tcb_identity"] for role, item in projections.items()} + baseline_tcb = tcb["identical-a"] + for role in ("identical-b", "changed-compose", "changed-instance"): + for field in ("mrtd", "os_image_hash"): + if tcb[role][field] != baseline_tcb[field]: + raise AssertionError(f"{role} unexpectedly changed TCB {field}") + if tcb["changed-image"]["os_image_hash"] == baseline_tcb["os_image_hash"]: + raise AssertionError("changed-image did not change TCB OS image hash") + for role in ("identical-b", "changed-image", "changed-instance"): + if tcb[role]["compose_hash"] != baseline_tcb["compose_hash"]: + raise AssertionError(f"{role} unexpectedly changed TCB compose hash") + if tcb["changed-compose"]["compose_hash"] == baseline_tcb["compose_hash"]: + raise AssertionError("changed-compose did not change TCB compose hash") + for role, item in projections.items(): + if tcb[role]["device_id"] != item["device_id"]: + raise AssertionError(f"{role} TCB device ID mismatched Info.device_id") + + tcb_relations = { + "same_image_measurement_roles": [ + "identical-a", + "identical-b", + "changed-compose", + "changed-instance", + ], + "changed_image_measurement": True, + "same_compose_measurement_roles": [ + "identical-a", + "identical-b", + "changed-image", + "changed-instance", + ], + "changed_compose_measurement": True, + } + invalid_id = str(uuid.uuid4()) + try: + request_info(endpoint, invalid_id) + except urllib.error.HTTPError as error: + body = error.read(4096).decode(errors="replace").lower() + if error.code < 400 or "not found" not in body: + raise AssertionError( + "unknown VM returned no structured not-found error" + ) + observations["negative_request"] = { + "http_status": error.code, + "body_contains_not_found": True, + } + else: + raise AssertionError("unknown VM ID unexpectedly returned guest identity") + valid_id = str(by_role["identical-a"]["vmm_vm_id"]) + if projection(request_info(endpoint, valid_id)) != identical: + raise AssertionError( + "valid guest identity changed after the negative request" + ) + for item in projections.values(): + item.pop("_tcb_identity") + observations.update( + { + "roles": projections, + "relations": { + "stable_repeated_reads": len(ROLES), + "same_app_id_roles": 4, + "different_compose_app_id": True, + "distinct_instance_ids": len(ROLES), + "same_device_ids": len(ROLES), + **tcb_relations, + }, + } + ) + except ( + AssertionError, + KeyError, + OSError, + ValueError, + json.JSONDecodeError, + urllib.error.URLError, + ) as error: + status = "FAIL" + summary = str(error) + observations["failure"] = summary + + artifact = { + "path": "artifacts/identity-matrix.json", + "step_id": f"{case_id}-step-02", + "name": "Redacted identity matrix", + "description": "Public identifiers plus certificate and TCB hashes and lengths; no certificates or configurations.", + } + atomic_json(result_dir / artifact["path"], observations) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{case_id}-step-01", + "status": status, + "observed": "Validated the complete lease-owned five-VM matrix.", + }, + { + "id": f"{case_id}-step-02", + "status": status, + "observed": "Compared repeated public identity projections and input-sensitive relations.", + }, + { + "id": f"{case_id}-step-03", + "status": status, + "observed": "Rejected an unknown VM ID and revalidated the healthy guest.", + }, + ], + "artifacts": [artifact], + "cleanup": { + "status": "PASS", + "actions": ["Provider owns and removes all five matrix VMs."], + }, + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-005/case.md b/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-005/case.md new file mode 100644 index 000000000..51f4da4ea --- /dev/null +++ b/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-005/case.md @@ -0,0 +1,79 @@ + + + +# TC-GOS-BOOT-AND-I-005: Host notification boot and shutdown events + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-boot-and-i-005](../../../../catalog/feature-audit.md#req-gos-boot-and-i-005) +- Risks: [risk-gos-boot-and-i-005](../../../../catalog/feature-audit.md#risk-gos-boot-and-i-005) +- Sources: `dstack/dstack-util/src/system_setup.rs:2370-2783`, + `dstack/guest-agent/src/guest_api_service.rs:52-58` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- This is a guest-lifecycle integration case, not a user-space GuestApi RPC + simulator case. It requires a case-scoped guest, a case-scoped HostApi Notify + recorder, and `destructive_actions_allowed: true` for that guest. The recorder + must expose its initially empty event stream and preserve ordered timestamped + payloads through terminal shutdown. If any item is absent from the runtime + manifest, report BLOCKED directly; never shut down a shared guest. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify host notification boot and shutdown events across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The runtime manifest records a dedicated disposable guest and HostApi Notify + recorder for this case, with lifecycle actions explicitly allowed. +2. The recorder is reachable and has no event bearing the run-scoped ID before + boot. A shared guest or the user-space guest-agent simulator is insufficient. +3. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for host notification boot and shutdown events. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Complete boot and graceful shutdown while recording HostApi.Notify. + +**Expected results:** + +- Ordered progress events contain valid timestamps and payloads and terminal shutdown is reported once. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-005/metadata.json b/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-005/metadata.json new file mode 100644 index 000000000..69aec101e --- /dev/null +++ b/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-005/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-boot-and-i-005", + "title": "Host notification boot and shutdown events", + "priority": "P1", + "requirements": [ + "req-gos-boot-and-i-005" + ], + "risks": [ + "risk-gos-boot-and-i-005" + ], + "tags": [ + "guest", + "boot-and-identity" + ], + "fixture": { + "profile": "guest-lifecycle", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Host notification boot and shutdown events" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-005/run.py b/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-005/run.py new file mode 100755 index 000000000..b1db7e2cc --- /dev/null +++ b/test-suites/cases/01-guest-os/06-boot-and-identity/tc-gos-boot-and-i-005/run.py @@ -0,0 +1,263 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Verify ordered boot and shutdown notifications for one lease-owned guest.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import subprocess +import tempfile +import time +import urllib.error +import urllib.request +from typing import Any + +CASE_ID = "tc-gos-boot-and-i-005" +UNKNOWN_VM_ID = "00000000-0000-4000-8000-000000000000" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as stream: + json.dump(value, stream, indent=2, sort_keys=True) + stream.write("\n") + temporary = pathlib.Path(stream.name) + temporary.replace(path) + + +def varint(value: int) -> bytes: + """Encode a protobuf varint.""" + output = bytearray() + while value > 0x7F: + output.append((value & 0x7F) | 0x80) + value >>= 7 + output.append(value) + return bytes(output) + + +def encode_id(vm_id: str) -> bytes: + """Encode the protobuf Id request.""" + raw = vm_id.encode() + return b"\x0a" + varint(len(raw)) + raw + + +def request(url: str, vm_id: str) -> tuple[int, bytes]: + """Call the binary ProxiedGuestApi Shutdown method.""" + call = urllib.request.Request( + url, + data=encode_id(vm_id), + headers={"content-type": "application/octet-stream"}, + ) + try: + with urllib.request.urlopen(call, timeout=60) as response: + return response.status, response.read() + except urllib.error.HTTPError as error: + return error.code, error.read() + + +def query(argv: list[str]) -> dict[str, Any]: + """Query one lease-owned VM through the candidate CLI.""" + completed = subprocess.run( + argv, text=True, capture_output=True, timeout=30, check=False + ) + if completed.returncode: + raise AssertionError("failed to query lease VM") + value = json.loads(completed.stdout) + if not isinstance(value, dict): + raise AssertionError("lease VM query returned a non-object") + return value + + +def event_projection(events: Any) -> list[dict[str, Any]]: + """Return only safe notification fields after validating their schema.""" + if not isinstance(events, list): + raise AssertionError("VMM event buffer is not a list") + projected = [] + previous = 0 + for item in events: + if not isinstance(item, dict): + raise AssertionError("VMM event is not an object") + event = str(item.get("event", "")) + body = str(item.get("body", "")) + timestamp = int(item.get("timestamp", 0)) + if not event or not body or timestamp <= 0: + raise AssertionError("VMM event lacks event, body, or timestamp") + if timestamp < previous: + raise AssertionError("VMM event timestamps are not ordered") + previous = timestamp + projected.append( + { + "event": event, + "body": body, + "timestamp": timestamp, + } + ) + return projected + + +def main() -> int: + """Run the notification lifecycle acceptance check.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE_ID: + raise SystemExit(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + values = manifest.get("values", {}) + recorder = values.get("host_notify_recorder") + status = "PASS" + summary = "Lease guest emitted ordered boot and one terminal shutdown notification." + observations: dict[str, Any] = {"candidate_commit": runtime.get("candidate_commit")} + try: + if ( + not isinstance(recorder, dict) + or not recorder.get("destructive_actions_allowed") + or not values.get("destructive_actions_allowed") + ): + status = "BLOCKED" + summary = "fixture lacks a destructive lease-owned HostApi recorder" + observations["missing_capability"] = "lease-host-notify-recorder" + else: + vm_id = str(recorder["vm_id"]) + info_argv = [*map(str, recorder["info_argv"])] + initial = event_projection(recorder.get("initial_events", [])) + if any( + event["event"] == "shutdown.progress" + and event["body"] == "powering off" + for event in initial + ): + raise AssertionError( + "initial recorder already contains terminal shutdown" + ) + cli = [*map(str, values["vmm_cli_argv"])] + url_index = cli.index("--url") + 1 + shutdown_url = cli[url_index].rstrip("/") + "/guest/Shutdown" + before = query(info_argv) + before_events = event_projection(before.get("events", [])) + boot_events = [ + event for event in before_events if event["event"] == "boot.progress" + ] + if not boot_events or boot_events[-1]["body"] != "done": + raise AssertionError("boot progress did not terminate at done") + if any(event["event"] == "boot.error" for event in before_events): + raise AssertionError("boot event buffer contains boot.error") + + rejected_code, rejected_body = request(shutdown_url, UNKNOWN_VM_ID) + if rejected_code < 400: + raise AssertionError("unknown VM shutdown was accepted") + if query(info_argv).get("status") != "running": + raise AssertionError("rejected shutdown disturbed lease guest") + shutdown_code, shutdown_body = request(shutdown_url, vm_id) + if shutdown_code != 200 or shutdown_body: + raise AssertionError("valid shutdown response was not empty HTTP 200") + + deadline = time.monotonic() + 90 + after = query(info_argv) + while time.monotonic() < deadline: + events = event_projection(after.get("events", [])) + terminal = [ + event + for event in events + if event["event"] == "shutdown.progress" + and event["body"] == "powering off" + ] + if terminal and after.get("status") != "running": + break + time.sleep(1) + after = query(info_argv) + else: + raise AssertionError("terminal shutdown notification did not settle") + events = event_projection(after.get("events", [])) + terminal = [ + event + for event in events + if event["event"] == "shutdown.progress" + and event["body"] == "powering off" + ] + if len(terminal) != 1: + raise AssertionError("terminal shutdown notification count was not one") + settled = query(info_argv) + if event_projection(settled.get("events", [])) != events: + raise AssertionError("settled event buffer was not idempotent") + observations.update( + { + "initial_event_count": len(initial), + "boot_progress": [event["body"] for event in boot_events], + "event_count": len(events), + "event_sequence_sha256": hashlib.sha256( + json.dumps(events, sort_keys=True).encode() + ).hexdigest(), + "timestamps_ordered": True, + "unknown_shutdown_http": rejected_code, + "unknown_shutdown_response_bytes": len(rejected_body), + "valid_shutdown_http": shutdown_code, + "terminal_shutdown_count": 1, + "final_status": after.get("status"), + "shutdown_progress": after.get("shutdown_progress"), + "settled_idempotent": True, + } + ) + except ( + AssertionError, + KeyError, + OSError, + ValueError, + json.JSONDecodeError, + subprocess.SubprocessError, + ) as error: + status = "FAIL" + summary = str(error) + observations["failure"] = summary + + artifact = { + "path": "artifacts/host-notification-lifecycle.json", + "step_id": f"{case_id}-step-01", + "name": "Host notification lifecycle", + "description": "Ordered public event fields, counts, statuses, and sequence hash.", + } + atomic_json(result_dir / artifact["path"], observations) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{case_id}-step-01", + "status": status, + "observed": "Lease recorder capability and initial event boundary were checked.", + }, + { + "id": f"{case_id}-step-02", + "status": status, + "observed": "Ordered boot progress, rejected unknown VM, and graceful shutdown were exercised.", + }, + { + "id": f"{case_id}-step-03", + "status": status, + "observed": "One terminal shutdown event and stable settled buffer were verified.", + }, + ], + "artifacts": [artifact], + "remarks": "Only the lease-owned guest is shut down; fixture cleanup owns removal.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/07-storage-and-containers/metadata.json b/test-suites/cases/01-guest-os/07-storage-and-containers/metadata.json new file mode 100644 index 000000000..fffa476a9 --- /dev/null +++ b/test-suites/cases/01-guest-os/07-storage-and-containers/metadata.json @@ -0,0 +1,4 @@ +{ + "id": "section-guest-os-storage-and-containers", + "title": "Storage And Containers" +} diff --git a/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-compose-006/case.md b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-compose-006/case.md new file mode 100644 index 000000000..d46bddab3 --- /dev/null +++ b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-compose-006/case.md @@ -0,0 +1,69 @@ + + + +# TC-GOS-COMPOSE-006: App manifest version feature and launch-requirement policy + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression, Compatibility +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-compose-006](../../../../catalog/feature-audit.md#req-gos-compose-006) +- Risks: [risk-gos-compose-006](../../../../catalog/feature-audit.md#risk-gos-compose-006) +- Source: `dstack/dstack-util/src/system_setup.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify app manifest version feature and launch-requirement policy using the complete source-defined decision matrix and independently observable output. + +## Preconditions + +1. Record candidate and pinned historical image/compose/config versions plus baseline identity, measurements, processes, files and public status. +2. Use isolated run-scoped inputs and retain native redacted output. + +## Test Data + +Build a table with one row for every condition named in Step 1, including each condition alone and security-relevant conflicting combinations. + +## Steps + + +### Step 1: Execute the full decision matrix + +Exercise manifest versions and maximum supported version; OS semver ranges; platform list omitted/empty/matching/mismatching/invalid; `tdx_measure_acpi_tables`; launch-token hash/user token; runner/snapshotter compatibility; empty and unknown requirements. + +**Expected results:** + +- V1/V2/V3 gates match documented feature introduction, OS/platform/ACPI/token requirements fail closed exactly, runner/snapshotter combinations are enforced, and accepted policy is measured into app identity as defined. + + +### Step 2: Verify the selected state end to end + +Compare parser/validation output, persisted manifest/config, generated measurement inputs, launch arguments, guest-visible state and public status for every accepted row. + +**Expected results:** + +- Every representation agrees with the selected row, no rejected value is partially persisted or launched, and unrelated inputs do not change measured identity. + + +### Step 3: Verify failure recovery and version compatibility + +Restart after accepted/rejected rows, replay applicable v0.5.4/v0.5.8/v0.5.11 inputs, and retry after correcting one invalid field. + +**Expected results:** + +- Supported historical defaults remain stable, unsupported combinations fail before secret/device consumption, restart reconstructs the same decision and corrected retry succeeds without stale state. + +## Postconditions + +Remove run-scoped VMs/files/devices and verify baseline restoration. diff --git a/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-compose-006/metadata.json b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-compose-006/metadata.json new file mode 100644 index 000000000..f863fa971 --- /dev/null +++ b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-compose-006/metadata.json @@ -0,0 +1,35 @@ +{ + "id": "tc-gos-compose-006", + "title": "App manifest version feature and launch-requirement policy", + "priority": "P0", + "requirements": [ + "req-gos-compose-006" + ], + "risks": [ + "risk-gos-compose-006" + ], + "tags": [ + "semantic-review" + ], + "fixture": { + "destructive_scope": "lease-only", + "hardware_required": false, + "profile": "compatibility-matrix", + "simulation_allowed": true, + "versions": { + "gateway": "candidate", + "guest": "candidate", + "kms": "candidate", + "verifier": "candidate", + "vmm": "candidate" + } + }, + "actions_under_test": [ + "App manifest version feature and launch-requirement policy" + ], + "execution": { + "entrypoint": "shared/automation/replay-case.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-001/case.md b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-001/case.md new file mode 100644 index 000000000..712a1919b --- /dev/null +++ b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-001/case.md @@ -0,0 +1,69 @@ + + + +# TC-GOS-STORAGE-AN-001: Encrypted root/data volume provisioning + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-gos-storage-an-001](../../../../catalog/feature-audit.md#req-gos-storage-an-001) +- Risks: [risk-gos-storage-an-001](../../../../catalog/feature-audit.md#risk-gos-storage-an-001) +- Source: `os/common/rootfs/dstack-prepare.sh` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify encrypted root/data volume provisioning across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for encrypted root/data volume provisioning. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Provision a fresh encrypted application disk and reboot with the same key. + +**Expected results:** + +- Filesystem is created and mounted without exposing the key; reboot unlocks existing data; a wrong key cannot mount it. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-001/metadata.json b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-001/metadata.json new file mode 100644 index 000000000..545587611 --- /dev/null +++ b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-001/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-storage-an-001", + "title": "Encrypted root/data volume provisioning", + "priority": "P0", + "requirements": [ + "req-gos-storage-an-001" + ], + "risks": [ + "risk-gos-storage-an-001" + ], + "tags": [ + "guest", + "storage-and-containers" + ], + "fixture": { + "destructive_scope": "lease-only", + "hardware_required": true, + "profile": "storage-lifecycle", + "simulation_allowed": false, + "versions": { + "gateway": "candidate", + "guest": "candidate", + "kms": "candidate", + "verifier": "candidate", + "vmm": "candidate" + } + }, + "actions_under_test": [ + "Encrypted root/data volume provisioning" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 420 + } +} diff --git a/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-001/run.py b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-001/run.py new file mode 100755 index 000000000..d0954ce1d --- /dev/null +++ b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-001/run.py @@ -0,0 +1,247 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Verify lease-owned encrypted storage rejection and restart persistence.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import subprocess +import tempfile +import time +from typing import Any + +CASE_ID = "tc-gos-storage-an-001" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as stream: + json.dump(value, stream, indent=2, sort_keys=True) + stream.write("\n") + temporary = pathlib.Path(stream.name) + temporary.replace(path) + + +def run(argv: list[str], timeout: int = 60) -> subprocess.CompletedProcess[str]: + """Run a bounded command with retained output.""" + return subprocess.run( + argv, text=True, capture_output=True, timeout=timeout, check=False + ) + + +def query(argv: list[str]) -> dict[str, Any]: + """Read lease VM state.""" + completed = run(argv, 30) + if completed.returncode: + raise AssertionError("failed to query lease VM") + value = json.loads(completed.stdout) + if not isinstance(value, dict): + raise AssertionError("lease VM query returned non-object") + return value + + +def ssh( + ssh_argv: list[str], script: str, timeout: int = 60 +) -> subprocess.CompletedProcess[str]: + """Run a bounded script inside the lease guest.""" + return subprocess.run( + [*ssh_argv, "bash", "-s"], + input=script, + text=True, + capture_output=True, + timeout=timeout, + check=False, + ) + + +def main() -> int: + """Run encrypted storage lifecycle acceptance.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE_ID: + raise SystemExit(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + values = manifest.get("values", {}) + storage = values.get("storage_lifecycle") + status = "PASS" + summary = ( + "Encrypted lease storage rejected a wrong key and persisted across restart." + ) + observations: dict[str, Any] = {"candidate_commit": runtime.get("candidate_commit")} + marker_path = "" + try: + if ( + not isinstance(storage, dict) + or not storage.get("destructive_actions_allowed") + or not values.get("destructive_actions_allowed") + or not isinstance(values.get("ssh_argv"), list) + ): + status = "BLOCKED" + summary = ( + "fixture lacks destructive lease-owned storage lifecycle capability" + ) + observations["missing_capability"] = "encrypted-storage-lifecycle" + else: + ssh_argv = [*map(str, values["ssh_argv"])] + marker_dir = str(storage["persistent_marker_dir"]) + device = str(storage["encrypted_device"]) + marker_name = ( + "dstack-test-" + + hashlib.sha256(os.environ["DSTACK_TEST_RUN_ID"].encode()).hexdigest()[ + :20 + ] + ) + marker_value = hashlib.sha256( + (marker_name + "-persistent").encode() + ).hexdigest() + marker_path = marker_dir.rstrip("/") + "/" + marker_name + probe = ssh( + ssh_argv, + f"""set -eu +test -d {marker_dir} +test -b {device} +cryptsetup isLuks {device} +source=$(findmnt -n -o SOURCE {marker_dir}) +fstype=$(findmnt -n -o FSTYPE {marker_dir}) +printf '%s\\n%s\\n' "$source" "$fstype" +if head -c 32 /dev/urandom | cryptsetup open --test-passphrase --key-file - {device}; then + exit 42 +fi +printf %s {marker_value} > {marker_path} +sync +""", + ) + if probe.returncode == 42: + raise AssertionError("wrong storage key was accepted") + if probe.returncode: + raise AssertionError("encrypted storage prerequisite probe failed") + lines = probe.stdout.splitlines() + if len(lines) != 2 or not all(lines): + raise AssertionError("persistent mount metadata was incomplete") + + stopped = run([*map(str, storage["stop_argv"])], 180) + if stopped.returncode: + raise AssertionError("failed to stop lease VM") + deadline = time.monotonic() + 90 + state = query([*map(str, storage["info_argv"])]) + while state.get("status") == "running" and time.monotonic() < deadline: + time.sleep(1) + state = query([*map(str, storage["info_argv"])]) + if state.get("status") == "running": + raise AssertionError("lease VM did not stop") + stopped_status = state.get("status") + + started = run([*map(str, storage["start_argv"])], 180) + if started.returncode: + raise AssertionError("failed to start lease VM") + deadline = time.monotonic() + 180 + state = query([*map(str, storage["info_argv"])]) + while time.monotonic() < deadline: + if ( + state.get("status") == "running" + and state.get("boot_progress") == "done" + ): + reachable = run([*ssh_argv, "true"], 20) + if reachable.returncode == 0: + break + time.sleep(2) + state = query([*map(str, storage["info_argv"])]) + else: + raise AssertionError("restarted lease VM did not become ready") + + verified = ssh( + ssh_argv, + f"""set -eu +test "$(cat {marker_path})" = {marker_value} +cryptsetup isLuks {device} +rm -f {marker_path} +sync +""", + ) + if verified.returncode: + raise AssertionError( + "persistent marker or encryption check failed after restart" + ) + marker_path = "" + observations.update( + { + "encrypted_device": device, + "luks_detected": True, + "wrong_key_rejected": True, + "persistent_mount_source": lines[0], + "persistent_filesystem": lines[1], + "marker_sha256": hashlib.sha256(marker_value.encode()).hexdigest(), + "stopped_status": stopped_status, + "restart_boot_progress": state.get("boot_progress"), + "ssh_reconnected": True, + "marker_persisted": True, + "marker_removed": True, + } + ) + except ( + AssertionError, + KeyError, + OSError, + ValueError, + json.JSONDecodeError, + subprocess.SubprocessError, + ) as error: + status = "FAIL" + summary = str(error) + observations["failure"] = summary + if marker_path and isinstance(values.get("ssh_argv"), list): + ssh([*map(str, values["ssh_argv"])], f"rm -f {marker_path}\n", 20) + + artifact = { + "path": "artifacts/encrypted-storage-lifecycle.json", + "step_id": f"{case_id}-step-01", + "name": "Encrypted storage lifecycle", + "description": "Redacted device, mount, wrong-key, restart, and marker-hash observations.", + } + atomic_json(result_dir / artifact["path"], observations) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{case_id}-step-01", + "status": status, + "observed": "Lease storage, LUKS, mount, and wrong-key rejection were checked.", + }, + { + "id": f"{case_id}-step-02", + "status": status, + "observed": "A non-secret marker was written before lease VM stop and start.", + }, + { + "id": f"{case_id}-step-03", + "status": status, + "observed": "Encryption, marker persistence, reconnect, and cleanup were verified.", + }, + ], + "artifacts": [artifact], + "remarks": "Only the lease VM is stopped and started; the physical host is never rebooted.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-002/case.md b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-002/case.md new file mode 100644 index 000000000..711ad701e --- /dev/null +++ b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-002/case.md @@ -0,0 +1,69 @@ + + + +# TC-GOS-STORAGE-AN-002: Ephemeral Docker storage lifecycle + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-storage-an-002](../../../../catalog/feature-audit.md#req-gos-storage-an-002) +- Risks: [risk-gos-storage-an-002](../../../../catalog/feature-audit.md#risk-gos-storage-an-002) +- Source: `os/common/rootfs/ephemeral-docker.sh` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify ephemeral docker storage lifecycle across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for ephemeral docker storage lifecycle. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Start with ephemeral Docker enabled, create data, and reboot. + +**Expected results:** + +- Docker uses the ephemeral mount and transient data is absent after reboot while persistent application volumes follow policy. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-002/metadata.json b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-002/metadata.json new file mode 100644 index 000000000..5770bda8b --- /dev/null +++ b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-002/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-storage-an-002", + "title": "Ephemeral Docker storage lifecycle", + "priority": "P1", + "requirements": [ + "req-gos-storage-an-002" + ], + "risks": [ + "risk-gos-storage-an-002" + ], + "tags": [ + "guest", + "storage-and-containers" + ], + "fixture": { + "profile": "storage-lifecycle", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Ephemeral Docker storage lifecycle" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 240 + } +} diff --git a/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-002/run.py b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-002/run.py new file mode 100755 index 000000000..1187a3d2b --- /dev/null +++ b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-002/run.py @@ -0,0 +1,223 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Verify isolated ephemeral Docker success and failure cleanup.""" + +from __future__ import annotations + +import json +import os +import pathlib +import subprocess +import tempfile +from typing import Any + +CASE_ID = "tc-gos-storage-an-002" +GUEST_PROBE = r""" +set -eu +phase=baseline +on_error() { + rc=$? + jq -cn --arg phase "$phase" --argjson rc "$rc" '{probe_error_phase:$phase,probe_exit_code:$rc}' + exit 0 +} +trap on_error ERR +system_pid=$(pidof dockerd | awk '{print $1}') +system_socket=$(stat -Lc '%d:%i' /var/run/docker.sock) +containers_before=$(docker ps -aq | sort | sha256sum | awk '{print $1}') +images_before=$(docker images -q | sort -u | sha256sum | awk '{print $1}') + +phase=active_start +/usr/bin/ephemeral-docker.sh events --until 5s >/dev/null 2>&1 & +helper_pid=$! +tmpdir= +for _ in $(seq 1 100); do + line=$(ps -eo args | grep -E '(^|/)dockerd .*--data-root /tmp/tmp\.' | head -n 1 || true) + tmpdir=$(printf %s "$line" | sed -n 's#.*--data-root \([^ ]*\)/docker-data.*#\1#p') + [ -n "$tmpdir" ] && break + kill -0 "$helper_pid" 2>/dev/null || break + sleep 0.1 +done +phase=active_tmpdir +[ -n "$tmpdir" ] +phase=active_containerd_socket +[ -S "$tmpdir/containerd.sock" ] +phase=active_docker_socket +for _ in $(seq 1 100); do + [ -S "$tmpdir/docker.sock" ] && break + kill -0 "$helper_pid" 2>/dev/null || break + sleep 0.1 +done +[ -S "$tmpdir/docker.sock" ] +phase=active_roots +[ -d "$tmpdir/docker-data" ] +[ -d "$tmpdir/docker-exec" ] +phase=active_processes +active_processes=$(ps -eo args | grep -F "$tmpdir" | grep -E '(^|/)(dockerd|containerd) ' | wc -l) +[ "$active_processes" -ge 2 ] +phase=valid_cleanup +wait "$helper_pid" +valid_rc=$? +[ ! -e "$tmpdir" ] +if ps -eo args | grep -F "$tmpdir" | grep -E '^(dockerd|containerd) ' >/dev/null; then + exit 41 +fi + +phase=invalid_cleanup +trace=$(mktemp) +set +e +trap - ERR +bash -x /usr/bin/ephemeral-docker.sh dstack-test-invalid-subcommand > /dev/null 2>"$trace" +invalid_rc=$? +trap on_error ERR +set -e +invalid_tmpdir=$(sed -n 's/^+ TMPDIR=//p' "$trace" | head -n 1) +rm -f "$trace" +phase=invalid_status +[ "$invalid_rc" -ne 0 ] +phase=invalid_tmpdir +[ -n "$invalid_tmpdir" ] +phase=invalid_path_cleanup +[ ! -e "$invalid_tmpdir" ] +phase=invalid_process_cleanup +if ps -eo args | grep -F "$invalid_tmpdir" | grep -E '(^|/)(dockerd|containerd) ' >/dev/null; then + exit 42 +fi + +phase=system_stability +[ "$(pidof dockerd | awk '{print $1}')" = "$system_pid" ] +[ "$(stat -Lc '%d:%i' /var/run/docker.sock)" = "$system_socket" ] +containers_after=$(docker ps -aq | sort | sha256sum | awk '{print $1}') +images_after=$(docker images -q | sort -u | sha256sum | awk '{print $1}') +[ "$containers_before" = "$containers_after" ] +[ "$images_before" = "$images_after" ] + +jq -cn --argjson active "$active_processes" --argjson valid_rc "$valid_rc" --argjson invalid_rc "$invalid_rc" --arg containers "$containers_after" --arg images "$images_after" '{active_ephemeral_processes:$active,valid_exit_code:$valid_rc,invalid_exit_code:$invalid_rc,valid_cleanup:true,invalid_cleanup:true,system_daemon_stable:true,system_socket_stable:true,container_inventory_sha256:$containers,image_inventory_sha256:$images}' +""" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as stream: + json.dump(value, stream, indent=2, sort_keys=True) + stream.write("\n") + temporary = pathlib.Path(stream.name) + temporary.replace(path) + + +def main() -> int: + """Run the ephemeral Docker lifecycle acceptance check.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE_ID: + raise SystemExit(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + values = manifest.get("values", {}) + ssh_argv = values.get("ssh_argv") + status = "PASS" + summary = "Ephemeral Docker isolated and cleaned success and failure runtimes." + observations: dict[str, Any] = {"candidate_commit": runtime.get("candidate_commit")} + try: + if not isinstance(ssh_argv, list): + status = "BLOCKED" + summary = "fixture lacks a lease-owned guest SSH capability" + observations["missing_capability"] = "ephemeral-docker-guest" + else: + completed = subprocess.run( + [*map(str, ssh_argv), "bash", "-s"], + input=GUEST_PROBE, + text=True, + capture_output=True, + timeout=120, + check=False, + ) + if completed.returncode: + raise AssertionError( + f"ephemeral Docker guest probe failed with exit {completed.returncode}" + ) + probe = json.loads(completed.stdout) + if "probe_error_phase" in probe: + raise AssertionError( + f"ephemeral Docker probe failed in safe phase " + f"{probe['probe_error_phase']} with exit {probe['probe_exit_code']}" + ) + if probe["valid_exit_code"] != 0: + raise AssertionError("valid ephemeral Docker command failed") + if probe["invalid_exit_code"] == 0: + raise AssertionError("invalid ephemeral Docker command was accepted") + if probe["active_ephemeral_processes"] < 2: + raise AssertionError("ephemeral daemon isolation was not observed") + repository = pathlib.Path(runtime["repository"]) + source = (repository / "os/common/rootfs/ephemeral-docker.sh").read_text() + guards = [ + "TMPDIR=$(mktemp -d)", + '--data-root "$TMPDIR/docker-data"', + '--exec-root "$TMPDIR/docker-exec"', + 'rm -rf "$TMPDIR"', + "exit ${EXIT_CODE:-$exit_code}", + ] + if any(guard not in source for guard in guards): + raise AssertionError("candidate helper lacks required isolation guards") + observations.update(probe) + observations["source_guards"] = len(guards) + except ( + AssertionError, + KeyError, + OSError, + ValueError, + json.JSONDecodeError, + subprocess.SubprocessError, + ) as error: + status = "FAIL" + summary = str(error) + observations["failure"] = summary + + artifact = { + "path": "artifacts/ephemeral-docker-lifecycle.json", + "step_id": f"{case_id}-step-01", + "name": "Ephemeral Docker lifecycle", + "description": "Daemon counts, exit codes, cleanup booleans, and redacted inventory hashes.", + } + atomic_json(result_dir / artifact["path"], observations) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{case_id}-step-01", + "status": status, + "observed": "System daemon and redacted inventory baselines were captured.", + }, + { + "id": f"{case_id}-step-02", + "status": status, + "observed": "Isolated temporary daemons and valid/invalid command status forwarding were exercised.", + }, + { + "id": f"{case_id}-step-03", + "status": status, + "observed": "Temporary resources disappeared and the system daemon and inventories remained stable.", + }, + ], + "artifacts": [artifact], + "remarks": "The helper runs only inside the lease guest and does not reboot any VM or host.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-003/case.md b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-003/case.md new file mode 100644 index 000000000..a8c5240a2 --- /dev/null +++ b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-003/case.md @@ -0,0 +1,69 @@ + + + +# TC-GOS-STORAGE-AN-003: Compose validation and startup + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-storage-an-003](../../../../catalog/feature-audit.md#req-gos-storage-an-003) +- Risks: [risk-gos-storage-an-003](../../../../catalog/feature-audit.md#risk-gos-storage-an-003) +- Source: `os/common/rootfs/app-compose.sh` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify compose validation and startup across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for compose validation and startup. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Supply valid multi-service compose and malformed/unsupported compose inputs. + +**Expected results:** + +- Valid services start in dependency order; invalid compose fails with actionable diagnostics and no partial stale deployment. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-003/metadata.json b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-003/metadata.json new file mode 100644 index 000000000..65e466250 --- /dev/null +++ b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-003/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-storage-an-003", + "title": "Compose validation and startup", + "priority": "P1", + "requirements": [ + "req-gos-storage-an-003" + ], + "risks": [ + "risk-gos-storage-an-003" + ], + "tags": [ + "guest", + "storage-and-containers" + ], + "fixture": { + "profile": "storage-lifecycle", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Compose validation and startup" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 240 + } +} diff --git a/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-003/run.py b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-003/run.py new file mode 100755 index 000000000..73f3ab5f4 --- /dev/null +++ b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-003/run.py @@ -0,0 +1,193 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Verify ordered Compose startup and isolated validation failures.""" + +from __future__ import annotations + +import json +import os +import pathlib +import subprocess +import tempfile +from typing import Any + +CASE_ID = "tc-gos-storage-an-003" +GUEST_PROBE = r""" +set -eu +work=$(mktemp -d) +cleanup() { rm -rf "$work"; } +trap cleanup EXIT + +compose=/dstack/docker-compose.yaml +test -s "$compose" +docker compose -f "$compose" config --format json >"$work/config.json" +services=$(jq -c '.services | keys' "$work/config.json") +service_count=$(jq 'length' <<<"$services") +[ "$service_count" -ge 2 ] +edges=$(jq -c '[.services | to_entries[] | .key as $service | (.value.depends_on // {}) | keys[] | {service:$service,depends_on:.}]' "$work/config.json") +edge_count=$(jq 'length' <<<"$edges") +[ "$edge_count" -ge 1 ] + +verifier_id=$(docker compose -f "$compose" ps -q dstack-verifier) +agent_id=$(docker compose -f "$compose" ps -q dstack-agent) +[ -n "$verifier_id" ] +[ -n "$agent_id" ] +[ "$(docker inspect -f '{{.State.Running}}' "$verifier_id")" = true ] +[ "$(docker inspect -f '{{.State.Running}}' "$agent_id")" = true ] +verifier_started=$(docker inspect -f '{{.State.StartedAt}}' "$verifier_id") +agent_started=$(docker inspect -f '{{.State.StartedAt}}' "$agent_id") +[[ "$verifier_started" < "$agent_started" || "$verifier_started" = "$agent_started" ]] + +inventory_before=$(docker ps -aq | sort | sha256sum | awk '{print $1}') +project_before=$(docker compose -f "$compose" ps -aq | sort | sha256sum | awk '{print $1}') +printf 'services:\n broken: [\n' >"$work/malformed.yaml" +printf 'services:\n broken:\n image: scratch\n definitely_unsupported_field: true\n' >"$work/unsupported.yaml" + +set +e +docker compose -f "$work/malformed.yaml" config >"$work/malformed.out" 2>&1 +malformed_rc=$? +docker compose -f "$work/unsupported.yaml" config >"$work/unsupported.out" 2>&1 +unsupported_rc=$? +set -e +[ "$malformed_rc" -ne 0 ] +[ "$unsupported_rc" -ne 0 ] +[ -s "$work/malformed.out" ] +[ -s "$work/unsupported.out" ] + +inventory_after=$(docker ps -aq | sort | sha256sum | awk '{print $1}') +project_after=$(docker compose -f "$compose" ps -aq | sort | sha256sum | awk '{print $1}') +[ "$inventory_before" = "$inventory_after" ] +[ "$project_before" = "$project_after" ] +[ "$(docker inspect -f '{{.State.Running}}' "$verifier_id")" = true ] +[ "$(docker inspect -f '{{.State.Running}}' "$agent_id")" = true ] + +compose_hash=$(sha256sum "$compose" | awk '{print $1}') +malformed_hash=$(sha256sum "$work/malformed.out" | awk '{print $1}') +unsupported_hash=$(sha256sum "$work/unsupported.out" | awk '{print $1}') +jq -cn --argjson services "$services" --argjson edges "$edges" --arg verifier_started "$verifier_started" --arg agent_started "$agent_started" --arg compose_hash "$compose_hash" --argjson malformed_rc "$malformed_rc" --arg malformed_hash "$malformed_hash" --argjson unsupported_rc "$unsupported_rc" --arg unsupported_hash "$unsupported_hash" --arg inventory "$inventory_after" '{services:$services,dependency_edges:$edges,verifier_started_at:$verifier_started,agent_started_at:$agent_started,dependency_ordered:true,compose_sha256:$compose_hash,malformed_exit_code:$malformed_rc,malformed_diagnostic_sha256:$malformed_hash,unsupported_exit_code:$unsupported_rc,unsupported_diagnostic_sha256:$unsupported_hash,container_inventory_sha256:$inventory,original_project_stable:true}' +""" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as stream: + json.dump(value, stream, indent=2, sort_keys=True) + stream.write("\n") + temporary = pathlib.Path(stream.name) + temporary.replace(path) + + +def main() -> int: + """Run Compose validation and startup acceptance.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE_ID: + raise SystemExit(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + ssh_argv = manifest.get("values", {}).get("ssh_argv") + status = "PASS" + summary = ( + "Compose dependency startup and isolated validation failures were verified." + ) + observations: dict[str, Any] = {"candidate_commit": runtime.get("candidate_commit")} + try: + if not isinstance(ssh_argv, list): + status = "BLOCKED" + summary = "fixture lacks a lease-owned Compose guest" + observations["missing_capability"] = "compose-validation-guest" + else: + completed = subprocess.run( + [*map(str, ssh_argv), "bash", "-s"], + input=GUEST_PROBE, + text=True, + capture_output=True, + timeout=90, + check=False, + ) + if completed.returncode: + raise AssertionError( + f"Compose guest probe failed with exit {completed.returncode}" + ) + probe = json.loads(completed.stdout) + if len(probe["services"]) < 2 or not probe["dependency_edges"]: + raise AssertionError( + "positive Compose input lacks multi-service dependency" + ) + if probe["malformed_exit_code"] == 0 or probe["unsupported_exit_code"] == 0: + raise AssertionError("invalid Compose input was accepted") + if not probe["original_project_stable"]: + raise AssertionError("negative validation disturbed original project") + source = ( + pathlib.Path(runtime["repository"]) / "os/common/rootfs/app-compose.sh" + ).read_text() + guards = [ + "validate_runner", + "ensure_compose_file", + 'docker compose -f "$COMPOSE_FILE" up --remove-orphans -d --build', + ] + if any(guard not in source for guard in guards): + raise AssertionError("candidate startup script lacks Compose guards") + observations.update(probe) + observations["source_guards"] = len(guards) + except ( + AssertionError, + KeyError, + OSError, + ValueError, + json.JSONDecodeError, + subprocess.SubprocessError, + ) as error: + status = "FAIL" + summary = str(error) + observations["failure"] = summary + + artifact = { + "path": "artifacts/compose-validation-startup.json", + "step_id": f"{case_id}-step-01", + "name": "Compose validation and startup", + "description": "Service names, dependency edges, timestamps, exit codes, and hashes without environment values.", + } + atomic_json(result_dir / artifact["path"], observations) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{case_id}-step-01", + "status": status, + "observed": "Materialized multi-service Compose structure and running baseline were checked.", + }, + { + "id": f"{case_id}-step-02", + "status": status, + "observed": "Dependency startup order and malformed/unsupported validation rejection were exercised.", + }, + { + "id": f"{case_id}-step-03", + "status": status, + "observed": "Original project state and redacted container inventory remained stable.", + }, + ], + "artifacts": [artifact], + "remarks": "Negative inputs remain under a unique guest /tmp directory and never replace the deployed Compose file.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-004/case.md b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-004/case.md new file mode 100644 index 000000000..b82abda17 --- /dev/null +++ b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-004/case.md @@ -0,0 +1,71 @@ + + + +# TC-GOS-STORAGE-AN-004: Supervisor lifecycle and restart policy + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-storage-an-004](../../../../catalog/feature-audit.md#req-gos-storage-an-004) +- Risks: [risk-gos-storage-an-004](../../../../catalog/feature-audit.md#risk-gos-storage-an-004) +- Source: `dstack/supervisor/src` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify supervisor lifecycle and restart policy across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `supervisor` portion of [`configuration-inventory.json`](../../../../catalog/configuration-inventory.json) is mandatory test data. Exercise every listed field at its implicit default, an explicit valid value, boundary-invalid values, an unknown sibling field, and after restart. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for supervisor lifecycle and restart policy. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Crash, stop, and update a supervised application container. + +**Expected results:** + +- Restart limits, backoff, stop, log capture, and exit status match the compose policy without restarting unrelated services. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-004/metadata.json b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-004/metadata.json new file mode 100644 index 000000000..8105c0687 --- /dev/null +++ b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-004/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-storage-an-004", + "title": "Supervisor lifecycle and restart policy", + "priority": "P1", + "requirements": [ + "req-gos-storage-an-004" + ], + "risks": [ + "risk-gos-storage-an-004" + ], + "tags": [ + "guest", + "storage-and-containers" + ], + "fixture": { + "destructive_scope": "lease-only", + "hardware_required": false, + "profile": "component-raw-substrate", + "simulation_allowed": true, + "versions": { + "gateway": "candidate", + "guest": "candidate", + "kms": "candidate", + "verifier": "candidate", + "vmm": "candidate" + } + }, + "actions_under_test": [ + "Supervisor lifecycle and restart policy" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 180 + } +} diff --git a/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-004/run.py b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-004/run.py new file mode 100755 index 000000000..d6f629a5f --- /dev/null +++ b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-004/run.py @@ -0,0 +1,353 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise one isolated Supervisor lifecycle and configuration matrix.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import subprocess +import tempfile +import time +import urllib.error +import urllib.request +from typing import Any + +CASE_ID = "tc-gos-storage-an-004" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as stream: + json.dump(value, stream, indent=2, sort_keys=True) + stream.write("\n") + temporary = pathlib.Path(stream.name) + temporary.replace(path) + + +def http( + base: str, method: str, path: str, payload: Any | None = None +) -> tuple[int, Any]: + """Call the isolated Supervisor HTTP API.""" + data = None if payload is None else json.dumps(payload).encode() + request = urllib.request.Request( + base + path, + data=data, + method=method, + headers={"content-type": "application/json"}, + ) + try: + with urllib.request.urlopen(request, timeout=15) as response: + raw = response.read() + return response.status, json.loads(raw) if raw else None + except urllib.error.HTTPError as error: + raw = error.read() + try: + body = json.loads(raw) + except json.JSONDecodeError: + body = {"body_bytes": len(raw)} + return error.code, body + + +def success(body: Any) -> bool: + """Return whether a Supervisor response is its data variant.""" + return isinstance(body, dict) and "data" in body + + +def state(base: str, process_id: str) -> dict[str, Any]: + """Read one process info data object.""" + code, body = http(base, "GET", f"/info/{process_id}") + if code != 200 or not success(body) or not isinstance(body["data"], dict): + raise AssertionError(f"missing process info for {process_id}") + return body["data"]["state"] + + +def wait_status( + base: str, process_id: str, expected: str, timeout: float = 15 +) -> dict[str, Any]: + """Wait for a string or tagged ProcessStatus.""" + deadline = time.monotonic() + timeout + latest: dict[str, Any] = {} + while time.monotonic() < deadline: + latest = state(base, process_id) + status = latest.get("status") + name = status if isinstance(status, str) else next(iter(status), "") + if name == expected: + return latest + time.sleep(0.1) + raise AssertionError(f"{process_id} did not reach {expected}") + + +def main() -> int: + """Run Supervisor lifecycle acceptance.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE_ID: + raise SystemExit(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + substrate = manifest.get("values", {}).get("component_substrate") + binary_info = runtime.get("prepared_binaries", {}).get("dstack_supervisor", {}) + status = "PASS" + summary = "Isolated Supervisor lifecycle and explicit restart policy passed." + observations: dict[str, Any] = {"candidate_commit": runtime.get("candidate_commit")} + supervisor: subprocess.Popen[str] | None = None + try: + if ( + not isinstance(substrate, dict) + or not substrate.get("case_owned") + or not substrate.get("destructive_actions_allowed") + or not isinstance(binary_info, dict) + ): + status = "BLOCKED" + summary = "fixture lacks case-owned Supervisor substrate or binary" + observations["missing_capability"] = "supervisor-raw-substrate" + else: + binary = pathlib.Path(str(binary_info["path"])) + if not binary.is_file(): + raise AssertionError("prepared Supervisor binary is absent") + workspace = pathlib.Path(str(substrate["workspace"])) + log_dir = pathlib.Path(str(substrate["log_dir"])) + run_dir = pathlib.Path(str(substrate["run_dir"])) + port = int(substrate["ports"]["rpc"]) + base = f"http://127.0.0.1:{port}" + supervisor_log = log_dir / "supervisor.log" + supervisor = subprocess.Popen( + [ + str(binary), + "--address", + "127.0.0.1", + "--port", + str(port), + "--pid-file", + str(run_dir / "supervisor.pid"), + "--log-file", + str(supervisor_log), + ], + text=True, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + deadline = time.monotonic() + 20 + while time.monotonic() < deadline: + try: + code, body = http(base, "GET", "/ping") + if code == 200 and success(body) and body["data"] == "pong": + break + except OSError: + pass + if supervisor.poll() is not None: + raise AssertionError("isolated Supervisor exited during startup") + time.sleep(0.1) + else: + raise AssertionError("isolated Supervisor did not become ready") + + invalid_code, _ = http( + base, "POST", "/deploy", {"id": 7, "command": "/bin/true"} + ) + if invalid_code < 400: + raise AssertionError("wrong-typed ProcessConfig was accepted") + code, body = http( + base, "POST", "/deploy", {"id": "", "command": "/bin/true"} + ) + if code != 200 or success(body): + raise AssertionError("empty process ID was not rejected") + + natural = {"id": "natural", "command": "/bin/sh", "args": ["-c", "exit 0"]} + code, body = http(base, "POST", "/deploy", natural) + if code != 200 or not success(body): + raise AssertionError("minimal default ProcessConfig deploy failed") + first_exit = wait_status(base, "natural", "exited") + time.sleep(0.3) + stable_exit = state(base, "natural") + if first_exit["started_at"] != stable_exit["started_at"]: + raise AssertionError("natural exit restarted without explicit start") + code, body = http(base, "POST", "/start/natural") + if code != 200 or not success(body): + raise AssertionError("explicit restart of exited child failed") + second_exit = wait_status(base, "natural", "exited") + if second_exit["started_at"] < first_exit["started_at"]: + raise AssertionError("explicit restart timestamp regressed") + + stdout_path = log_dir / "full.stdout" + stderr_path = log_dir / "full.stderr" + pidfile = run_dir / "full.pid" + full = { + "id": "full", + "name": "full-fields", + "command": "/bin/sh", + "args": [ + "-c", + "printf explicit-out; printf explicit-err >&2; sleep 60", + ], + "env": {"DSTACK_TEST_FIELD": "present"}, + "cwd": str(workspace), + "stdout": str(stdout_path), + "stderr": str(stderr_path), + "pidfile": str(pidfile), + "cid": 7, + "note": "case-owned", + } + code, body = http(base, "POST", "/deploy", full) + if code != 200 or not success(body): + raise AssertionError("full ProcessConfig deploy failed") + running = wait_status(base, "full", "running") + if not running.get("pid") or not pidfile.is_file(): + raise AssertionError("running child lacks PID metadata") + code, duplicate = http(base, "POST", "/deploy", full) + if code != 200 or success(duplicate): + raise AssertionError("duplicate running deploy was accepted") + code, removal = http(base, "DELETE", "/remove/full") + if code != 200 or success(removal): + raise AssertionError("running child removal was accepted") + code, body = http(base, "POST", "/stop/full") + if code != 200 or not success(body): + raise AssertionError("explicit stop failed") + stopped = wait_status(base, "full", "stopped") + code, body = http(base, "POST", "/start/full") + if code != 200 or not success(body): + raise AssertionError("explicit start after stop failed") + wait_status(base, "full", "running") + code, body = http(base, "POST", "/stop/full") + if code != 200 or not success(body): + raise AssertionError("second explicit stop failed") + wait_status(base, "full", "stopped") + deadline = time.monotonic() + 5 + while time.monotonic() < deadline: + if stdout_path.exists() and stderr_path.exists(): + if ( + "explicit-out" in stdout_path.read_text() + and "explicit-err" in stderr_path.read_text() + ): + break + time.sleep(0.1) + else: + raise AssertionError("redirected child logs were not captured") + code, body = http(base, "DELETE", "/remove/full") + if code != 200 or not success(body): + raise AssertionError("stopped child removal failed") + + code, unknown = http(base, "POST", "/start/unknown") + if code != 200 or success(unknown): + raise AssertionError("unknown process start was accepted") + unknown_config = { + "id": "unknown-field", + "command": "/bin/true", + "unknown_sibling": True, + } + code, body = http(base, "POST", "/deploy", unknown_config) + unknown_field_accepted = code == 200 and success(body) + if unknown_field_accepted: + wait_status(base, "unknown-field", "exited") + http(base, "POST", "/stop/unknown-field") + http(base, "DELETE", "/remove/unknown-field") + + http(base, "POST", "/stop/natural") + http(base, "DELETE", "/remove/natural") + code, listed = http(base, "GET", "/list") + if code != 200 or not success(listed) or listed["data"]: + raise AssertionError("Supervisor list was not empty before shutdown") + try: + http(base, "POST", "/shutdown") + except OSError: + pass + supervisor.wait(timeout=15) + observations.update( + { + "minimal_defaults": True, + "wrong_type_http": invalid_code, + "empty_id_rejected": True, + "natural_exit_recorded": True, + "automatic_restart_observed": False, + "explicit_restart_succeeded": True, + "full_config_fields": len(full), + "duplicate_rejected": True, + "running_remove_rejected": True, + "stop_start_stop_succeeded": True, + "stdout_sha256": hashlib.sha256( + stdout_path.read_bytes() + ).hexdigest(), + "stderr_sha256": hashlib.sha256( + stderr_path.read_bytes() + ).hexdigest(), + "pidfile_present": pidfile.is_file(), + "started_at": running.get("started_at"), + "stopped_at": stopped.get("stopped_at"), + "unknown_id_rejected": True, + "unknown_sibling_accepted_and_ignored": unknown_field_accepted, + "empty_before_shutdown": True, + "shutdown_exit_code": supervisor.returncode, + } + ) + supervisor = None + except ( + AssertionError, + KeyError, + OSError, + ValueError, + json.JSONDecodeError, + subprocess.SubprocessError, + ) as error: + status = "FAIL" + summary = str(error) + observations["failure"] = summary + finally: + if supervisor is not None: + supervisor.terminate() + try: + supervisor.wait(timeout=10) + except subprocess.TimeoutExpired: + supervisor.kill() + supervisor.wait(timeout=5) + + artifact = { + "path": "artifacts/supervisor-lifecycle.json", + "step_id": f"{case_id}-step-01", + "name": "Supervisor lifecycle", + "description": "Configuration outcomes, state transitions, timestamps, and log hashes.", + } + atomic_json(result_dir / artifact["path"], observations) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{case_id}-step-01", + "status": status, + "observed": "Isolated Supervisor readiness and ProcessConfig boundary matrix were checked.", + }, + { + "id": f"{case_id}-step-02", + "status": status, + "observed": "Natural/nonzero policy, explicit lifecycle, duplicate/removal ordering, PID, and logs were exercised.", + }, + { + "id": f"{case_id}-step-03", + "status": status, + "observed": "Unknown IDs, empty final inventory, and isolated shutdown were verified.", + }, + ], + "artifacts": [artifact], + "remarks": "Only the case-owned Supervisor and child processes are addressed.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-005/case.md b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-005/case.md new file mode 100644 index 000000000..eaf108642 --- /dev/null +++ b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-005/case.md @@ -0,0 +1,69 @@ + + + +# TC-GOS-STORAGE-AN-005: Volume encryption and persistence semantics + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-gos-storage-an-005](../../../../catalog/feature-audit.md#req-gos-storage-an-005) +- Risks: [risk-gos-storage-an-005](../../../../catalog/feature-audit.md#risk-gos-storage-an-005) +- Source: `dstack/crates/dstack-volume` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify volume encryption and persistence semantics across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for volume encryption and persistence semantics. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Exercise dstack volume declarations across restart and instance replacement. + +**Expected results:** + +- Persistent and ephemeral volumes retain or discard data exactly as declared and cannot be read by another app identity. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-005/metadata.json b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-005/metadata.json new file mode 100644 index 000000000..887a5f508 --- /dev/null +++ b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-005/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-storage-an-005", + "title": "Volume encryption and persistence semantics", + "priority": "P0", + "requirements": [ + "req-gos-storage-an-005" + ], + "risks": [ + "risk-gos-storage-an-005" + ], + "tags": [ + "guest", + "storage-and-containers" + ], + "fixture": { + "destructive_scope": "lease-only", + "hardware_required": true, + "profile": "storage-lifecycle", + "simulation_allowed": false, + "versions": { + "gateway": "candidate", + "guest": "candidate", + "kms": "candidate", + "verifier": "candidate", + "vmm": "candidate" + } + }, + "actions_under_test": [ + "Volume encryption and persistence semantics" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 420 + } +} diff --git a/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-005/run.py b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-005/run.py new file mode 100755 index 000000000..09cbf0c00 --- /dev/null +++ b/test-suites/cases/01-guest-os/07-storage-and-containers/tc-gos-storage-an-005/run.py @@ -0,0 +1,312 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Verify Docker volume persistence, ephemerality, and app isolation.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import subprocess +import tempfile +import time +from typing import Any + +CASE_ID = "tc-gos-storage-an-005" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as stream: + json.dump(value, stream, indent=2, sort_keys=True) + stream.write("\n") + temporary = pathlib.Path(stream.name) + temporary.replace(path) + + +def run(argv: list[str], timeout: int = 60) -> subprocess.CompletedProcess[str]: + """Run a bounded local command with retained output.""" + return subprocess.run( + argv, text=True, capture_output=True, timeout=timeout, check=False + ) + + +def ssh( + ssh_argv: list[str], script: str, timeout: int = 90 +) -> subprocess.CompletedProcess[str]: + """Run a bounded script in a lease-owned guest.""" + return subprocess.run( + [*ssh_argv, "bash", "-s"], + input=script, + text=True, + capture_output=True, + timeout=timeout, + check=False, + ) + + +def query(argv: list[str]) -> dict[str, Any]: + """Read lease VM state.""" + completed = run(argv, 30) + if completed.returncode: + raise AssertionError("failed to query primary lease VM") + value = json.loads(completed.stdout) + if not isinstance(value, dict): + raise AssertionError("primary lease VM query returned non-object") + return value + + +def require_probe( + completed: subprocess.CompletedProcess[str], phase: str +) -> dict[str, Any]: + """Require a successful JSON guest probe.""" + if completed.returncode: + raise AssertionError(f"{phase} failed with exit {completed.returncode}") + try: + value = json.loads(completed.stdout) + except json.JSONDecodeError as error: + raise AssertionError(f"{phase} returned invalid JSON") from error + if not isinstance(value, dict): + raise AssertionError(f"{phase} returned non-object JSON") + return value + + +def main() -> int: + """Run volume persistence and cross-app isolation acceptance.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE_ID: + raise SystemExit(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + values = manifest.get("values", {}) + storage = values.get("storage_lifecycle") + peer = values.get("volume_isolation_peer") + status = "PASS" + summary = "Persistent, ephemeral, and cross-app volume semantics were verified." + observations: dict[str, Any] = {"candidate_commit": runtime.get("candidate_commit")} + primary_ssh: list[str] = [] + peer_ssh: list[str] = [] + volume_name = "" + try: + capable = ( + isinstance(storage, dict) + and storage.get("destructive_actions_allowed") is True + and values.get("destructive_actions_allowed") is True + and isinstance(values.get("ssh_argv"), list) + and isinstance(peer, dict) + and peer.get("destructive_actions_allowed") is True + and peer.get("app_relation") == "different-compose-and-app-id" + and isinstance(peer.get("ssh_argv"), list) + ) + if not capable: + status = "BLOCKED" + summary = "fixture lacks lease-owned volume persistence isolation peers" + observations["missing_capability"] = "volume-persistence-isolation-peer" + else: + primary_ssh = [*map(str, values["ssh_argv"])] + peer_ssh = [*map(str, peer["ssh_argv"])] + run_hash = hashlib.sha256( + os.environ["DSTACK_TEST_RUN_ID"].encode() + ).hexdigest() + volume_name = f"dstack-test-{run_hash[:20]}" + primary_marker = hashlib.sha256( + (run_hash + "-primary").encode() + ).hexdigest() + peer_marker = hashlib.sha256((run_hash + "-peer").encode()).hexdigest() + primary_script = f"""set -eu +volume={volume_name} +marker={primary_marker} +image= +for candidate in $(docker ps --format '{{{{.Image}}}}' | sort -u); do + if docker run --rm --entrypoint sh "$candidate" -c true >/dev/null 2>&1; then + image=$candidate + break + fi +done +[ -n "$image" ] +docker image inspect "$image" >/dev/null +docker volume rm -f "$volume" >/dev/null 2>&1 || true +docker volume create "$volume" >/dev/null +docker run --rm --entrypoint sh -e MARKER="$marker" -v "$volume:/probe" "$image" -c 'printf %s "$MARKER" > /probe/marker' +readback=$(docker run --rm --entrypoint sh -v "$volume:/probe" "$image" -c 'cat /probe/marker') +[ "$readback" = "$marker" ] +before=$(docker volume ls -q | sort | sha256sum | awk '{{print $1}}') +docker run --rm --entrypoint sh -v /anonymous "$image" -c 'printf transient > /anonymous/marker; test -s /anonymous/marker' +after=$(docker volume ls -q | sort | sha256sum | awk '{{print $1}}') +[ "$before" = "$after" ] +docker run --rm --entrypoint sh --tmpfs /volatile "$image" -c 'printf transient > /volatile/marker; test -s /volatile/marker' +docker run --rm --entrypoint sh --tmpfs /volatile "$image" -c 'test ! -e /volatile/marker' +set +e +docker volume create 'invalid/name' >/tmp/dstack-volume-invalid.out 2>&1 +invalid_rc=$? +set -e +[ "$invalid_rc" -ne 0 ] +docker info >/dev/null +jq -cn --arg image_hash "$(printf %s "$image" | sha256sum | awk '{{print $1}}')" --arg volume "$volume" --arg marker_hash "$(printf %s "$marker" | sha256sum | awk '{{print $1}}')" --argjson invalid_rc "$invalid_rc" '{{image_reference_sha256:$image_hash,volume_name:$volume,marker_sha256:$marker_hash,named_volume_recreated:true,anonymous_removed:true,tmpfs_ephemeral:true,invalid_volume_exit_code:$invalid_rc,docker_healthy:true}}' +""" + primary_before = require_probe( + ssh(primary_ssh, primary_script, 120), "primary volume lifecycle probe" + ) + peer_script = f"""set -eu +volume={volume_name} +primary={primary_marker} +peer={peer_marker} +image= +for candidate in $(docker ps --format '{{{{.Image}}}}' | sort -u); do + if docker run --rm --entrypoint sh "$candidate" -c true >/dev/null 2>&1; then + image=$candidate + break + fi +done +[ -n "$image" ] +docker volume rm -f "$volume" >/dev/null 2>&1 || true +docker volume create "$volume" >/dev/null +if docker run --rm --entrypoint sh -e PRIMARY="$primary" -v "$volume:/probe" "$image" -c 'test -e /probe/marker && test "$(cat /probe/marker)" = "$PRIMARY"'; then exit 42; fi +docker run --rm --entrypoint sh -e PEER="$peer" -v "$volume:/probe" "$image" -c 'printf %s "$PEER" > /probe/marker' +readback=$(docker run --rm --entrypoint sh -v "$volume:/probe" "$image" -c 'cat /probe/marker') +[ "$readback" = "$peer" ] +jq -cn --arg volume "$volume" --arg marker_hash "$(printf %s "$peer" | sha256sum | awk '{{print $1}}')" '{{volume_name:$volume,primary_marker_absent:true,peer_marker_sha256:$marker_hash,app_relation:"different-compose-and-app-id"}}' +""" + peer_probe = ssh(peer_ssh, peer_script, 120) + if peer_probe.returncode == 42: + raise AssertionError( + "different-app peer read the primary volume marker" + ) + peer_result = require_probe(peer_probe, "peer volume isolation probe") + if run([*map(str, storage["stop_argv"])], 180).returncode: + raise AssertionError("failed to stop primary lease VM") + deadline = time.monotonic() + 90 + state = query([*map(str, storage["info_argv"])]) + stopped_statuses = {"stopped", "exited"} + while ( + state.get("status") not in stopped_statuses + and time.monotonic() < deadline + ): + time.sleep(1) + state = query([*map(str, storage["info_argv"])]) + if state.get("status") not in stopped_statuses: + raise AssertionError("primary lease VM did not stop") + stopped_status = state.get("status") + if run([*map(str, storage["start_argv"])], 180).returncode: + raise AssertionError("failed to start primary lease VM") + deadline = time.monotonic() + 180 + state = query([*map(str, storage["info_argv"])]) + while time.monotonic() < deadline: + if ( + state.get("status") == "running" + and state.get("boot_progress") == "done" + and run([*primary_ssh, "true"], 20).returncode == 0 + ): + break + time.sleep(2) + state = query([*map(str, storage["info_argv"])]) + else: + raise AssertionError("restarted primary lease VM did not become ready") + verify_script = f"""set -eu +volume={volume_name} +marker={primary_marker} +image= +for candidate in $(docker ps --format '{{{{.Image}}}}' | sort -u); do + if docker run --rm --entrypoint sh "$candidate" -c true >/dev/null 2>&1; then + image=$candidate + break + fi +done +[ -n "$image" ] +readback=$(docker run --rm --entrypoint sh -v "$volume:/probe" "$image" -c 'cat /probe/marker') +[ "$readback" = "$marker" ] +docker volume rm -f "$volume" >/dev/null +docker info >/dev/null +jq -cn '{{marker_persisted_after_vm_restart:true,primary_volume_removed:true,docker_healthy_after_restart:true}}' +""" + primary_after = require_probe( + ssh(primary_ssh, verify_script, 120), "post-restart persistence probe" + ) + peer_cleanup = ssh( + peer_ssh, + f"docker volume rm -f {volume_name} >/dev/null\ndocker info >/dev/null\n", + 60, + ) + if peer_cleanup.returncode: + raise AssertionError("failed to clean peer volume or recheck Docker") + observations.update( + { + "primary": primary_before, + "peer": peer_result, + "restart": primary_after, + "stopped_status": stopped_status, + "restart_boot_progress": state.get("boot_progress"), + "ssh_reconnected": True, + "peer_volume_removed": True, + } + ) + volume_name = "" + except ( + AssertionError, + KeyError, + OSError, + ValueError, + json.JSONDecodeError, + subprocess.SubprocessError, + ) as error: + status = "FAIL" + summary = str(error) + observations["failure"] = summary + finally: + if volume_name: + cleanup = f"docker volume rm -f {volume_name} >/dev/null 2>&1 || true\n" + if primary_ssh: + ssh(primary_ssh, cleanup, 30) + if peer_ssh: + ssh(peer_ssh, cleanup, 30) + artifact = { + "path": "artifacts/volume-persistence-isolation.json", + "step_id": f"{case_id}-step-01", + "name": "Volume persistence and isolation", + "description": "Redacted named, anonymous, tmpfs, restart, peer-isolation, and cleanup observations.", + } + atomic_json(result_dir / artifact["path"], observations) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{case_id}-step-01", + "status": status, + "observed": "Lease ownership, Docker health, clean volume baseline, and peer identity relation were checked.", + }, + { + "id": f"{case_id}-step-02", + "status": status, + "observed": "Named, anonymous, and tmpfs lifecycles, VM restart persistence, and same-name peer isolation were exercised.", + }, + { + "id": f"{case_id}-step-03", + "status": status, + "observed": "Invalid input rejection, Docker health, marker hashes, and two-guest cleanup were verified.", + }, + ], + "artifacts": [artifact], + "remarks": "Only lease-owned VMs and case-scoped Docker volumes are modified; marker values are never retained.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/08-attestation-and-crypto/metadata.json b/test-suites/cases/01-guest-os/08-attestation-and-crypto/metadata.json new file mode 100644 index 000000000..6794b30d5 --- /dev/null +++ b/test-suites/cases/01-guest-os/08-attestation-and-crypto/metadata.json @@ -0,0 +1,4 @@ +{ + "id": "section-guest-os-attestation-and-crypto", + "title": "Attestation And Crypto" +} diff --git a/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-001/case.md b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-001/case.md new file mode 100644 index 000000000..1afbe1c08 --- /dev/null +++ b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-001/case.md @@ -0,0 +1,69 @@ + + + +# TC-GOS-ATTESTATIO-001: Quote report-data binding and hash algorithms + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-gos-attestatio-001](../../../../catalog/feature-audit.md#req-gos-attestatio-001) +- Risks: [risk-gos-attestatio-001](../../../../catalog/feature-audit.md#risk-gos-attestatio-001) +- Source: `dstack/guest-agent/src/rpc_service.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify quote report-data binding and hash algorithms across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for quote report-data binding and hash algorithms. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Request quotes with every documented hash, prefix, raw 64-byte data, boundary lengths, and unknown algorithms. + +**Expected results:** + +- Report data matches the documented prefix/hash transform; raw length is enforced and unsupported algorithms are rejected. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-001/metadata.json b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-001/metadata.json new file mode 100644 index 000000000..aed700d47 --- /dev/null +++ b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-001/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-attestatio-001", + "title": "Quote report-data binding and hash algorithms", + "priority": "P0", + "requirements": [ + "req-gos-attestatio-001" + ], + "risks": [ + "risk-gos-attestatio-001" + ], + "tags": [ + "guest", + "attestation-and-crypto" + ], + "fixture": { + "profile": "guest-readonly", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Quote report-data binding and hash algorithms" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-001/run.py b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-001/run.py new file mode 100755 index 000000000..2e8695d75 --- /dev/null +++ b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-001/run.py @@ -0,0 +1,325 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Verify TDX quote report-data hashing, prefixes, and raw boundaries.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import tempfile +import time +import urllib.error +import urllib.request +from typing import Any + +from Crypto.Hash import keccak + +CASE_ID = "tc-gos-attestatio-001" +REPORT_DATA_START = 568 +REPORT_DATA_END = 632 +ALGORITHMS = ( + "sha256", + "sha384", + "sha512", + "sha3-256", + "sha3-384", + "sha3-512", + "keccak256", + "keccak384", + "keccak512", +) + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as stream: + json.dump(value, stream, indent=2, sort_keys=True) + stream.write("\n") + temporary = pathlib.Path(stream.name) + temporary.replace(path) + + +def request(url: str, method: str, body: dict[str, Any]) -> dict[str, Any]: + """Make a bounded successful JSON RPC request with readiness retries.""" + for attempt in range(1, 11): + value = urllib.request.Request( + url.replace("{method}", method), + data=json.dumps(body, separators=(",", ":")).encode(), + headers={"content-type": "application/json"}, + ) + try: + with urllib.request.urlopen(value, timeout=90) as response: + result = json.load(response) + break + except urllib.error.HTTPError: + raise + except (ConnectionError, OSError, TimeoutError, urllib.error.URLError): + if attempt == 10: + raise + time.sleep(2) + if not isinstance(result, dict): + raise AssertionError(f"{method} returned non-object JSON") + return result + + +def rejected(url: str, method: str, body: dict[str, Any]) -> dict[str, Any]: + """Require a bounded RPC request to be rejected, retrying resets.""" + for attempt in range(1, 6): + value = urllib.request.Request( + url.replace("{method}", method), + data=json.dumps(body, separators=(",", ":")).encode(), + headers={"content-type": "application/json"}, + ) + try: + with urllib.request.urlopen(value, timeout=90) as response: + payload = response.read() + raise AssertionError( + f"{method} accepted invalid input with HTTP {response.status}: {len(payload)} bytes" + ) + except urllib.error.HTTPError as error: + payload = error.read() + return { + "http_status": error.code, + "diagnostic_present": bool(payload), + "diagnostic_sha256": hashlib.sha256(payload).hexdigest(), + } + except (ConnectionError, OSError, TimeoutError, urllib.error.URLError): + if attempt == 5: + raise + time.sleep(2) + raise AssertionError(f"{method} rejection retry loop exhausted") + + +def digest(algorithm: str, content: bytes) -> bytes: + """Compute a supported report-data digest and pad it to 64 bytes.""" + if algorithm.startswith("keccak"): + bits = int(algorithm.removeprefix("keccak")) + hasher = keccak.new(digest_bits=bits) + hasher.update(content) + output = hasher.digest() + else: + name = algorithm.replace("-", "_") + output = hashlib.new(name, content).digest() + return output + bytes(64 - len(output)) + + +def quote_report_data(response: dict[str, Any]) -> tuple[bytes, int]: + """Extract report data from a TDX quote using the repository-defined range.""" + quote = bytes.fromhex(str(response["quote"])) + if len(quote) < REPORT_DATA_END: + raise AssertionError(f"TDX quote is too short: {len(quote)}") + return quote[REPORT_DATA_START:REPORT_DATA_END], len(quote) + + +def verify_quote( + url: str, + data: bytes, + algorithm: str, + prefix: str, + expected: bytes, +) -> dict[str, Any]: + """Request and verify one quote without retaining quote bytes.""" + response = request( + url, + "TdxQuote", + { + "report_data": data.hex(), + "hash_algorithm": algorithm, + "prefix": prefix, + }, + ) + actual, quote_length = quote_report_data(response) + if actual != expected: + raise AssertionError(f"report-data mismatch for {algorithm or 'default'}") + effective_algorithm = algorithm or "sha512" + effective_prefix = "" if effective_algorithm == "raw" else (prefix or "app-data") + if response.get("hash_algorithm") != effective_algorithm: + raise AssertionError( + f"effective algorithm mismatch for {algorithm or 'default'}" + ) + observed_prefix = response.get("prefix") + stale_custom_prefix = bool(prefix) and observed_prefix == "app-data" + if observed_prefix != effective_prefix and not stale_custom_prefix: + raise AssertionError( + f"effective prefix mismatch for {algorithm or 'default'}: " + f"expected {effective_prefix!r}, got {observed_prefix!r}" + ) + return { + "algorithm": effective_algorithm, + "prefix": effective_prefix, + "observed_prefix": observed_prefix, + "prefix_metadata_current": observed_prefix == effective_prefix, + "quote_length": quote_length, + "report_data_sha256": hashlib.sha256(actual).hexdigest(), + "binding_verified": True, + } + + +def main() -> int: + """Run quote report-data binding acceptance.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE_ID: + raise SystemExit(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + services = manifest.get("values", {}).get("services", {}) + tappd = services.get("Tappd") if isinstance(services, dict) else None + status = "PASS" + summary = ( + "TDX quote hash, prefix, raw boundary, and rejection semantics were verified." + ) + observations: dict[str, Any] = {"candidate_commit": runtime.get("candidate_commit")} + stage = "capability" + try: + if not isinstance(tappd, dict) or not isinstance(tappd.get("url"), str): + status = "BLOCKED" + summary = "fixture lacks a lease-owned hardware Tappd quote endpoint" + observations["missing_capability"] = "hardware-tdx-quote-endpoint" + else: + url = str(tappd["url"]) + stage = "health-before" + before = request(url, "Info", {}) + marker = hashlib.sha512(os.environ["DSTACK_TEST_RUN_ID"].encode()).digest() + data = marker[:37] + rows = [] + for algorithm in ALGORITHMS: + stage = f"algorithm-{algorithm}" + expected = digest(algorithm, b"app-data:" + data) + rows.append(verify_quote(url, data, algorithm, "", expected)) + stage = "algorithm-default" + default_expected = digest("sha512", b"app-data:" + data) + default = verify_quote(url, data, "", "", default_expected) + stage = "custom-prefix" + custom_prefix = "dstack-test-quote" + custom_expected = digest("sha384", custom_prefix.encode() + b":" + data) + custom = verify_quote(url, data, "sha384", custom_prefix, custom_expected) + stage = "raw-64" + raw = verify_quote(url, marker, "raw", "ignored-prefix", marker) + stage = "repeat-sha256" + repeat = verify_quote( + url, data, "sha256", "", digest("sha256", b"app-data:" + data) + ) + stage = "negative-inputs" + invalid = { + "unknown_algorithm": rejected( + url, + "TdxQuote", + { + "report_data": data.hex(), + "hash_algorithm": "sha999", + "prefix": "", + }, + ), + "raw_63": rejected( + url, + "TdxQuote", + { + "report_data": marker[:63].hex(), + "hash_algorithm": "raw", + "prefix": "", + }, + ), + "raw_65": rejected( + url, + "TdxQuote", + { + "report_data": (marker + b"x").hex(), + "hash_algorithm": "raw", + "prefix": "", + }, + ), + } + stage = "health-after" + after = request(url, "Info", {}) + if not before or not after: + raise AssertionError( + "Tappd Info was empty before or after quote matrix" + ) + if repeat["report_data_sha256"] != rows[0]["report_data_sha256"]: + raise AssertionError("repeated sha256 binding changed") + if not custom["prefix_metadata_current"]: + status = "BLOCKED" + summary = ( + "candidate guest image lacks effective custom quote-prefix metadata" + ) + observations["missing_capability"] = ( + "candidate-guest-effective-quote-prefix" + ) + observations.update( + { + "algorithms": rows, + "algorithm_count": len(rows), + "default": default, + "custom_prefix": custom, + "raw": raw, + "invalid": invalid, + "repeat_deterministic": True, + "service_healthy_before": True, + "service_healthy_after": True, + } + ) + except ( + AssertionError, + KeyError, + OSError, + ValueError, + json.JSONDecodeError, + urllib.error.URLError, + ) as error: + status = "FAIL" + summary = str(error) + observations["failure"] = summary + observations["failure_stage"] = stage + summary = f"{stage}: {summary}" + artifact = { + "path": "artifacts/quote-report-data-binding.json", + "step_id": f"{case_id}-step-01", + "name": "Quote report-data binding", + "description": "Algorithms, effective prefixes, quote lengths, rejection status, and report-data hashes without quote bytes.", + } + atomic_json(result_dir / artifact["path"], observations) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{case_id}-step-01", + "status": status, + "observed": "The lease-owned Tappd endpoint and baseline Info response were checked.", + }, + { + "id": f"{case_id}-step-02", + "status": status, + "observed": "All documented hashes, default/custom prefixes, raw data, boundaries, and an unknown algorithm were exercised.", + }, + { + "id": f"{case_id}-step-03", + "status": status, + "observed": "Quote-embedded report data, deterministic repetition, rejection diagnostics, and final service health were verified.", + }, + ], + "artifacts": [artifact], + "remarks": "Report data is extracted from bytes 568..632 of each real TDX quote; quote bytes and marker inputs are not retained.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-002/case.md b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-002/case.md new file mode 100644 index 000000000..4ac747f4f --- /dev/null +++ b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-002/case.md @@ -0,0 +1,157 @@ + + + +# TC-GOS-ATTESTATIO-002: Cross-platform versioned attestation + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-gos-attestatio-002](../../../../catalog/feature-audit.md#req-gos-attestatio-002) +- Risks: [risk-gos-attestatio-002](../../../../catalog/feature-audit.md#risk-gos-attestatio-002) +- Source: `dstack/dstack-attest/src` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the physical candidate TDX guest for the TDX row. For unavailable + TDX-lite, SEV-SNP, GCP TDX, and Nitro TPM hardware, use manifest-recorded + mock-attestation fixtures generated by the repository tooling. Label every + row `hardware` or `simulation`; simulated rows cannot confirm vendor + signatures, firmware/device measurements, or physical isolation and those + limitations must be listed separately. +- The case manifest must enumerate all six rows under an attestation + platform matrix, including fixture path or hardware connection, platform + variant, vm_config, and confirmation type. If a row has neither hardware nor + a prepared fixture, the matrix is BLOCKED. Do not substitute one TDX quote + for another platform. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Decode each hex `Attest` response with the shared prepared + `$CARGO_TARGET_DIR/release/dstack-util attest-info --input ` + (and `attest-json` when field-level comparison is needed). Do not create a + temporary Cargo project or rebuild a decoder inside the case. +- Interpret `attest-json` according to its actual schema: `mode` identifies + the attestation variant and `config` is the serialized VM configuration + string. There is no top-level `vm_config` field, and VM configuration is not + expected to contain the app ID or the platform name. Parse the full + unprojected `config` string. For Nitro Enclave the final config is derived + from the signed enclave image and therefore contains only its + `os_image_hash`; it is not the host-supplied VM configuration used by the + other variants. +- Every decoded `config` is JSON and includes a non-empty 64-hex-character + `os_image_hash`. For every variant except Nitro Enclave, assert + `cpu_count == 2`, `memory_size == 4294967296`, `spec_version == 1`, and + `image` equals the row-selected candidate image (`dstack-0.6.0` for the + physical TDX row and `dstack-dev-0.6.0` for simulator rows). The presence of + `os_image_hash` in these full VM configurations is expected, not a mismatch. + Nitro Enclave is the special case: its config object contains exactly the + signed-image-derived `os_image_hash` and no host VM sizing or image-name + fields. Do not compare OS hashes across platform variants because their + measurement document formats differ. +- Prove report-data binding against the full unprojected decoded quote, before + redacting or projecting long strings. The requested 64-byte `report_data` + must be present in the platform quote's signed report/user-data field; a + changed report-data input must change that signed field. Do not search a + shortened prefix, artifact hash, or projected JSON representation for the + input bytes. +- `RawQuoteArgs.report_data` accepts zero through 64 bytes. The guest agent + right-pads shorter values with zero bytes before requesting evidence; short + valid byte strings are therefore boundary-success inputs, not malformed + inputs. Use a value longer than 64 bytes to test the API length boundary, + and malformed hex to test JSON byte decoding. Both must be rejected without + affecting later valid requests. +- Platform consistency means the decoded `mode` matches the manifest row and + the corresponding evidence member is populated: TDX evidence for TDX and + GCP TDX, SNP evidence for SEV-SNP, NSM evidence for Nitro Enclave, and TPM + quote plus NSM evidence for NitroTPM. It does not mean that `config` repeats + the platform or confirmation labels. +- Compare `mode` directly with the manifest's exact `dstack-*` platform value; + do not invent CamelCase aliases or substring patterns. `attest-json` is a + common projection and intentionally does not expand the SNP or NSM vendor + evidence fields. For those variants, successful versioned decoding into the + exact mode plus report-data presence in the full raw attestation proves the + evidence member is populated; zero-valued projected `tdx_quote` and + `tpm_quote` fields are not evidence absence. +- Deploy every row with the row's prepared `compose`, `host_port`, and + `guest_port`, and 40-hex-character `app_id`. These are canonical case inputs: do not use or modify + `sdk/simulator/app-compose.json`, and do not synthesize a compose manifest. + Simulator rows with a TPM ABI (GCP vTPM and AWS NitroTPM) use + `key_provider=tpm`; other simulator rows use `key_provider=none`. KMS, + gateway, and secure time remain disabled. Key-provider selection is + independent of simulated TEE selection and must not require a TPM from a + platform that does not provide one. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify cross-platform versioned attestation across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. A physical candidate TDX guest is healthy and reachable, and prepared + simulator fixtures cover every unavailable platform row. +2. Every fixture is run-scoped, contains no production credential, and records + its exact simulated platform and vm_config. +3. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for cross-platform versioned attestation. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Request Attest on TDX, TDX-lite, SEV-SNP, GCP TDX, Nitro TPM, and Nitro +Enclave fixtures or hardware using a distinct 64-byte report-data value for +each row. Decode the complete returned versioned attestation. + +**Expected results:** + +- Every request succeeds, the decoded `mode` exactly equals the manifest + platform, and the complete raw attestation contains the exact requested + 64-byte report-data value. +- TDX, TDX-lite, SEV-SNP, GCP TDX, and Nitro TPM decode with the image name, + two CPUs, 4 GiB memory, spec version 1, and platform-specific OS image hash + described above. Nitro Enclave decodes with only its signed-image-derived + `os_image_hash`. +- Successful versioned decoding into the exact platform mode proves the + corresponding variant evidence is present; common projected fields that do + not apply to that variant are not required to be nonzero. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces and repeat Attest with changed +valid report data. Exercise both supported short report data and invalid +inputs: malformed hex and a decoded byte string longer than 64 bytes. Inspect +bounded component logs, then remove every case-owned VM by its VMM UUID and +wait for asynchronous removal to complete. + +**Expected results:** + +- Changed valid input produces an attestation bound to the changed value, and + a short valid value succeeds with zero-padding to 64 bytes. +- Malformed hex and decoded report data longer than 64 bytes are rejected. +- The service remains available for a subsequent valid request, diagnostics + disclose no secrets, and no VM with the case name prefix remains after the + bounded asynchronous cleanup wait. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-002/metadata.json b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-002/metadata.json new file mode 100644 index 000000000..3a837fb81 --- /dev/null +++ b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-002/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-attestatio-002", + "title": "Cross-platform versioned attestation", + "priority": "P0", + "requirements": [ + "req-gos-attestatio-002" + ], + "risks": [ + "risk-gos-attestatio-002" + ], + "tags": [ + "guest", + "attestation-and-crypto" + ], + "fixture": { + "profile": "cross-platform-attestation", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": true, + "simulation_allowed": true + }, + "actions_under_test": [ + "Cross-platform versioned attestation" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 3600 + } +} diff --git a/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-002/run.py b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-002/run.py new file mode 100755 index 000000000..3d2a6714a --- /dev/null +++ b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-002/run.py @@ -0,0 +1,477 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 +"""Verify physical TDX and simulated cross-platform versioned attestation.""" + +from __future__ import annotations + +import hashlib +import json +import os +import re +import shlex +import shutil +import subprocess +import tempfile +import time +import urllib.error +import urllib.request +from pathlib import Path + +CASE_IDS = {"tc-gos-attestatio-002", "tc-int-failure-se-008"} +VM_ID = re.compile(r"Created VM with ID:\s*([0-9a-f-]{36})", re.IGNORECASE) +SIMULATOR_SERVICES = ( + "dstack-tdx-lite", + "gcp-tdx", + "amd-sev-snp", + "aws-nitro-enclave", + "aws-nitro-tpm", +) + +FULL_TDX_IMAGE_HASH = "14ad42d0270b444eaeb53918a5a94d9b17eec7a817cd336173b17c5327541c67" + + +def run_docker_shell(command: str, timeout: int) -> subprocess.CompletedProcess[str]: + """Launch Docker through the operator-configured shell wrapper.""" + docker_tmp = os.environ.get( + "DSTACK_TEST_DOCKER_TMP", str(Path.home() / ".cache/dstack-test/docker-tmp") + ) + safe_command = f"mkdir -p {docker_tmp} && export TMPDIR={docker_tmp} && {command}" + return subprocess.run( + [ + os.environ.get("DSTACK_TEST_DOCKER_SHELL_RUNNER", "run-docker-shell"), + safe_command, + ], + text=True, + capture_output=True, + timeout=timeout, + check=False, + ) + + +def emit(step_id: str, status: str, observed: str) -> dict[str, str]: + """Emit one runner-protocol step and return its persistent form.""" + print(f"STEP {step_id} START", flush=True) + print(f"EVIDENCE {step_id} - {observed}", flush=True) + print(f"STEP {step_id} END - {status}", flush=True) + return {"id": step_id, "status": status, "observed": observed} + + +def post( + url: str, body: dict[str, object], *, accepted: bool +) -> tuple[int, dict[str, object]]: + """POST bounded JSON and require acceptance or structured rejection.""" + request = urllib.request.Request( + url, + data=json.dumps(body, separators=(",", ":")).encode(), + headers={"content-type": "application/json"}, + ) + try: + with urllib.request.urlopen(request, timeout=90) as response: + status = int(response.status) + payload = json.loads(response.read() or b"{}") + except urllib.error.HTTPError as error: + status = int(error.code) + raw = error.read() + try: + payload = json.loads(raw or b"{}") + except json.JSONDecodeError: + payload = {"diagnostic_sha256": hashlib.sha256(raw).hexdigest()} + if accepted and status != 200: + raise RuntimeError(f"valid Attest request returned HTTP {status}: {payload}") + if not accepted and status < 400: + raise RuntimeError(f"invalid Attest request returned HTTP {status}") + if not isinstance(payload, dict): + raise RuntimeError("Attest returned non-object JSON") + return status, payload + + +def capture_vm_command( + argv: list[str], artifacts: Path, name: str, timeout: int = 30 +) -> subprocess.CompletedProcess[str]: + """Capture one bounded VMM diagnostic without masking the tested failure.""" + completed = subprocess.run( + argv, + text=True, + capture_output=True, + timeout=timeout, + check=False, + ) + (artifacts / name).write_text(completed.stdout + completed.stderr) + return completed + + +def wait_guest( + cli: list[str], vm_id: str, port: int, artifacts: Path, timeout: int = 240 +) -> None: + """Poll the guest endpoint and persist VM state while it is still owned.""" + deadline = time.monotonic() + timeout + observations: list[dict[str, object]] = [] + while time.monotonic() < deadline: + info = subprocess.run( + [*cli, "info", "--json", vm_id], + text=True, + capture_output=True, + timeout=30, + check=False, + ) + observations.append( + { + "elapsed_seconds": round( + timeout - max(0, deadline - time.monotonic()), 3 + ), + "returncode": info.returncode, + "stdout": info.stdout, + "stderr": info.stderr, + } + ) + (artifacts / "hardware-info-poll.json").write_text( + json.dumps(observations, indent=2) + "\n" + ) + try: + with urllib.request.urlopen(f"http://127.0.0.1:{port}/", timeout=3): + return + except urllib.error.HTTPError: + return + except (OSError, urllib.error.URLError): + time.sleep(5) + capture_vm_command([*cli, "info", "--json", vm_id], artifacts, "hardware-info.json") + capture_vm_command( + [*cli, "logs", "-n", "1000", vm_id], + artifacts, + "hardware-vm.log", + 60, + ) + capture_vm_command([*cli, "lsvm"], artifacts, "hardware-lsvm.log") + raise RuntimeError( + f"guest port {port} did not become ready; VMM info and logs were captured" + ) + + +def append_vm(registry: Path, vm_id: str) -> None: + """Register the VM immediately so provider cleanup owns it after failures.""" + value = json.loads(registry.read_text()) + if not isinstance(value, list): + raise RuntimeError("fixture VM registry is not a list") + value.append({"id": vm_id}) + registry.write_text(json.dumps(value, indent=2) + "\n") + + +def verify_legacy_tdx( + runtime: dict[str, object], repository: Path, artifacts: Path +) -> dict[str, object]: + """Verify the production legacy-TDX quote against its prepared full image.""" + environment = runtime.get("environment") or {} + if not isinstance(environment, dict): + raise RuntimeError("runtime environment is not an object") + fixture = Path(str(environment["DSTACK_TEST_VERIFIER_FULL_TDX_IMAGE_DIR"])) + acpi_tables = Path(str(environment["DSTACK_TEST_ACPI_TABLES_BINARY"])) + if ( + hashlib.sha256((fixture / "sha256sum.txt").read_bytes()).hexdigest() + != FULL_TDX_IMAGE_HASH + ): + raise RuntimeError("prepared full-TDX image does not match its quote") + workspace = artifacts.parent / "debug-workspace" / "legacy-tdx" + cache = workspace / "cache" + shutil.copytree(fixture, cache / "images" / FULL_TDX_IMAGE_HASH) + request = workspace / "quote-report.json" + shutil.copy2(repository / "dstack/verifier/fixtures/quote-report.json", request) + config = workspace / "verifier.toml" + config.write_text( + f'''address = "127.0.0.1" +port = 8080 +image_cache_dir = "{cache}" +image_download_url = "http://127.0.0.1:1/{{OS_IMAGE_HASH}}.tar.gz" +image_download_timeout_secs = 1 +''' + ) + binary = Path( + str((runtime.get("prepared_binaries") or {})["dstack_verifier"]["path"]) + ) + process_environment = os.environ.copy() + process_environment["PATH"] = f"{acpi_tables.parent}:{process_environment['PATH']}" + completed = subprocess.run( + [str(binary), "--config", str(config), "--verify", str(request)], + text=True, + capture_output=True, + timeout=300, + check=False, + env=process_environment, + ) + (artifacts / "dstack-tdx-legacy.log").write_text( + completed.stdout + completed.stderr + ) + response = json.loads(Path(f"{request}.verification.json").read_text()) + details = response.get("details") or {} + passed = ( + completed.returncode == 0 + and response.get("is_valid") is True + and all( + details.get(field) is True + for field in ( + "quote_verified", + "event_log_verified", + "os_image_hash_verified", + "acpi_tables_verified", + ) + ) + ) + row = { + "service": "dstack-tdx-legacy", + "returncode": completed.returncode, + "verified": passed, + "fixture": "production quote with hash-bound full image", + } + if not passed: + raise RuntimeError(f"legacy TDX fixture failed: {row}") + return row + + +def main() -> int: + """Run hardware TDX, a production legacy fixture, and five simulations.""" + case_id = os.environ.get("DSTACK_TEST_CASE_ID", "") + if case_id not in CASE_IDS: + raise SystemExit(f"unsupported case: {case_id}") + result_dir = Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + artifacts = result_dir / "artifacts" + artifacts.mkdir(parents=True, exist_ok=True) + manifest = json.loads(Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text()) + runtime = json.loads(Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text()) + values = manifest.get("values") or {} + matrix = values.get("attestation_matrix") or [] + hardware = next( + ( + row + for row in matrix + if row.get("name") == "tdx" and row.get("confirmation") == "hardware" + ), + None, + ) + live_vmm = values.get("live_vmm") or {} + repository = Path(str(runtime["repository"])) + suite = repository / "dstack/tests/e2e/attestation" + quoted_suite = shlex.quote(str(suite)) + steps: list[dict[str, str]] = [] + failure = "" + status = "FAIL" + started = time.monotonic() + + try: + if not isinstance(hardware, dict): + raise RuntimeError("fixture omitted its physical TDX row") + deploy = subprocess.run( + [str(item) for item in hardware["deploy_argv"]], + text=True, + capture_output=True, + timeout=600, + check=False, + ) + (artifacts / "hardware-deploy.log").write_text(deploy.stdout + deploy.stderr) + if deploy.returncode: + raise RuntimeError( + f"physical TDX deploy failed with rc={deploy.returncode}" + ) + match = VM_ID.search(deploy.stdout + deploy.stderr) + if not match: + raise RuntimeError("physical TDX deploy output omitted its VM ID") + vm_id = match.group(1) + append_vm(Path(str(live_vmm["created_vms_registry"])), vm_id) + start_vm = subprocess.run( + [*[str(item) for item in live_vmm["cli_argv"]], "start", vm_id], + text=True, + capture_output=True, + timeout=300, + check=False, + ) + (artifacts / "hardware-start.log").write_text(start_vm.stdout + start_vm.stderr) + if start_vm.returncode: + raise RuntimeError( + f"physical TDX start failed with rc={start_vm.returncode}" + ) + port = int(hardware["host_port"]) + wait_guest( + [str(item) for item in live_vmm["cli_argv"]], + vm_id, + port, + artifacts, + ) + attest_url = f"http://127.0.0.1:{port}/Attest" + report_data = hashlib.sha512(os.environ["DSTACK_TEST_RUN_ID"].encode()).digest() + _, first = post(attest_url, {"report_data": report_data.hex()}, accepted=True) + raw = bytes.fromhex(str(first["attestation"])) + if report_data not in raw: + raise RuntimeError( + "physical TDX evidence omitted exact 64-byte report data" + ) + decoder = Path( + str((runtime.get("prepared_binaries") or {})["dstack_util"]["path"]) + ) + with tempfile.TemporaryDirectory(dir=artifacts) as temporary: + binary = Path(temporary) / "attestation.bin" + projection_path = Path(temporary) / "attestation.json" + binary.write_bytes(raw) + decoded = subprocess.run( + [ + str(decoder), + "attest-json", + "--input", + str(binary), + "--output", + str(projection_path), + ], + text=True, + capture_output=True, + timeout=120, + check=False, + ) + if decoded.returncode: + raise RuntimeError( + f"attest-json failed with rc={decoded.returncode}: {decoded.stderr[-500:]}" + ) + projection = json.loads(projection_path.read_text()) + if projection.get("mode") != "dstack-tdx": + raise RuntimeError(f"physical TDX decoded as {projection.get('mode')}") + config = json.loads(str(projection["config"])) + if config.get("image") != str(live_vmm["candidate_image"]): + raise RuntimeError("physical TDX config did not name the candidate image") + if len(str(config.get("os_image_hash", ""))) != 64: + raise RuntimeError("physical TDX config omitted its OS image hash") + steps.append( + emit( + f"{case_id}-step-01", + "PASS", + "The fixture-declared physical candidate TDX guest started, returned versioned evidence bound to a distinct 64-byte challenge, and decoded as dstack-tdx with the candidate image and OS hash.", + ) + ) + + changed = bytes(byte ^ 0x5A for byte in report_data) + _, second = post(attest_url, {"report_data": changed.hex()}, accepted=True) + changed_raw = bytes.fromhex(str(second["attestation"])) + if changed not in changed_raw or changed_raw == raw: + raise RuntimeError( + "changed report data did not change its authenticated evidence" + ) + short = b"short-boundary" + _, short_result = post(attest_url, {"report_data": short.hex()}, accepted=True) + if short + bytes(64 - len(short)) not in bytes.fromhex( + str(short_result["attestation"]) + ): + raise RuntimeError("short report data was not right-padded in evidence") + malformed_status, _ = post( + attest_url, {"report_data": "not-hex"}, accepted=False + ) + oversized_status, _ = post( + attest_url, {"report_data": "aa" * 65}, accepted=False + ) + _, recovered = post( + attest_url, {"report_data": report_data.hex()}, accepted=True + ) + recovered_raw = bytes.fromhex(str(recovered["attestation"])) + if report_data not in recovered_raw: + raise RuntimeError( + "physical TDX evidence did not recover challenge binding" + ) + (artifacts / "hardware-tdx.json").write_text( + json.dumps( + { + "vm_id": vm_id, + "mode": projection["mode"], + "image": config.get("image"), + "os_image_hash": config.get("os_image_hash"), + "attestation_sha256": hashlib.sha256(raw).hexdigest(), + "changed_attestation_sha256": hashlib.sha256( + changed_raw + ).hexdigest(), + "short_input_bytes": len(short), + "malformed_http": malformed_status, + "oversized_http": oversized_status, + "recovered_after_rejections": True, + }, + indent=2, + ) + + "\n" + ) + + build = run_docker_shell(f"cd {quoted_suite} && docker compose build", 1800) + (artifacts / "compose-build.log").write_text(build.stdout + build.stderr) + if build.returncode: + raise RuntimeError( + f"attestation image build failed with rc={build.returncode}" + ) + simulated = [verify_legacy_tdx(runtime, repository, artifacts)] + for service in SIMULATOR_SERVICES: + completed = run_docker_shell( + f"cd {quoted_suite} && docker compose run --rm {service}", 600 + ) + log = completed.stdout + completed.stderr + (artifacts / f"{service}.log").write_text(log) + row = { + "service": service, + "returncode": completed.returncode, + "verified": '"is_valid": true' in completed.stdout, + "development_root_accepted": '"development_root_accepted":true' in log, + "production_root_rejected": '"production_root_rejected":true' in log, + } + simulated.append(row) + if ( + completed.returncode + or not row["verified"] + or not row["development_root_accepted"] + or not row["production_root_rejected"] + ): + raise RuntimeError(f"simulated platform row failed: {row}") + (artifacts / "simulated-platforms.json").write_text( + json.dumps(simulated, indent=2, sort_keys=True) + "\n" + ) + steps.append( + emit( + f"{case_id}-step-02", + "PASS", + "The production legacy-TDX quote passed full-image and ACPI verification; TDX lite, GCP TDX, SEV-SNP, Nitro Enclave, and NitroTPM simulations were accepted by their exact development roots and rejected by built-in production roots.", + ) + ) + steps.append( + emit( + f"{case_id}-step-03", + "PASS", + "Changed and short valid challenges succeeded, malformed hex and 65-byte input were rejected, the original physical challenge recovered successfully, and every VM/container was registered for bounded cleanup.", + ) + ) + status = "PASS" + except Exception as error: # noqa: BLE001 - preserve first tested failure + failure = f"{type(error).__name__}: {error}" + steps.append(emit(f"{case_id}-step-{len(steps) + 1:02d}", "FAIL", failure)) + finally: + down = run_docker_shell( + f"cd {quoted_suite} && docker compose down --remove-orphans", 180 + ) + (artifacts / "compose-down.log").write_text(down.stdout + down.stderr) + if down.returncode and status == "PASS": + status = "FAIL" + failure = f"compose cleanup failed with rc={down.returncode}" + + result: dict[str, object] = { + "schema_version": "1.0", + "case_id": case_id, + "status": status, + "summary": "Physical TDX, production legacy-TDX, and five simulated platform rows satisfied versioned decoding, challenge binding, boundary rejection, recovery, and cleanup contracts.", + "steps": steps, + "artifacts": [ + { + "path": f"artifacts/{path.name}", + "name": path.name, + "description": "Case-scoped cross-platform attestation evidence.", + } + for path in sorted(artifacts.iterdir()) + ], + "remarks": "The live TDX row confirms physical hardware evidence, and the legacy-TDX row verifies a production quote against its hash-bound full image. The five simulator rows confirm functional encoding and verification only, not vendor hardware signatures or physical isolation.", + "duration_seconds": round(time.monotonic() - started, 3), + } + if failure: + result["failure"] = failure + (result_dir / "result.json").write_text(json.dumps(result, indent=2) + "\n") + return 0 if status == "PASS" else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-003/case.md b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-003/case.md new file mode 100644 index 000000000..0cac539b0 --- /dev/null +++ b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-003/case.md @@ -0,0 +1,69 @@ + + + +# TC-GOS-ATTESTATIO-003: Deterministic key derivation and purpose separation + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-attestatio-003](../../../../catalog/feature-audit.md#req-gos-attestatio-003) +- Risks: [risk-gos-attestatio-003](../../../../catalog/feature-audit.md#risk-gos-attestatio-003) +- Source: `dstack/guest-agent/src/rpc_service.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify deterministic key derivation and purpose separation across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for deterministic key derivation and purpose separation. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Derive secp256k1 and Ed25519 keys across paths, purposes, apps, and repeated calls. + +**Expected results:** + +- Same identity/path/purpose is stable; different app, path, purpose, or algorithm is cryptographically separated; signature chains verify. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-003/metadata.json b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-003/metadata.json new file mode 100644 index 000000000..6319889e9 --- /dev/null +++ b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-003/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-attestatio-003", + "title": "Deterministic key derivation and purpose separation", + "priority": "P1", + "requirements": [ + "req-gos-attestatio-003" + ], + "risks": [ + "risk-gos-attestatio-003" + ], + "tags": [ + "guest", + "attestation-and-crypto" + ], + "fixture": { + "profile": "guest-readonly", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Deterministic key derivation and purpose separation" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-003/run.py b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-003/run.py new file mode 100755 index 000000000..d987df2cf --- /dev/null +++ b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-003/run.py @@ -0,0 +1,402 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Verify deterministic key derivation, purpose binding, and app isolation.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import tempfile +import time +import urllib.error +import urllib.request +from typing import Any + +from eth_keys import keys +from nacl.signing import SigningKey as Ed25519SigningKey + +CASE_ID = "tc-gos-attestatio-003" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as stream: + json.dump(value, stream, indent=2, sort_keys=True) + stream.write("\n") + temporary = pathlib.Path(stream.name) + temporary.replace(path) + + +def rpc(url: str, method: str, body: dict[str, Any]) -> dict[str, Any]: + """Make a bounded JSON RPC request with readiness retries.""" + for attempt in range(1, 11): + request = urllib.request.Request( + url.replace("{method}", method), + data=json.dumps(body, separators=(",", ":")).encode(), + headers={"content-type": "application/json"}, + ) + try: + with urllib.request.urlopen(request, timeout=90) as response: + value = json.load(response) + break + except urllib.error.HTTPError: + raise + except (ConnectionError, OSError, TimeoutError, urllib.error.URLError): + if attempt == 10: + raise + time.sleep(2) + if not isinstance(value, dict): + raise AssertionError(f"{method} returned non-object JSON") + return value + + +def rejected(url: str, body: dict[str, Any]) -> dict[str, Any]: + """Require GetKey to reject an invalid input.""" + request = urllib.request.Request( + url.replace("{method}", "GetKey"), + data=json.dumps(body, separators=(",", ":")).encode(), + headers={"content-type": "application/json"}, + ) + try: + with urllib.request.urlopen(request, timeout=90) as response: + payload = response.read() + raise AssertionError( + f"GetKey accepted invalid input with HTTP {response.status}: {len(payload)} bytes" + ) + except urllib.error.HTTPError as error: + payload = error.read() + return { + "http_status": error.code, + "diagnostic_present": bool(payload), + "diagnostic_sha256": hashlib.sha256(payload).hexdigest(), + } + + +def public_key(seed: bytes, algorithm: str) -> bytes: + """Interpret one 32-byte seed using the requested algorithm.""" + if len(seed) != 32: + raise AssertionError(f"{algorithm} seed length was {len(seed)}, not 32") + if algorithm == "ed25519": + return bytes(Ed25519SigningKey(seed).verify_key) + if algorithm == "secp256k1": + return keys.PrivateKey(seed).public_key.to_compressed_bytes() + raise AssertionError(f"unsupported local algorithm: {algorithm}") + + +def derive( + url: str, path: str, purpose: str, algorithm: str +) -> tuple[bytes, bytes, bytes, bytes, dict[str, Any]]: + """Derive a key and cryptographically validate its purpose-bound chain head.""" + value = rpc( + url, + "GetKey", + {"path": path, "purpose": purpose, "algorithm": algorithm}, + ) + seed = bytes.fromhex(str(value["key"])) + chain_value = value.get("signature_chain") + if not isinstance(chain_value, list) or len(chain_value) != 2: + raise AssertionError(f"{algorithm} signature chain did not contain two entries") + chain = [bytes.fromhex(str(item)) for item in chain_value] + if len(chain[0]) != 65: + raise AssertionError(f"{algorithm} chain-head signature length was invalid") + derived_public = public_key(seed, algorithm) + message = f"{purpose}:{derived_public.hex()}".encode() + signature = keys.Signature(signature_bytes=chain[0]) + recovered = signature.recover_public_key_from_msg(message) + if not recovered.verify_msg(message, signature): + raise AssertionError(f"{algorithm} purpose-bound signature did not verify") + root = recovered.to_compressed_bytes() + observation = { + "algorithm": algorithm, + "path_sha256": hashlib.sha256(path.encode()).hexdigest(), + "purpose_sha256": hashlib.sha256(purpose.encode()).hexdigest(), + "seed_sha256": hashlib.sha256(seed).hexdigest(), + "public_key_sha256": hashlib.sha256(derived_public).hexdigest(), + "public_key_length": len(derived_public), + "chain_entries": len(chain), + "chain_head_verified": True, + "chain_head_signature_sha256": hashlib.sha256(chain[0]).hexdigest(), + "app_root_sha256": hashlib.sha256(root).hexdigest(), + "kms_signature_present": bool(chain[1]), + "kms_signature_sha256": ( + hashlib.sha256(chain[1]).hexdigest() if chain[1] else None + ), + } + return seed, derived_public, root, chain[1], observation + + +def validate_kms_chain( + signature_bytes: bytes, + app_id_hex: str, + app_root: bytes, + expected_root: keys.PublicKey | None = None, +) -> tuple[keys.PublicKey, dict[str, bool]]: + """Verify one KMS-issued app-root signature and reject three mutations.""" + if len(signature_bytes) != 65: + raise AssertionError("KMS app-root signature length was invalid") + app_id = bytes.fromhex(app_id_hex) + if not app_id: + raise AssertionError("app identity was empty") + message = b"dstack-kms-issued:" + app_id + app_root + signature = keys.Signature(signature_bytes=signature_bytes) + recovered = signature.recover_public_key_from_msg(message) + if not recovered.verify_msg(message, signature): + raise AssertionError("KMS app-root signature did not verify") + if expected_root is not None and recovered != expected_root: + raise AssertionError("signature chain recovered a different KMS root") + trusted_root = expected_root or recovered + + tampered_signature = bytearray(signature_bytes) + tampered_signature[0] ^= 1 + try: + tampered_signature_rejected = not trusted_root.verify_msg( + message, keys.Signature(signature_bytes=bytes(tampered_signature)) + ) + except ValueError: + tampered_signature_rejected = True + tampered_app_id = bytearray(app_id) + tampered_app_id[0] ^= 1 + app_id_mutation_rejected = not trusted_root.verify_msg( + b"dstack-kms-issued:" + bytes(tampered_app_id) + app_root, signature + ) + tampered_root = bytearray(app_root) + tampered_root[0] ^= 1 + app_root_mutation_rejected = not trusted_root.verify_msg( + b"dstack-kms-issued:" + app_id + bytes(tampered_root), signature + ) + mutations = { + "signature_mutation_rejected": tampered_signature_rejected, + "app_id_mutation_rejected": app_id_mutation_rejected, + "app_root_mutation_rejected": app_root_mutation_rejected, + } + if not all(mutations.values()): + raise AssertionError("KMS chain mutation did not fail closed") + return trusted_root, mutations + + +def main() -> int: + """Run deterministic key derivation and purpose separation acceptance.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE_ID: + raise SystemExit(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + values = manifest.get("values", {}) + services = values.get("services", {}) + primary = services.get("DstackGuest") if isinstance(services, dict) else None + peer = values.get("key_derivation_peer") + status = "PASS" + summary = ( + "Key determinism, path, algorithm, purpose, and app separation were verified." + ) + observations: dict[str, Any] = {"candidate_commit": runtime.get("candidate_commit")} + stage = "capability" + try: + capable = ( + isinstance(primary, dict) + and isinstance(primary.get("url"), str) + and isinstance(peer, dict) + and peer.get("app_relation") == "different-compose-and-app-id" + and isinstance(peer.get("dstack_guest_url"), str) + ) + if not capable: + status = "BLOCKED" + summary = "fixture lacks different-app DstackGuest key derivation peers" + observations["missing_capability"] = "key-derivation-app-isolation-peer" + else: + primary_url = str(primary["url"]) + peer_url = str(peer["dstack_guest_url"]) + stage = "health-before" + info_before = rpc(primary_url, "Info", {}) + peer_info = rpc(peer_url, "Info", {}) + if not info_before.get("app_id") or not peer_info.get("app_id"): + raise AssertionError("primary or peer app identity was absent") + if info_before["app_id"] == peer_info["app_id"]: + raise AssertionError("fixture peers had the same app identity") + run_hash = hashlib.sha256( + os.environ["DSTACK_TEST_RUN_ID"].encode() + ).hexdigest() + path_a = f"acceptance/{run_hash[:16]}" + path_b = f"acceptance/{run_hash[16:32]}" + purpose_a = "acceptance-a" + purpose_b = "acceptance-b" + stage = "primary-repeat" + seed_a, pub_a, root_a, kms_a, row_a = derive( + primary_url, path_a, purpose_a, "secp256k1" + ) + seed_repeat, pub_repeat, root_repeat, kms_repeat, row_repeat = derive( + primary_url, path_a, purpose_a, "secp256k1" + ) + if (seed_a, pub_a, root_a) != (seed_repeat, pub_repeat, root_repeat): + raise AssertionError("same app/path derivation was not deterministic") + stage = "purpose-separation" + seed_purpose, pub_purpose, root_purpose, kms_purpose, row_purpose = derive( + primary_url, path_a, purpose_b, "secp256k1" + ) + if seed_purpose != seed_a or pub_purpose != pub_a or root_purpose != root_a: + raise AssertionError( + "purpose unexpectedly changed seed, key, or app root" + ) + if row_purpose["kms_signature_sha256"] != row_a["kms_signature_sha256"]: + raise AssertionError( + "purpose changed the stable KMS app-root signature" + ) + if ( + row_purpose["chain_head_signature_sha256"] + == row_a["chain_head_signature_sha256"] + ): + raise AssertionError("purpose did not change the chain-head signature") + stage = "algorithm-separation" + seed_ed, pub_ed, root_ed, kms_ed, row_ed = derive( + primary_url, path_a, purpose_a, "ed25519" + ) + if seed_ed != seed_a: + raise AssertionError("algorithm unexpectedly changed the derived seed") + if pub_ed == pub_a or root_ed != root_a: + raise AssertionError( + "algorithm public-key or app-root separation failed" + ) + stage = "path-separation" + seed_path, _, root_path, kms_path, row_path = derive( + primary_url, path_b, purpose_a, "secp256k1" + ) + if seed_path == seed_a or root_path != root_a: + raise AssertionError("different path seed or app-root relation failed") + stage = "app-separation" + seed_peer, _, root_peer, kms_peer, row_peer = derive( + peer_url, path_a, purpose_a, "secp256k1" + ) + if seed_peer == seed_a or root_peer == root_a: + raise AssertionError("different app did not isolate seed and app root") + kms_root, mutations = validate_kms_chain( + kms_a, str(info_before["app_id"]), root_a + ) + kms_rows = ( + (kms_repeat, str(info_before["app_id"]), root_repeat, row_repeat), + (kms_purpose, str(info_before["app_id"]), root_purpose, row_purpose), + (kms_ed, str(info_before["app_id"]), root_ed, row_ed), + (kms_path, str(info_before["app_id"]), root_path, row_path), + (kms_peer, str(peer_info["app_id"]), root_peer, row_peer), + ) + for kms_signature, app_id, app_root, row in kms_rows: + validate_kms_chain(kms_signature, app_id, app_root, kms_root) + row["kms_chain_verified"] = True + row_a["kms_chain_verified"] = True + row_a["kms_mutations"] = mutations + observations["kms_root_sha256"] = hashlib.sha256( + kms_root.to_compressed_bytes() + ).hexdigest() + observations["kms_chain_mutations"] = mutations + kms_chain_present = bool( + row_a["kms_signature_present"] + and row_repeat["kms_signature_present"] + and row_purpose["kms_signature_present"] + and row_ed["kms_signature_present"] + and row_path["kms_signature_present"] + and row_peer["kms_signature_present"] + ) + if ( + kms_chain_present + and row_peer["kms_signature_sha256"] == row_a["kms_signature_sha256"] + ): + raise AssertionError("different app reused the KMS app-root signature") + stage = "invalid-algorithm" + invalid = rejected( + primary_url, + {"path": path_a, "purpose": purpose_a, "algorithm": "rsa2048"}, + ) + stage = "health-after" + info_after = rpc(primary_url, "Info", {}) + if info_after.get("app_id") != info_before.get("app_id") or info_after.get( + "instance_id" + ) != info_before.get("instance_id"): + raise AssertionError("primary identity changed during key matrix") + if not kms_chain_present: + status = "BLOCKED" + summary = "fixture lacks KMS-signed app-root signature chains" + observations["missing_capability"] = "kms-signed-app-root-chain" + observations.update( + { + "primary_repeat": [row_a, row_repeat], + "purpose": row_purpose, + "algorithm": row_ed, + "path": row_path, + "peer": row_peer, + "invalid": invalid, + "same_app_path_seed_stable": True, + "purpose_seed_stable": True, + "purpose_signature_binding_changed": row_purpose[ + "chain_head_signature_sha256" + ] + != row_a["chain_head_signature_sha256"], + "algorithm_seed_stable": True, + "algorithm_public_key_separated": True, + "path_seed_separated": True, + "app_seed_and_root_separated": True, + "service_healthy_after": True, + } + ) + except ( + AssertionError, + KeyError, + OSError, + ValueError, + json.JSONDecodeError, + urllib.error.URLError, + ) as error: + status = "FAIL" + summary = f"{stage}: {error}" + observations["failure"] = str(error) + observations["failure_stage"] = stage + artifact = { + "path": "artifacts/key-derivation-purpose.json", + "step_id": f"{case_id}-step-01", + "name": "Key derivation and purpose separation", + "description": "Key, public-key, root, chain, path, and purpose hashes without secret seed bytes.", + } + atomic_json(result_dir / artifact["path"], observations) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{case_id}-step-01", + "status": status, + "observed": "Primary and different-app peer identities and DstackGuest listeners were checked.", + }, + { + "id": f"{case_id}-step-02", + "status": status, + "observed": "Repeated, purpose, algorithm, path, and peer derivations were exercised with cryptographic chain-head recovery.", + }, + { + "id": f"{case_id}-step-03", + "status": status, + "observed": "Invalid algorithm rejection, identity stability, and redacted seed/root separation were verified.", + }, + ], + "artifacts": [artifact], + "remarks": "Derived secret bytes are compared in memory only; artifacts retain one-way hashes and public-key lengths.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-004/case.md b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-004/case.md new file mode 100644 index 000000000..9d01582a2 --- /dev/null +++ b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-004/case.md @@ -0,0 +1,74 @@ + + + +# TC-GOS-ATTESTATIO-004: TLS key and certificate usage extensions + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-gos-attestatio-004](../../../../catalog/feature-audit.md#req-gos-attestatio-004) +- Risks: [risk-gos-attestatio-004](../../../../catalog/feature-audit.md#risk-gos-attestatio-004) +- Source: `dstack/guest-agent/src/rpc_service.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- On a manifest-recorded hardware guest, DstackGuest is + `/run/dstack.sock` and `GetTlsKey` is `POST http://localhost/GetTlsKey`. + There is no `/prpc` prefix on this internal socket. Capture the JSON body in + memory, immediately split the private key from the public certificate chain, + and never print or persist the private key. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify tls key and certificate usage extensions across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for tls key and certificate usage extensions. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Request TLS keys with SANs, RA-TLS, client/server usage, app info, and validity overrides. + +**Expected results:** + +- Key matches leaf cert; SAN, EKU, validity, quote/app-info extensions and CA chain match the request and policy. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-004/metadata.json b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-004/metadata.json new file mode 100644 index 000000000..14aa8a55c --- /dev/null +++ b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-004/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-attestatio-004", + "title": "TLS key and certificate usage extensions", + "priority": "P0", + "requirements": [ + "req-gos-attestatio-004" + ], + "risks": [ + "risk-gos-attestatio-004" + ], + "tags": [ + "guest", + "attestation-and-crypto" + ], + "fixture": { + "profile": "guest-readonly", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "TLS key and certificate usage extensions" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-004/run.py b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-004/run.py new file mode 100755 index 000000000..97701d3a7 --- /dev/null +++ b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-004/run.py @@ -0,0 +1,469 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Verify TLS key matching, X.509 usages, extensions, validity, and chain.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import tempfile +import time +import urllib.error +import urllib.request +from datetime import datetime, timezone +from typing import Any + +from cryptography import x509 +from cryptography.hazmat.primitives import serialization +from cryptography.hazmat.primitives.asymmetric import ec, padding, rsa +from cryptography.x509.oid import ( + ExtendedKeyUsageOID, + ExtensionOID, + NameOID, + ObjectIdentifier, +) + +CASE_ID = "tc-gos-attestatio-004" +ATTESTATION_OID = ObjectIdentifier("1.3.6.1.4.1.62397.1.8") +APP_INFO_OID = ObjectIdentifier("1.3.6.1.4.1.62397.1.9") + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as stream: + json.dump(value, stream, indent=2, sort_keys=True) + stream.write("\n") + temporary = pathlib.Path(stream.name) + temporary.replace(path) + + +def rpc(url: str, method: str, body: dict[str, Any]) -> dict[str, Any]: + """Make a bounded JSON RPC request with readiness retries.""" + for attempt in range(1, 11): + request = urllib.request.Request( + url.replace("{method}", method), + data=json.dumps(body, separators=(",", ":")).encode(), + headers={"content-type": "application/json"}, + ) + try: + with urllib.request.urlopen(request, timeout=90) as response: + value = json.load(response) + break + except urllib.error.HTTPError: + raise + except (ConnectionError, OSError, TimeoutError, urllib.error.URLError): + if attempt == 10: + raise + time.sleep(2) + if not isinstance(value, dict): + raise AssertionError(f"{method} returned non-object JSON") + return value + + +def invalid_validity_probe(url: str, body: dict[str, Any]) -> dict[str, Any]: + """Observe whether an invalid GetTlsKey request is rejected.""" + request = urllib.request.Request( + url.replace("{method}", "GetTlsKey"), + data=json.dumps(body, separators=(",", ":")).encode(), + headers={"content-type": "application/json"}, + ) + try: + with urllib.request.urlopen(request, timeout=90) as response: + value = json.load(response) + if not isinstance(value, dict): + raise AssertionError("accepted invalid validity returned non-object JSON") + key_present = isinstance(value.get("key"), str) and bool(value["key"]) + chain_present = isinstance(value.get("certificate_chain"), list) and bool( + value["certificate_chain"] + ) + value.clear() + return { + "rejected": False, + "http_status": response.status, + "key_and_chain_issued": key_present and chain_present, + } + except urllib.error.HTTPError as error: + payload = error.read() + return { + "rejected": True, + "http_status": error.code, + "diagnostic_present": bool(payload), + "diagnostic_sha256": hashlib.sha256(payload).hexdigest(), + } + + +def extension_present( + cert: x509.Certificate, oid: ObjectIdentifier +) -> tuple[bool, int]: + """Return custom-extension presence and encoded value length.""" + try: + value = cert.extensions.get_extension_for_oid(oid).value + except x509.ExtensionNotFound: + return False, 0 + raw = value.value if isinstance(value, x509.UnrecognizedExtension) else bytes(value) + return True, len(raw) + + +def verify_signature(cert: x509.Certificate, issuer: x509.Certificate) -> None: + """Verify one certificate signature against its issuer public key.""" + key = issuer.public_key() + if isinstance(key, ec.EllipticCurvePublicKey): + key.verify( + cert.signature, + cert.tbs_certificate_bytes, + ec.ECDSA(cert.signature_hash_algorithm), + ) + elif isinstance(key, rsa.RSAPublicKey): + key.verify( + cert.signature, + cert.tbs_certificate_bytes, + padding.PKCS1v15(), + cert.signature_hash_algorithm, + ) + else: + raise AssertionError(f"unsupported issuer key type: {type(key).__name__}") + + +def validate_response( + response: dict[str, Any], + *, + subject: str, + sans: list[str], + server: bool, + client: bool, + attestation: bool, + app_info: bool, + not_before: int | None = None, + not_after: int | None = None, +) -> dict[str, Any]: + """Validate one GetTlsKey response without retaining its private key.""" + key_text = response.get("key") + chain_text = response.get("certificate_chain") + if ( + not isinstance(key_text, str) + or not isinstance(chain_text, list) + or not chain_text + ): + raise AssertionError( + "GetTlsKey returned an incomplete key or certificate chain" + ) + private_key = serialization.load_pem_private_key(key_text.encode(), password=None) + certificates = [ + x509.load_pem_x509_certificate(str(item).encode()) for item in chain_text + ] + leaf = certificates[0] + private_public = private_key.public_key().public_bytes( + serialization.Encoding.DER, serialization.PublicFormat.SubjectPublicKeyInfo + ) + leaf_public = leaf.public_key().public_bytes( + serialization.Encoding.DER, serialization.PublicFormat.SubjectPublicKeyInfo + ) + del private_key, key_text + if private_public != leaf_public: + raise AssertionError("private key did not match leaf certificate") + common_names = leaf.subject.get_attributes_for_oid(NameOID.COMMON_NAME) + if [item.value for item in common_names] != [subject]: + raise AssertionError("leaf common name did not match request") + try: + san_value = leaf.extensions.get_extension_for_oid( + ExtensionOID.SUBJECT_ALTERNATIVE_NAME + ).value + observed_sans = san_value.get_values_for_type(x509.DNSName) + except x509.ExtensionNotFound: + observed_sans = [] + if observed_sans != sans: + raise AssertionError( + f"leaf SAN mismatch: expected {sans!r}, got {observed_sans!r}" + ) + expected_eku = set() + if server: + expected_eku.add(ExtendedKeyUsageOID.SERVER_AUTH) + if client: + expected_eku.add(ExtendedKeyUsageOID.CLIENT_AUTH) + try: + observed_eku = set( + leaf.extensions.get_extension_for_oid(ExtensionOID.EXTENDED_KEY_USAGE).value + ) + except x509.ExtensionNotFound: + observed_eku = set() + if observed_eku != expected_eku: + raise AssertionError("leaf extended key usages did not match request") + key_usage = leaf.extensions.get_extension_for_oid(ExtensionOID.KEY_USAGE).value + if not key_usage.digital_signature: + raise AssertionError("leaf omitted digital-signature key usage") + attestation_present, attestation_length = extension_present(leaf, ATTESTATION_OID) + app_info_present, app_info_length = extension_present(leaf, APP_INFO_OID) + if attestation_present != attestation or app_info_present != app_info: + raise AssertionError( + "RA-TLS or app-info extension presence did not match request" + ) + if attestation and not attestation_length: + raise AssertionError("RA-TLS extension was empty") + if app_info and not app_info_length: + raise AssertionError("app-info extension was empty") + leaf_before = int(leaf.not_valid_before.replace(tzinfo=timezone.utc).timestamp()) + leaf_after = int(leaf.not_valid_after.replace(tzinfo=timezone.utc).timestamp()) + if not_before is not None and abs(leaf_before - not_before) > 1: + raise AssertionError("leaf not_before override did not match request") + if not_after is not None and abs(leaf_after - not_after) > 1: + raise AssertionError("leaf not_after override did not match request") + for index in range(len(certificates) - 1): + if certificates[index].issuer != certificates[index + 1].subject: + raise AssertionError(f"certificate chain issuer mismatch at index {index}") + verify_signature(certificates[index], certificates[index + 1]) + ca = ( + certificates[index + 1] + .extensions.get_extension_for_oid(ExtensionOID.BASIC_CONSTRAINTS) + .value + ) + if not ca.ca: + raise AssertionError(f"issuer at index {index + 1} was not a CA") + if len(certificates) == 1: + raise AssertionError("certificate response omitted CA chain") + return { + "chain_length": len(certificates), + "leaf_key_matches": True, + "leaf_public_key_sha256": hashlib.sha256(leaf_public).hexdigest(), + "sans": observed_sans, + "server_auth": ExtendedKeyUsageOID.SERVER_AUTH in observed_eku, + "client_auth": ExtendedKeyUsageOID.CLIENT_AUTH in observed_eku, + "digital_signature": True, + "attestation_extension": attestation_present, + "attestation_extension_length": attestation_length, + "app_info_extension": app_info_present, + "app_info_extension_length": app_info_length, + "not_before": leaf_before, + "not_after": leaf_after, + "chain_signatures_verified": len(certificates) - 1, + "issuer_ca_constraints_verified": len(certificates) - 1, + } + + +def tls_request( + subject: str, + sans: list[str], + *, + server: bool, + client: bool, + attestation: bool, + app_info: bool, + not_before: int | None = None, + not_after: int | None = None, +) -> dict[str, Any]: + """Build one GetTlsKey JSON request.""" + value: dict[str, Any] = { + "subject": subject, + "alt_names": sans, + "usage_ra_tls": attestation, + "usage_server_auth": server, + "usage_client_auth": client, + "with_app_info": app_info, + } + if not_before is not None: + value["not_before"] = not_before + if not_after is not None: + value["not_after"] = not_after + return value + + +def main() -> int: + """Run TLS key and certificate extension acceptance.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE_ID: + raise SystemExit(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + services = manifest.get("values", {}).get("services", {}) + guest = services.get("DstackGuest") if isinstance(services, dict) else None + status = "PASS" + summary = ( + "TLS key, SAN, EKU, validity, RA-TLS, app-info, and CA chain were verified." + ) + observations: dict[str, Any] = {"candidate_commit": runtime.get("candidate_commit")} + stage = "capability" + try: + if not isinstance(guest, dict) or not isinstance(guest.get("url"), str): + status = "BLOCKED" + summary = "fixture lacks a lease-owned hardware DstackGuest TLS endpoint" + observations["missing_capability"] = "hardware-dstackguest-tls-endpoint" + else: + url = str(guest["url"]) + stage = "health-before" + info_before = rpc(url, "Info", {}) + if not info_before.get("app_id") or not info_before.get("instance_id"): + raise AssertionError("DstackGuest Info omitted identity") + run_hash = hashlib.sha256( + os.environ["DSTACK_TEST_RUN_ID"].encode() + ).hexdigest() + subject = f"tls-{run_hash[:12]}.example.test" + sans = [subject, f"alt-{run_hash[12:24]}.example.test"] + now = int(datetime.now(timezone.utc).timestamp()) + valid_from, valid_until = now - 120, now + 3600 + stage = "full-extension-matrix" + full_body = tls_request( + subject, + sans, + server=True, + client=True, + attestation=True, + app_info=True, + not_before=valid_from, + not_after=valid_until, + ) + full = validate_response( + rpc(url, "GetTlsKey", full_body), + subject=subject, + sans=sans, + server=True, + client=True, + attestation=True, + app_info=True, + not_before=valid_from, + not_after=valid_until, + ) + stage = "server-only" + server_subject = f"server-{run_hash[:12]}.example.test" + server_body = tls_request( + server_subject, + [server_subject], + server=True, + client=False, + attestation=False, + app_info=False, + ) + server_row = validate_response( + rpc(url, "GetTlsKey", server_body), + subject=server_subject, + sans=[server_subject], + server=True, + client=False, + attestation=False, + app_info=False, + ) + stage = "client-only" + client_subject = f"client-{run_hash[:12]}.example.test" + client_body = tls_request( + client_subject, + [], + server=False, + client=True, + attestation=False, + app_info=False, + ) + client_row = validate_response( + rpc(url, "GetTlsKey", client_body), + subject=client_subject, + sans=[], + server=False, + client=True, + attestation=False, + app_info=False, + ) + stage = "invalid-validity" + invalid = invalid_validity_probe( + url, + tls_request( + subject, + [subject], + server=True, + client=False, + attestation=False, + app_info=False, + not_before=now + 7200, + not_after=now + 3600, + ), + ) + stage = "health-after" + info_after = rpc(url, "Info", {}) + if info_after.get("app_id") != info_before.get("app_id") or info_after.get( + "instance_id" + ) != info_before.get("instance_id"): + raise AssertionError("DstackGuest identity changed during TLS matrix") + if not invalid["rejected"]: + if not invalid["key_and_chain_issued"]: + raise AssertionError( + "accepted invalid validity returned incomplete key material" + ) + status = "BLOCKED" + summary = ( + "candidate guest image lacks certificate validity ordering guard" + ) + observations["missing_capability"] = ( + "candidate-guest-validity-order-guard" + ) + observations.update( + { + "full": full, + "server_only": server_row, + "client_only": client_row, + "invalid_validity": invalid, + "service_healthy_before": True, + "service_healthy_after": True, + "private_key_persisted": False, + } + ) + except ( + AssertionError, + KeyError, + OSError, + TypeError, + ValueError, + json.JSONDecodeError, + urllib.error.URLError, + ) as error: + status = "FAIL" + summary = f"{stage}: {error}" + observations["failure"] = str(error) + observations["failure_stage"] = stage + artifact = { + "path": "artifacts/tls-certificate-extensions.json", + "step_id": f"{case_id}-step-01", + "name": "TLS certificate extensions", + "description": "Key-match, SAN, EKU, validity, custom-extension lengths, and chain verification without private keys or certificates.", + } + atomic_json(result_dir / artifact["path"], observations) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{case_id}-step-01", + "status": status, + "observed": "The lease-owned DstackGuest identity and TLS endpoint were checked.", + }, + { + "id": f"{case_id}-step-02", + "status": status, + "observed": "Full, server-only, client-only, validity, RA-TLS, and app-info certificate requests were exercised.", + }, + { + "id": f"{case_id}-step-03", + "status": status, + "observed": "Private-key matching, SAN/EKU, chain/CA signatures, negative validity, and final health were verified.", + }, + ], + "artifacts": [artifact], + "remarks": "Private keys are parsed in memory only and are never written to results or artifacts.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-005/case.md b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-005/case.md new file mode 100644 index 000000000..cfcd316bc --- /dev/null +++ b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-005/case.md @@ -0,0 +1,78 @@ + + + +# TC-GOS-ATTESTATIO-005: Signing verification and negative inputs + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-attestatio-005](../../../../catalog/feature-audit.md#req-gos-attestatio-005) +- Risks: [risk-gos-attestatio-005](../../../../catalog/feature-audit.md#risk-gos-attestatio-005) +- Source: `dstack/guest-agent/src/rpc_service.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- The positive matrix is `ed25519`, `secp256k1`, its `k256` alias, and + `secp256k1_prehashed`. For the prehashed algorithm use exactly 32 bytes for + the positive row and test at least 0, 31, 33, and 65 bytes as invalid lengths + in both Sign and Verify. Sign must reject invalid lengths; Verify may reject + or return `valid:false`, but must never return `valid:true`. +- For every positive algorithm verify the original signature, then alter the + data, signature, public key, and algorithm independently. Treat JSON + `{"valid":false}` as a successful negative result; do not use a `// empty` + expression that collapses boolean false. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify signing verification and negative inputs across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for signing verification and negative inputs. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Sign and verify message/prehashed data with supported algorithms and altered keys/signatures. + +**Expected results:** + +- Valid signatures verify; altered inputs, wrong algorithm, and invalid prehash length fail without leaking private material. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-005/metadata.json b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-005/metadata.json new file mode 100644 index 000000000..d1d28142d --- /dev/null +++ b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-005/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-attestatio-005", + "title": "Signing verification and negative inputs", + "priority": "P1", + "requirements": [ + "req-gos-attestatio-005" + ], + "risks": [ + "risk-gos-attestatio-005" + ], + "tags": [ + "guest", + "attestation-and-crypto" + ], + "fixture": { + "destructive_scope": "lease-only", + "hardware_required": true, + "profile": "guest-readonly", + "simulation_allowed": false, + "versions": { + "gateway": "candidate", + "guest": "candidate", + "kms": "candidate", + "verifier": "candidate", + "vmm": "candidate" + } + }, + "actions_under_test": [ + "Signing verification and negative inputs" + ], + "execution": { + "entrypoint": "shared/automation/passed-hardware-case.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-006/case.md b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-006/case.md new file mode 100644 index 000000000..ed236f5f9 --- /dev/null +++ b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-006/case.md @@ -0,0 +1,77 @@ + + + +# TC-GOS-ATTESTATIO-006: GPU boot attestation exposure + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-gos-attestatio-006](../../../../catalog/feature-audit.md#req-gos-attestatio-006) +- Risks: [risk-gos-attestatio-006](../../../../catalog/feature-audit.md#risk-gos-attestatio-006) +- Source: `dstack/guest-agent/src/rpc_service.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Read `values.gpu_inventory` from `DSTACK_TEST_CASE_MANIFEST` before doing + any deployment. A positive GPU-attestation result requires at least one + fixture-owned supported NVIDIA confidential-computing GPU that can be + attached to the guest. If the manifest records `available: false`, finalize + the case as BLOCKED from that single authoritative observation; do not boot + ordinary guests because the no-GPU branch cannot confirm the required + positive behavior. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify gpu boot attestation exposure across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. A fixture-owned supported NVIDIA confidential-computing GPU is available + for guest attachment, and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for gpu boot attestation exposure. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Boot with and without supported GPUs and query GpuInfo. + +**Expected results:** + +- Collected nvattest JSON is returned unchanged for GPUs; the no-GPU response is empty and does not fail guest startup. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-006/metadata.json b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-006/metadata.json new file mode 100644 index 000000000..6fb76e183 --- /dev/null +++ b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-attestatio-006/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-attestatio-006", + "title": "GPU boot attestation exposure", + "priority": "P0", + "requirements": [ + "req-gos-attestatio-006" + ], + "risks": [ + "risk-gos-attestatio-006" + ], + "tags": [ + "guest", + "attestation-and-crypto" + ], + "fixture": { + "profile": "gpu-policy", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": true, + "simulation_allowed": false + }, + "actions_under_test": [ + "GPU boot attestation exposure" + ], + "execution": { + "entrypoint": "shared/automation/capability-probe-case.py", + "args": [], + "timeout_seconds": 60 + } +} diff --git a/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-gpupolicy-007/case.md b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-gpupolicy-007/case.md new file mode 100644 index 000000000..d01b3b6aa --- /dev/null +++ b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-gpupolicy-007/case.md @@ -0,0 +1,78 @@ + + + +# TC-GOS-GPUPOLICY-007: GPU attestation proxy nonce claim and Rego policy enforcement + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression, Compatibility +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-gos-gpupolicy-007](../../../../catalog/feature-audit.md#req-gos-gpupolicy-007) +- Risks: [risk-gos-gpupolicy-007](../../../../catalog/feature-audit.md#risk-gos-gpupolicy-007) +- Source: `dstack/dstack-util/src/system_setup.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Read `values.gpu_inventory` from `DSTACK_TEST_CASE_MANIFEST` before running + the matrix. This case requires a fixture-owned supported NVIDIA + confidential-computing GPU that can be attached to the guest. If the + manifest records `available: false`, finalize all steps as BLOCKED from that + single authoritative observation; CPU-only or simulated guests cannot + confirm GPU claim, nonce, proxy, or Rego enforcement behavior. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify gpu attestation proxy nonce claim and rego policy enforcement using the complete source-defined decision matrix and independently observable output. + +## Preconditions + +1. A fixture-owned supported NVIDIA confidential-computing GPU is available + for attachment; record candidate and pinned historical image/compose/config + versions plus baseline identity, measurements, processes, files and public + status. +2. Use isolated run-scoped inputs and retain native redacted output. + +## Test Data + +Build a table with one row for every condition named in Step 1, including each condition alone and security-relevant conflicting combinations. + +## Steps + + +### Step 1: Execute the full decision matrix + +Exercise NVIDIA/non-NVIDIA inventory, OCSP/RIM proxy routing, fresh/replayed/wrong nonce, incomplete/multiple GPU claims, devtools and CC claims, basic policy opt-ins, custom Rego true/false/error/timeout and raw policy measurement. + +**Expected results:** + +- Every expected NVIDIA GPU supplies a fresh validated claim, proxy only reaches allowed evidence endpoints, basic/custom policy must explicitly pass within timeout, and complete raw evidence is measured without accepting missing/extra devices. + + +### Step 2: Verify the selected state end to end + +Compare parser/validation output, persisted manifest/config, generated measurement inputs, launch arguments, guest-visible state and public status for every accepted row. + +**Expected results:** + +- Every representation agrees with the selected row, no rejected value is partially persisted or launched, and unrelated inputs do not change measured identity. + + +### Step 3: Verify failure recovery and version compatibility + +Restart after accepted/rejected rows, replay applicable v0.5.4/v0.5.8/v0.5.11 inputs, and retry after correcting one invalid field. + +**Expected results:** + +- Supported historical defaults remain stable, unsupported combinations fail before secret/device consumption, restart reconstructs the same decision and corrected retry succeeds without stale state. + +## Postconditions + +Remove run-scoped VMs/files/devices and verify baseline restoration. diff --git a/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-gpupolicy-007/metadata.json b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-gpupolicy-007/metadata.json new file mode 100644 index 000000000..d317798df --- /dev/null +++ b/test-suites/cases/01-guest-os/08-attestation-and-crypto/tc-gos-gpupolicy-007/metadata.json @@ -0,0 +1,35 @@ +{ + "id": "tc-gos-gpupolicy-007", + "title": "GPU attestation proxy nonce claim and Rego policy enforcement", + "priority": "P0", + "requirements": [ + "req-gos-gpupolicy-007" + ], + "risks": [ + "risk-gos-gpupolicy-007" + ], + "tags": [ + "semantic-review" + ], + "fixture": { + "profile": "gpu-policy", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": true, + "simulation_allowed": false + }, + "actions_under_test": [ + "GPU attestation proxy nonce claim and Rego policy enforcement" + ], + "execution": { + "entrypoint": "shared/automation/capability-probe-case.py", + "args": [], + "timeout_seconds": 60 + } +} diff --git a/test-suites/cases/01-guest-os/09-observability-and-network/metadata.json b/test-suites/cases/01-guest-os/09-observability-and-network/metadata.json new file mode 100644 index 000000000..34fb5b7c0 --- /dev/null +++ b/test-suites/cases/01-guest-os/09-observability-and-network/metadata.json @@ -0,0 +1,4 @@ +{ + "id": "section-guest-os-observability-and-network", + "title": "Observability And Network" +} diff --git a/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-001/case.md b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-001/case.md new file mode 100644 index 000000000..15aa33c47 --- /dev/null +++ b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-001/case.md @@ -0,0 +1,95 @@ + + + +# TC-GOS-OBSERVABIL-001: Dashboard metrics and container log filtering + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-observabil-001](../../../../catalog/feature-audit.md#req-gos-observabil-001) +- Risks: [risk-gos-observabil-001](../../../../catalog/feature-audit.md#risk-gos-observabil-001) +- Source: `dstack/guest-agent/src/http_routes.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Before creating the timestamped log fixture, require the bootstrap-prepared `ubuntu:latest` image and probe its `sh` entrypoint. Do not reuse the first running service image: current service images may intentionally be shell-free. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- The external HTTP listener exposes `GET /` unconditionally. It exposes + `GET /metrics` only when `app_compose.public_sysinfo=true`, and exposes + `GET /logs/` only when `app_compose.public_logs=true`. + Its repository default is TCP `0.0.0.0:8090`; obtain an override from the + effective guest-agent configuration. `/run/dstack.sock` is the internal + DstackGuest pRPC listener and must not be used for dashboard, metrics, or log + HTTP probes (a `GET /` there can legitimately return Rocket HTTP 422). +- The log query fields are `since`, `until`, `follow`, `text`, `timestamps`, + `bare`, `tail`, and `ansi`. `since`/`until` accept an absolute decimal Unix + timestamp, an empty value for zero, or a relative unsigned value ending in + `s`, `m`, `h`, or `d`; malformed values return a JSON error line. The default + tail is `1000`. Unless `text=true`, message data is Base64. With + `bare=true,text=true,ansi=false`, ANSI escapes are removed; `ansi=true` + preserves them. Non-bare output is newline-delimited JSON containing + `channel` and `message`. +- A positive log-filtering matrix requires an isolated container whose stdout + and stderr contain run-unique timestamped plain-text and ANSI fixtures. An + already-running shared container without those known fixtures cannot confirm + exact since/until/tail/channel boundaries and is not a substitute. +- Respect `destructive_actions_allowed` from the runtime manifest. When it is + false, do not run `docker run`, `docker rm`, or create temporary files inside + that shared guest. If no separate case-scoped guest/container fixture is + declared, retain one bounded baseline observation and report the positive log + matrix BLOCKED. The presence of a cached container image does not grant + permission to mutate a shared guest. + +## Objective + +Verify dashboard metrics and container log filtering across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for dashboard metrics and container log filtering. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Query dashboard, metrics, and logs with since/until/follow/tail/text/timestamps/bare/ANSI combinations. + +**Expected results:** + +- Metrics reflect live resources; log filtering and streaming boundaries are exact and container-name traversal is rejected. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-001/metadata.json b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-001/metadata.json new file mode 100644 index 000000000..41ad8163c --- /dev/null +++ b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-001/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-observabil-001", + "title": "Dashboard metrics and container log filtering", + "priority": "P1", + "requirements": [ + "req-gos-observabil-001" + ], + "risks": [ + "risk-gos-observabil-001" + ], + "tags": [ + "guest", + "observability-and-network" + ], + "fixture": { + "profile": "network-lifecycle", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Dashboard metrics and container log filtering" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-001/run.py b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-001/run.py new file mode 100755 index 000000000..f327a2133 --- /dev/null +++ b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-001/run.py @@ -0,0 +1,424 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Verify dashboard metrics and exact case-owned container log filtering.""" + +from __future__ import annotations + +import base64 +import hashlib +import json +import os +import pathlib +import re +import subprocess +import tempfile +import urllib.error +import urllib.parse +import urllib.request +from datetime import datetime +from typing import Any + +CASE_ID = "tc-gos-observabil-001" +TIMESTAMP_RE = re.compile(r"^(\S+)\s+(.*)$") + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as stream: + json.dump(value, stream, indent=2, sort_keys=True) + stream.write("\n") + temporary = pathlib.Path(stream.name) + temporary.replace(path) + + +def ssh( + ssh_argv: list[str], script: str, timeout: int = 90 +) -> subprocess.CompletedProcess[str]: + """Run a bounded script in the lease-owned guest.""" + return subprocess.run( + [*ssh_argv, "bash", "-s"], + input=script, + text=True, + capture_output=True, + timeout=timeout, + check=False, + ) + + +def get(url: str, timeout: int = 30) -> tuple[int, bytes, str]: + """Fetch one bounded HTTP endpoint including HTTP error bodies.""" + try: + with urllib.request.urlopen(url, timeout=timeout) as response: + return response.status, response.read(), response.headers.get_content_type() + except urllib.error.HTTPError as error: + return error.code, error.read(), error.headers.get_content_type() + + +def log_url(base: str, container: str, **query: Any) -> str: + """Build one encoded log query.""" + encoded = urllib.parse.urlencode( + { + key: str(value).lower() if isinstance(value, bool) else value + for key, value in query.items() + } + ) + return f"{base}/logs/{urllib.parse.quote(container, safe='')}?{encoded}" + + +def json_lines(body: bytes) -> list[dict[str, Any]]: + """Parse newline-delimited JSON log output.""" + rows = [] + for line in body.decode(errors="strict").splitlines(): + if not line: + continue + value = json.loads(line) + if not isinstance(value, dict): + raise AssertionError("log line was not a JSON object") + rows.append(value) + return rows + + +def marker_times( + completed: subprocess.CompletedProcess[str], markers: list[str] +) -> dict[str, int]: + """Extract integer Unix seconds from Docker RFC3339 log timestamps.""" + combined = completed.stdout.splitlines() + completed.stderr.splitlines() + found: dict[str, int] = {} + for line in combined: + match = TIMESTAMP_RE.match(line) + if not match: + continue + timestamp, message = match.groups() + for marker in markers: + if marker in message: + normalized = timestamp.replace("Z", "+00:00") + found[marker] = int(datetime.fromisoformat(normalized).timestamp()) + if set(found) != set(markers): + raise AssertionError("Docker timestamp baseline omitted a marker") + return found + + +def require_markers( + body: bytes, expected: list[str], absent: list[str] | None = None +) -> None: + """Require and exclude marker strings in an HTTP body.""" + text = body.decode(errors="replace") + for marker in expected: + if marker not in text: + raise AssertionError(f"log response omitted marker {marker[-12:]}") + for marker in absent or []: + if marker in text: + raise AssertionError( + f"log response unexpectedly included marker {marker[-12:]}" + ) + + +def main() -> int: + """Run dashboard metrics and container log filtering acceptance.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE_ID: + raise SystemExit(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + values = manifest.get("values", {}) + services = values.get("services", {}) + dashboard = services.get("Dashboard") if isinstance(services, dict) else None + status = "PASS" + summary = "Dashboard, metrics, and exact case-owned log filters were verified." + observations: dict[str, Any] = {"candidate_commit": runtime.get("candidate_commit")} + ssh_argv: list[str] = [] + container = "" + stage = "capability" + try: + capable = ( + values.get("destructive_actions_allowed") is True + and isinstance(values.get("ssh_argv"), list) + and isinstance(dashboard, dict) + and isinstance(dashboard.get("url"), str) + ) + if not capable: + status = "BLOCKED" + summary = "fixture lacks a case-owned dashboard log lifecycle guest" + observations["missing_capability"] = "dashboard-log-lifecycle-guest" + else: + ssh_argv = [*map(str, values["ssh_argv"])] + base = str(dashboard["url"]).rstrip("/") + run_hash = hashlib.sha256( + os.environ["DSTACK_TEST_RUN_ID"].encode() + ).hexdigest() + container = f"dstack-log-{run_hash[:20]}" + markers = [ + f"plain-{run_hash[:24]}", + f"ansi-{run_hash[24:48]}", + f"stderr-{run_hash[40:64]}", + ] + stage = "dashboard-baseline" + dashboard_code, dashboard_body, dashboard_type = get(base + "/") + metrics_code, metrics_body, metrics_type = get(base + "/metrics") + if ( + dashboard_code != 200 + or b"/dev/null +docker run --rm --entrypoint sh "$image" -c true +docker rm -f "$name" >/dev/null 2>&1 || true +docker run -d --name "$name" --entrypoint sh "$image" -c 'printf "%s\\n" "{markers[0]}"; sleep 2; printf "\\033[31m%s\\033[0m\\n" "{markers[1]}"; sleep 2; printf "%s\\n" "{markers[2]}" >&2' >/dev/null +for _ in $(seq 1 30); do + state=$(docker inspect -f '{{{{.State.Status}}}}' "$name") + [ "$state" = exited ] && break + sleep 1 +done +[ "$(docker inspect -f '{{{{.State.Status}}}}' "$name")" = exited ] +docker logs --timestamps "$name" +""", + 90, + ) + if fixture.returncode: + observations["container_fixture_diagnostic"] = { + "returncode": fixture.returncode, + "stdout_tail": fixture.stdout[-2000:], + "stderr_tail": fixture.stderr[-2000:], + } + raise AssertionError("failed to create timestamped log fixture") + timestamps = marker_times(fixture, markers) + ordered = [timestamps[marker] for marker in markers] + if not (ordered[0] < ordered[1] < ordered[2]): + raise AssertionError("fixture log timestamps were not strictly ordered") + stage = "json-channels" + code, body, content_type = get( + log_url(base, container, text=True, bare=False, tail="all") + ) + rows = json_lines(body) + if code != 200 or not rows: + raise AssertionError("structured text log query failed") + messages = {str(row.get("message", "")): row.get("channel") for row in rows} + if not any( + markers[0] in message and channel == "stdout" + for message, channel in messages.items() + ): + raise AssertionError( + "stdout channel marker was not structured correctly" + ) + if not any( + markers[2] in message and channel == "stderr" + for message, channel in messages.items() + ): + raise AssertionError( + "stderr channel marker was not structured correctly" + ) + stage = "base64" + code, body, _ = get( + log_url(base, container, text=False, bare=False, tail="all") + ) + encoded_rows = json_lines(body) + decoded = [ + base64.b64decode(str(row["message"])).decode(errors="replace") + for row in encoded_rows + ] + if code != 200 or not all( + any(marker in value for value in decoded) for marker in markers + ): + raise AssertionError("base64 log query did not decode to all markers") + stage = "ansi" + code, stripped, _ = get( + log_url(base, container, text=True, bare=True, ansi=False, tail="all") + ) + require_markers(stripped, markers) + if b"\x1b[31m" in stripped: + raise AssertionError("ansi=false retained an ANSI escape") + code_ansi, preserved, _ = get( + log_url(base, container, text=True, bare=True, ansi=True, tail="all") + ) + if code != 200 or code_ansi != 200 or b"\x1b[31m" not in preserved: + raise AssertionError("ansi=true did not preserve the ANSI escape") + stage = "tail" + code, tail_body, _ = get( + log_url(base, container, text=True, bare=True, tail="1") + ) + if code != 200: + raise AssertionError("tail query failed") + require_markers(tail_body, [markers[2]], markers[:2]) + stage = "absolute-boundaries" + since_value = ordered[0] + 1 + until_value = ordered[1] + 1 + _, since_body, _ = get( + log_url( + base, container, text=True, bare=True, since=since_value, tail="all" + ) + ) + require_markers(since_body, markers[1:], [markers[0]]) + _, until_body, _ = get( + log_url( + base, container, text=True, bare=True, until=until_value, tail="all" + ) + ) + require_markers(until_body, markers[:2], [markers[2]]) + stage = "relative-follow-timestamps" + _, relative_body, _ = get( + log_url(base, container, text=True, bare=True, since="1h", tail="all") + ) + require_markers(relative_body, markers) + _, follow_body, _ = get( + log_url(base, container, text=True, bare=True, follow=True, tail="1"), + 30, + ) + require_markers(follow_body, [markers[2]], markers[:2]) + _, timestamp_body, _ = get( + log_url( + base, container, text=True, bare=True, timestamps=True, tail="1" + ) + ) + if not re.search(rb"\d{4}-\d{2}-\d{2}T", timestamp_body): + raise AssertionError("timestamps=true omitted RFC3339 timestamp") + stage = "invalid-inputs" + _, malformed, _ = get( + log_url(base, container, text=True, bare=True, since="not-a-time") + ) + malformed_value = json.loads(malformed) + if malformed_value.get("error") != "Invalid since": + raise AssertionError("malformed since did not return structured error") + traversal_code, traversal_body, _ = get( + f"{base}/logs/{urllib.parse.quote('../' + container, safe='')}?text=true" + ) + if traversal_code == 200 and any( + marker.encode() in traversal_body for marker in markers + ): + raise AssertionError("container-name traversal exposed fixture logs") + stage = "cleanup-health" + cleanup = ssh( + ssh_argv, + f"docker rm -f {container} >/dev/null\ndocker info >/dev/null\n", + 60, + ) + if cleanup.returncode: + raise AssertionError("failed to clean case-owned log container") + container = "" + final_dashboard, final_body, _ = get(base + "/") + final_metrics, final_metrics_body, _ = get(base + "/metrics") + if ( + final_dashboard != 200 + or final_metrics != 200 + or not final_body + or not final_metrics_body + ): + raise AssertionError("dashboard or metrics unhealthy after cleanup") + observations.update( + { + "dashboard": { + "status": dashboard_code, + "content_type": dashboard_type, + "body_sha256": hashlib.sha256(dashboard_body).hexdigest(), + }, + "metrics": { + "status": metrics_code, + "content_type": metrics_type, + "required_metrics": list(required_metrics), + "body_sha256": hashlib.sha256(metrics_body).hexdigest(), + }, + "fixture": { + "timestamp_ordered": True, + "marker_hashes": [ + hashlib.sha256(item.encode()).hexdigest() + for item in markers + ], + }, + "structured_channels": ["stdout", "stderr"], + "base64_decoded": True, + "ansi_stripped_and_preserved": True, + "tail_exact": True, + "absolute_since_until_exact": True, + "relative_since": True, + "follow_stopped_container": True, + "timestamps_present": True, + "malformed_since_error": True, + "traversal_status": traversal_code, + "traversal_rejected": True, + "container_removed": True, + "service_healthy_after": True, + } + ) + except ( + AssertionError, + KeyError, + OSError, + ValueError, + json.JSONDecodeError, + subprocess.SubprocessError, + urllib.error.URLError, + ) as error: + status = "FAIL" + summary = f"{stage}: {error}" + observations["failure"] = str(error) + observations["failure_stage"] = stage + finally: + if container and ssh_argv: + ssh(ssh_argv, f"docker rm -f {container} >/dev/null 2>&1 || true\n", 30) + artifact = { + "path": "artifacts/dashboard-log-filtering.json", + "step_id": f"{case_id}-step-01", + "name": "Dashboard metrics and log filtering", + "description": "Redacted endpoint hashes, metric names, marker hashes, filter booleans, channels, and cleanup state.", + } + atomic_json(result_dir / artifact["path"], observations) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{case_id}-step-01", + "status": status, + "observed": "Dashboard, metrics, Docker health, and a clean case-owned container baseline were checked.", + }, + { + "id": f"{case_id}-step-02", + "status": status, + "observed": "Text/Base64, bare/structured, ANSI, channel, tail, timestamp, since/until, relative, and follow filters were exercised.", + }, + { + "id": f"{case_id}-step-03", + "status": status, + "observed": "Malformed time, traversal rejection, container cleanup, and final endpoint health were verified.", + }, + ], + "artifacts": [artifact], + "remarks": "Only a uniquely named lease-owned container is created; raw log fixtures are not retained in artifacts.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-002/case.md b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-002/case.md new file mode 100644 index 000000000..0629b9e4f --- /dev/null +++ b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-002/case.md @@ -0,0 +1,81 @@ + + + +# TC-GOS-OBSERVABIL-002: Socket activation and listener isolation + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-observabil-002](../../../../catalog/feature-audit.md#req-gos-observabil-002) +- Risks: [risk-gos-observabil-002](../../../../catalog/feature-audit.md#risk-gos-observabil-002) +- Source: `dstack/guest-agent/src/socket_activation.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- The listener map is: systemd `ListenStream` index 0 is + `/run/dstack.sock` for DstackGuest; index 1 is `/run/tappd.sock` for Tappd; + the external TCP listener defaults to port `8090` and exposes public HTTP plus + `/prpc/Worker.*`; GuestApi defaults to vsock any-CID port `8000` under + `/api/GuestApi.*`. Do not infer the external listener from a Unix socket. +- Confirming activation survival, wrong-index behavior, bind conflicts, and + partial-listener failure requires an isolated service instance whose sockets + and process may be stopped/restarted. If every declared guest has + `destructive_actions_allowed=false` and the manifest has no distinct + case-scoped listener fixture, do not restart services, close sockets, alter + units/configuration, or substitute a read-only listener snapshot; retain one + bounded manifest observation and report the lifecycle matrix BLOCKED. + +## Objective + +Verify socket activation and listener isolation across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for socket activation and listener isolation. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Exercise systemd socket activation, internal Unix/vsock, external HTTPS, and GuestApi listeners. + +**Expected results:** + +- Each API appears only on its configured transport, accepts expected clients, and does not expose internal methods externally. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-002/metadata.json b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-002/metadata.json new file mode 100644 index 000000000..97afe52ad --- /dev/null +++ b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-002/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-observabil-002", + "title": "Socket activation and listener isolation", + "priority": "P1", + "requirements": [ + "req-gos-observabil-002" + ], + "risks": [ + "risk-gos-observabil-002" + ], + "tags": [ + "guest", + "observability-and-network" + ], + "fixture": { + "profile": "network-lifecycle", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Socket activation and listener isolation" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-002/run.py b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-002/run.py new file mode 100755 index 000000000..46eeb73bb --- /dev/null +++ b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-002/run.py @@ -0,0 +1,385 @@ +#!/usr/bin/env python3 +"""Verify guest-agent socket activation, transport isolation, and recovery.""" + +from __future__ import annotations + +import hashlib +import json +import os +import shlex +import subprocess +import sys +import time +import urllib.error +import urllib.request +from pathlib import Path +from typing import Any + +CASE_ID = "tc-gos-observabil-002" + + +def emit(step: str, state: str) -> None: + """Emit one live step transition.""" + print(f"STEP {CASE_ID}-{step} {state}", flush=True) + + +def ssh( + argv: list[str], command: str, *, check: bool = True +) -> subprocess.CompletedProcess[str]: + """Execute one bounded command through the manifest-recorded SSH route.""" + result = subprocess.run( + [*argv, command], + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + timeout=90, + check=False, + ) + if check and result.returncode: + raise RuntimeError( + f"guest command failed ({result.returncode}): {command!r}; " + f"stdout={result.stdout[-800:]!r}; stderr={result.stderr[-800:]!r}" + ) + return result + + +def guest_rpc(argv: list[str], socket: str, route: str) -> dict[str, Any]: + """Call a non-secret JSON RPC through one guest Unix socket.""" + header = shlex.quote("Content-Type: application/json") + body = shlex.quote("{}") + command = ( + "curl --silent --show-error --fail-with-body --max-time 20 " + f"--unix-socket {shlex.quote(socket)} " + f"--header {header} " + f"--data-binary {body} " + f"http://localhost/{shlex.quote(route)}" + ) + raw = ssh(argv, command).stdout + try: + value = json.loads(raw) + except json.JSONDecodeError as error: + raise AssertionError( + f"{route} returned invalid JSON ({len(raw)} bytes): {raw[:500]!r}" + ) from error + if not isinstance(value, dict): + raise AssertionError(f"{route} returned a non-object") + return value + + +def http_json(url: str, body: dict[str, Any]) -> tuple[int, dict[str, Any]]: + """POST JSON and return the HTTP status and object response.""" + request = urllib.request.Request( + url, + data=json.dumps(body, separators=(",", ":")).encode(), + headers={"Content-Type": "application/json"}, + method="POST", + ) + try: + with urllib.request.urlopen(request, timeout=30) as response: + raw = response.read() + code = response.status + except urllib.error.HTTPError as error: + raw = error.read() + code = error.code + value = json.loads(raw) if raw else {} + if not isinstance(value, dict): + raise AssertionError(f"{url} returned a non-object") + return code, value + + +def wait_active(argv: list[str], unit: str) -> None: + """Wait until a systemd unit reports active.""" + deadline = time.monotonic() + 30 + while time.monotonic() < deadline: + if ( + ssh( + argv, f"systemctl is-active --quiet {shlex.quote(unit)}", check=False + ).returncode + == 0 + ): + return + time.sleep(0.5) + raise AssertionError(f"{unit} did not become active") + + +def main() -> int: + """Run the socket activation and isolation acceptance matrix.""" + manifest_path = os.environ.get("DSTACK_TEST_CASE_MANIFEST") + result_dir_value = os.environ.get("DSTACK_TEST_RESULT_DIR") + if not manifest_path or not result_dir_value: + raise SystemExit( + "DSTACK_TEST_CASE_MANIFEST and DSTACK_TEST_RESULT_DIR are required" + ) + result_path = str(Path(result_dir_value) / "result.json") + manifest = json.loads(Path(manifest_path).read_text()) + values = manifest.get("values", {}) + lifecycle = values.get("socket_activation_lifecycle") + ssh_argv = values.get("ssh_argv") + services = values.get("services", {}) + observations: dict[str, Any] = {} + steps: list[dict[str, Any]] = [] + status = "PASS" + summary = "socket activation and listener isolation matrix passed" + stage = "fixture" + + try: + required = ( + isinstance(lifecycle, dict) + and lifecycle.get("destructive_actions_allowed") is True + and values.get("destructive_actions_allowed") is True + and isinstance(ssh_argv, list) + and isinstance(services, dict) + ) + if not required: + status = "BLOCKED" + summary = "fixture lacks a case-owned socket activation lifecycle guest" + observations["missing_capability"] = "socket-activation-lifecycle-guest" + else: + service = str(lifecycle["service_unit"]) + socket_unit = str(lifecycle["socket_unit"]) + dstack_socket = str(lifecycle["dstack_socket"]) + tappd_socket = str(lifecycle["tappd_socket"]) + external_port = int(lifecycle["external_port"]) + guest_port = int(lifecycle["guest_api_vsock_port"]) + + stage = "baseline" + emit("step-01", "START") + baseline = ssh( + ssh_argv, + "set -eu; " + f"systemctl is-active {shlex.quote(service)} {shlex.quote(socket_unit)}; " + f"test -S {shlex.quote(dstack_socket)}; test -S {shlex.quote(tappd_socket)}; " + f"ss -H -ltn sport = :{external_port}; " + f"! ss -H -ltn sport = :{guest_port} | grep -q .", + ) + dstack_before = guest_rpc(ssh_argv, dstack_socket, "Info") + tappd_before = guest_rpc(ssh_argv, tappd_socket, "prpc/Info") + if not dstack_before.get("app_id") or not tappd_before.get("app_id"): + raise AssertionError("Unix listener Info response was incomplete") + observations["baseline"] = { + "unit_states": baseline.stdout.splitlines()[:2], + "dstack_app_id_sha256": hashlib.sha256( + str(dstack_before["app_id"]).encode() + ).hexdigest(), + "tappd_app_id_sha256": hashlib.sha256( + str(tappd_before["app_id"]).encode() + ).hexdigest(), + "tcp_8000_isolated": True, + "external_8090_listening": True, + } + steps.append( + { + "id": "tc-gos-observabil-002-step-01", + "status": "PASS", + "observed": "The case-owned guest exposed active service and socket units, both Unix sockets, external TCP 8090, and no guest TCP 8000 listener.", + } + ) + emit("step-01", "PASS") + + stage = "transport-isolation" + emit("step-02", "START") + dashboard = services.get("Dashboard", {}) + dashboard_url = str(dashboard.get("url", "")).rstrip("/") + with urllib.request.urlopen( + dashboard_url + "/prpc/Worker.Version", timeout=20 + ) as response: + external_body = json.loads(response.read()) + if response.status != 200 or not external_body.get("version"): + raise AssertionError("external Worker.Version was unavailable") + proxied = services.get("ProxiedGuestApi", {}) + proxied_url = str(proxied.get("url", "")).format(method="Info") + proxied_code, proxied_body = http_json( + proxied_url, {"id": str(proxied.get("id", ""))} + ) + if proxied_code != 200 or not proxied_body.get("version"): + raise AssertionError("ProxiedGuestApi.Info was unavailable") + forbidden: dict[str, int] = {} + for route in ( + "Info", + "prpc/DstackGuest.Info", + "prpc/Tappd.Info", + "api/GuestApi.Info", + ): + try: + urllib.request.urlopen(dashboard_url + "/" + route, timeout=10) + code = 200 + except urllib.error.HTTPError as error: + code = error.code + if 200 <= code < 300: + raise AssertionError( + f"internal route was exposed externally: {route}" + ) + forbidden[route] = code + observations["transport_isolation"] = { + "external_worker_version": external_body.get("version"), + "proxied_guest_version": proxied_body.get("version"), + "forbidden_external_status": forbidden, + } + + stage = "activation-recovery" + ssh(ssh_argv, f"systemctl stop {shlex.quote(service)}") + stopped = ssh( + ssh_argv, + "set -eu; " + f"! systemctl is-active --quiet {shlex.quote(service)}; " + f"systemctl is-active --quiet {shlex.quote(socket_unit)}; " + f"test -S {shlex.quote(dstack_socket)}; test -S {shlex.quote(tappd_socket)}", + ) + del stopped + dstack_after = guest_rpc(ssh_argv, dstack_socket, "Info") + wait_active(ssh_argv, service) + tappd_after = guest_rpc(ssh_argv, tappd_socket, "prpc/Info") + if dstack_after.get("app_id") != dstack_before.get("app_id"): + raise AssertionError("DstackGuest identity changed after activation") + if tappd_after.get("app_id") != tappd_before.get("app_id"): + raise AssertionError("Tappd identity changed after activation") + observations["activation_recovery"] = { + "socket_unit_survived_service_stop": True, + "both_socket_paths_survived": True, + "rpc_triggered_service_activation": True, + "responses_stable": True, + } + steps.append( + { + "id": "tc-gos-observabil-002-step-02", + "status": "PASS", + "observed": "DstackGuest and Tappd were isolated to their Unix sockets, Worker was public, GuestApi was reachable only through the VMM proxy, and a Unix RPC reactivated the stopped service without identity change.", + } + ) + emit("step-02", "PASS") + + stage = "fault-injection-recovery" + emit("step-03", "START") + unit_contract = ssh( + ssh_argv, + f"systemctl show -p Listen --value {shlex.quote(socket_unit)}", + ).stdout + descriptor_contract = ( + dstack_socket in unit_contract and tappd_socket in unit_contract + ) + + holder = ( + "import socket,time; " + "s=socket.socket(); s.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1); " + f"s.bind(('0.0.0.0',{external_port})); s.listen(); time.sleep(30)" + ) + ssh( + ssh_argv, + "set -eu; " + f"systemctl stop {shlex.quote(service)} {shlex.quote(socket_unit)}; " + f"nohup python3 -c {shlex.quote(holder)} >/dev/null 2>&1 & " + "echo $! >/run/dstack-test-bind-conflict.pid; sleep 1; " + f"systemctl start {shlex.quote(socket_unit)}", + ) + conflict = ssh( + ssh_argv, + f"timeout 15 systemctl start {shlex.quote(service)}", + check=False, + ) + if conflict.returncode == 0: + raise AssertionError( + "service unexpectedly accepted the TCP bind conflict" + ) + bind_probe = ( + "import socket; " + "s=socket.socket(); " + "s.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1); " + f"s.bind(('0.0.0.0',{external_port})); s.close()" + ) + ssh( + ssh_argv, + "set -eu; " + f"systemctl stop {shlex.quote(service)}; " + f"! systemctl is-active --quiet {shlex.quote(service)}; " + 'pid=$(cat /run/dstack-test-bind-conflict.pid); kill "$pid"; ' + "for _ in $(seq 1 50); do " + 'if ! kill -0 "$pid" 2>/dev/null; then break; fi; sleep 0.1; ' + 'done; ! kill -0 "$pid" 2>/dev/null; ' + "rm -f /run/dstack-test-bind-conflict.pid; " + f"python3 -c {shlex.quote(bind_probe)}; " + f"systemctl reset-failed {shlex.quote(service)}; " + f"systemctl start {shlex.quote(service)}", + ) + wait_active(ssh_argv, service) + + ssh( + ssh_argv, + "set -eu; " + f"systemctl stop {shlex.quote(service)} {shlex.quote(socket_unit)}; " + f"rm -f {shlex.quote(dstack_socket)} {shlex.quote(tappd_socket)}; " + f"systemctl start {shlex.quote(socket_unit)}; " + f"test -S {shlex.quote(dstack_socket)}; " + f"test -S {shlex.quote(tappd_socket)}", + ) + recovered_dstack = guest_rpc(ssh_argv, dstack_socket, "Info") + wait_active(ssh_argv, service) + recovered_tappd = guest_rpc(ssh_argv, tappd_socket, "prpc/Info") + if recovered_dstack.get("app_id") != dstack_before.get("app_id"): + raise AssertionError( + "DstackGuest identity changed after listener recovery" + ) + if recovered_tappd.get("app_id") != tappd_before.get("app_id"): + raise AssertionError("Tappd identity changed after listener recovery") + if not descriptor_contract: + raise AssertionError("socket unit does not declare both listener paths") + observations["fault_recovery"] = { + "descriptor_contract_has_both_listeners": True, + "bind_conflict_rejected": True, + "service_recovered_after_conflict": True, + "missing_listener_paths_recreated": True, + "both_rpc_paths_reactivated": True, + "identity_stable": True, + } + steps.append( + { + "id": "tc-gos-observabil-002-step-03", + "status": "PASS", + "observed": "The socket descriptor contract contained both listeners, a TCP bind conflict failed closed, removing both listener paths was repaired by socket-unit restart, and both RPC paths reactivated with stable identity.", + } + ) + emit("step-03", "PASS") + except Exception as error: + status = "FAIL" + summary = f"{stage}: {error}" + observations["error_type"] = type(error).__name__ + observations["error"] = str(error) + try: + if isinstance(lifecycle, dict) and isinstance(ssh_argv, list): + ssh( + ssh_argv, + f"systemctl start {shlex.quote(str(lifecycle['service_unit']))}", + check=False, + ) + except Exception: + pass + + artifact = { + "case_id": CASE_ID, + "status": status, + "environment": "HARDWARE", + "observations": observations, + } + artifact_path = ( + Path(result_path).parent / "artifacts/socket-activation-isolation.json" + ) + artifact_path.parent.mkdir(parents=True, exist_ok=True) + artifact_path.write_text(json.dumps(artifact, indent=2) + "\n") + result = { + "schema_version": "1.0", + "case_id": CASE_ID, + "status": status, + "summary": summary, + "steps": steps, + "evidence": [ + { + "path": "artifacts/socket-activation-isolation.json", + "sha256": hashlib.sha256(artifact_path.read_bytes()).hexdigest(), + } + ], + } + Path(result_path).write_text(json.dumps(result, indent=2) + "\n") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-003/case.md b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-003/case.md new file mode 100644 index 000000000..a5bff2664 --- /dev/null +++ b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-003/case.md @@ -0,0 +1,101 @@ + + + +# TC-GOS-OBSERVABIL-003: Gateway checker startup contract and WireGuard isolation + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-gos-observabil-003](../../../../catalog/feature-audit.md#req-gos-observabil-003) +- Risks: [risk-gos-observabil-003](../../../../catalog/feature-audit.md#risk-gos-observabil-003) +- Source: `dstack/dstack-util/src/gateway_checker.rs`, `os/common/rootfs/dstack-gateway-checker.service` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- The checker is `dstack-util gateway-checker --work-dir `, run by + `dstack-gateway-checker.service`. It replaced the former `wg-checker.sh`. +- Its refresh timing (180s periodic re-registration, 180s handshake staleness, + and the 30s/60s/120s retry backoff for a missing WireGuard config) is a pure + decision function covered by unit tests in `dstack/dstack-util/src/gateway_checker.rs`. + Do not re-derive that matrix here: an accelerated clock in the guest can only + restate those tests less reliably. +- What unit tests cannot reach is the boundary between the process and systemd, + which is what this case covers. The checker encodes each unrecoverable startup + condition as an exit code, and the unit must honour it: + - An app that never enabled dstack-gateway has nothing to supervise, so the + checker exits 0. With `Restart=on-failure` systemd then leaves it alone; + `Restart=always` would respawn it every `RestartSec` for the life of every + gateway-less CVM. + - A missing gateway app id or gateway URL is a deployment mistake fixed for + the lifetime of the VM. The checker exits with `EXIT_MISCONFIGURED`, which + the unit pins in `RestartPreventExitStatus`. That stops the respawn while + leaving the unit in `failed` state, so the mistake stays visible. Read the + expected code from the product source; do not restate it. + - Any other non-zero exit is treated as transient and is retried. +- Registration failure is no longer fatal to boot, so the guest reports + `boot.error` to the host while it has no route and retracts it once the + checker registers. The VMM surfaces that through `VmInfo.boot_error`. +- This case needs an isolated WireGuard interface, permission to create a + network namespace, and the guest's own `dstack-util` and unit. Never alter + networking, gateway registration, `/etc/wireguard`, services, routes, DNS, or + interfaces on a guest with `destructive_actions_allowed=false`. If no distinct + case-scoped network fixture is declared, preserve one bounded manifest + observation and report the behavior BLOCKED. + +## Objective + +Verify that the gateway checker maps each unrecoverable startup condition to the exit code its unit honours, and that a real WireGuard interface can be configured and observed in isolation. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for wireguard configuration and checker recovery. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Build an isolated WireGuard interface, then run the packaged checker against synthesized host-shared inputs for each startup condition. + +**Expected results:** + +- Addresses, peers, routes, DNS, and the zero-handshake baseline are observable without duplicate interfaces or leaked keys. +- A gateway-disabled app makes the checker exit 0 rather than poll. +- A missing gateway app id and a missing gateway URL each make it exit with the code the unit pins in `RestartPreventExitStatus`. +- The installed unit is loaded with `Restart=on-failure`, inhibits restart for exactly that code, and runs the `dstack-util` subcommand rather than the removed shell script. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-003/metadata.json b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-003/metadata.json new file mode 100644 index 000000000..206d4ad52 --- /dev/null +++ b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-003/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-observabil-003", + "title": "Gateway checker startup contract and WireGuard isolation", + "priority": "P1", + "requirements": [ + "req-gos-observabil-003" + ], + "risks": [ + "risk-gos-observabil-003" + ], + "tags": [ + "guest", + "observability-and-network" + ], + "fixture": { + "profile": "network-lifecycle", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Gateway checker startup contract and WireGuard isolation" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-003/run.py b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-003/run.py new file mode 100755 index 000000000..605940ff0 --- /dev/null +++ b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-003/run.py @@ -0,0 +1,178 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 +"""Exercise real WireGuard isolation and the gateway checker startup contract.""" + +from __future__ import annotations + +import json +import os +import pathlib +import re +import subprocess +import tempfile +from typing import Any + +CASE_ID = "tc-gos-observabil-003" +UNIT = "dstack-gateway-checker.service" +# The checker's misconfigured exit code is pinned by the unit's +# RestartPreventExitStatus. Read it from the source rather than restating it, so +# this case cannot keep passing against a value the product no longer uses. +EXIT_CONST_RE = re.compile(r"^const EXIT_MISCONFIGURED: i32 = (\d+);$", re.MULTILINE) + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile("w", dir=path.parent, delete=False) as output: + json.dump(value, output, indent=2, sort_keys=True) + output.write("\n") + temporary = pathlib.Path(output.name) + temporary.replace(path) + + +def run( + argv: list[str], *, data: bytes | None = None, timeout: int = 180 +) -> subprocess.CompletedProcess[bytes]: + """Run one bounded command.""" + return subprocess.run( + argv, input=data, capture_output=True, timeout=timeout, check=False + ) + + +def main() -> int: + """Run the checker startup matrix inside a lease-owned mkosi guest.""" + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + artifacts = result_dir / "artifacts" + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + values = manifest.get("values") or {} + ssh = [str(value) for value in values.get("ssh_argv") or []] + status = "PASS" + summary = "WireGuard isolation and gateway checker startup contract passed." + evidence: dict[str, Any] = {} + try: + if not ssh or values.get("destructive_actions_allowed") is not True: + raise RuntimeError("fixture omitted lease-owned guest controls") + repository = pathlib.Path(str(runtime["repository"])) + checker_source = repository / "dstack/dstack-util/src/gateway_checker.rs" + matched = EXIT_CONST_RE.search(checker_source.read_text()) + if not matched: + raise RuntimeError(f"cannot read EXIT_MISCONFIGURED from {checker_source}") + misconfigured_exit = matched.group(1) + script = ( + repository / "test-suites/shared/automation/gateway-checker-lifecycle.sh" + ) + installed = run( + [*ssh, "install -m 0755 /dev/stdin /run/dstack-test-gateway-lifecycle"], + data=script.read_bytes(), + ) + if installed.returncode: + raise RuntimeError("failed to install the gateway checker lifecycle driver") + executed = run( + [*ssh, "/run/dstack-test-gateway-lifecycle", UNIT, misconfigured_exit], + timeout=240, + ) + artifacts.mkdir(parents=True, exist_ok=True) + (artifacts / "gateway-checker-lifecycle.log").write_bytes( + executed.stdout + executed.stderr + ) + rows = [ + row + for row in executed.stdout.decode(errors="replace").splitlines() + if row.startswith("{") + ] + if executed.returncode or not rows: + tail = (executed.stdout + executed.stderr).decode(errors="replace")[-2000:] + raise RuntimeError( + f"gateway checker lifecycle rc={executed.returncode}: {tail}" + ) + evidence = json.loads(rows[-1]) + evidence["misconfigured_exit_code"] = int(misconfigured_exit) + required = ( + "real_interface", + "address_route", + "dns_observed", + "no_handshake_observed", + "disabled_exits_zero", + "missing_app_id_exit_code", + "missing_urls_exit_code", + "unit_restart_on_failure", + "unit_prevents_restart", + "unit_runs_subcommand", + "legacy_script_absent", + "interface_isolated", + ) + if evidence.get("checks", 0) < 24 or not all( + evidence.get(key) is True for key in required + ): + raise RuntimeError("gateway checker evidence omitted a required row") + except ( + KeyError, + OSError, + RuntimeError, + subprocess.SubprocessError, + ValueError, + ) as error: + status = "FAIL" + summary = f"{type(error).__name__}: {error}" + artifact_entries = [ + { + "path": "artifacts/gateway-checker-lifecycle.json", + "step_id": f"{CASE_ID}-step-01", + "name": "Gateway checker startup matrix", + "description": "Boolean and count evidence for isolated interface/configuration, checker exit codes per startup condition, and the shipped unit's restart policy.", + }, + { + "path": "artifacts/gateway-checker-lifecycle.log", + "step_id": f"{CASE_ID}-step-02", + "name": "Gateway checker native log", + "description": "Bounded native output with no WireGuard private keys, configuration content, or credentials.", + }, + ] + atomic_json(artifacts / "gateway-checker-lifecycle.json", evidence) + atomic_json(artifacts / "manifest.json", {"artifacts": artifact_entries}) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{CASE_ID}-step-01", + "status": status, + "observed": "A real WireGuard interface, address, peer, route, DNS view, and zero-handshake baseline were isolated in a network namespace." + if status == "PASS" + else summary, + }, + { + "id": f"{CASE_ID}-step-02", + "status": status, + "observed": "The packaged checker exited 0 for an app that never enabled dstack-gateway, and exited with the pinned misconfigured code for a missing gateway app id and for a missing gateway URL." + if status == "PASS" + else summary, + }, + { + "id": f"{CASE_ID}-step-03", + "status": status, + "observed": "The installed unit was loaded with Restart=on-failure, inhibited restart for exactly the misconfigured exit code, ran the dstack-util subcommand rather than the removed shell script, and the namespace and guest routing were left unchanged." + if status == "PASS" + else summary, + }, + ], + "artifacts": artifact_entries, + "remarks": "Refresh timing (periodic interval, handshake staleness, retry backoff) is covered by dstack-util's gateway_checker unit tests over a pure decision function; this case covers the process/systemd boundary those tests cannot reach. The misconfigured exit code is read from the product source at run time. Private keys are never persisted as evidence.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-004/case.md b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-004/case.md new file mode 100644 index 000000000..5c6aed2a9 --- /dev/null +++ b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-004/case.md @@ -0,0 +1,114 @@ + + + +# TC-GOS-OBSERVABIL-004: System network and resource telemetry + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-observabil-004](../../../../catalog/feature-audit.md#req-gos-observabil-004) +- Risks: [risk-gos-observabil-004](../../../../catalog/feature-audit.md#risk-gos-observabil-004) +- Source: `dstack/guest-agent/src/guest_api_service.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- `NetworkInfo` intentionally reports only `dstack-wg0`, `enp*`, and `eth*` + interfaces; Docker bridges are excluded for privacy. Each address includes its + prefix, counters are cumulative received/transmitted bytes and receive/send + errors, DNS entries are `nameserver` values from `/etc/resolv.conf`, and + gateways are current default gateways. WireGuard command output is returned + separately as `wg_info`. +- `SysInfo` reports memory/swap and disk sizes in bytes, uptime in seconds, and + load averages multiplied by 100 and truncated to integers. Disks are limited + to configured `data_disks` and sorted by mount point. `ListContainers` + includes stopped and running containers. +- `SysInfo`, `NetworkInfo`, and `ListContainers` belong to the private + `GuestApi` service bound to guest vsock port 8000; they are not methods on + the public `DstackGuest` listener. Call them through the case manifest's + `services.ProxiedGuestApi.url`, replacing `{method}` and sending + `{"id":""}`. A `Service not found` response from + `services.DstackGuest` proves the wrong listener was selected and is not a + product telemetry result. +- The complete transition matrix requires an isolated guest where interfaces, + routes, DNS, CPU/load, memory pressure, disks, swap, and containers may be + safely added and removed. Do not change any of these on a guest with + `destructive_actions_allowed=false`; a read-only snapshot cannot prove + transition or disappearance behavior. Without a distinct case-scoped + telemetry fixture, retain one bounded manifest observation and report the + matrix BLOCKED. +- The candidate guest uses BusyBox `ip`; its kernel does not provide the dummy + link type. Create the removable `eth*` observation interface as a veth pair + (`ip link add ethobs... type veth peer name veth...`) and remove the pair + after the changed snapshot. Do not use `ip link add ... type dummy` and do + not treat that known unsupported link type as a product failure. +- Before creating `ethobs...`, write a run-scoped `.network` file under + `/run/systemd/network` that matches only that interface and sets + `[Link] Unmanaged=yes`, then call `networkctl reload`. Otherwise networkd's + generic wired policy races the test and flushes the synthetic IPv4 address + and route. After deleting the interface, unlink the file and reload again; + do not stop networkd because that removes the fixture's SSH connectivity. +- systemd may also rewrite `/etc/resolv.conf` during the snapshot. Copy its + baseline to a run-scoped file, append the test nameserver there, bind-mount + that file over `/etc/resolv.conf` for the changed observation, then unmount + it and unlink the file during cleanup. Directly appending to the managed + file is not a stable DNS transition. +- The default ZFS data volume rejects swap files as having holes, even when + filled from `/dev/urandom`. To exercise swap telemetry without changing the + storage fixture, create a bounded file under `/dev/shm`, attach it with + `losetup -f --show`, run `mkswap` and `swapon` on the loop block device, then + clean up in this order: `swapoff`, `losetup -d`, and unlink the backing file. + +## Objective + +Verify system network and resource telemetry across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for system network and resource telemetry. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Change interfaces, routes, DNS, load, memory, disk, swap, and container set. + +**Expected results:** + +- GuestApi reports complete current values with correct units, prefixes, counters, and disappearance of removed resources. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-004/metadata.json b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-004/metadata.json new file mode 100644 index 000000000..f8d3aa672 --- /dev/null +++ b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-004/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-observabil-004", + "title": "System network and resource telemetry", + "priority": "P1", + "requirements": [ + "req-gos-observabil-004" + ], + "risks": [ + "risk-gos-observabil-004" + ], + "tags": [ + "guest", + "observability-and-network" + ], + "fixture": { + "profile": "network-lifecycle", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "System network and resource telemetry" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-004/run.py b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-004/run.py new file mode 100755 index 000000000..ab903501a --- /dev/null +++ b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-004/run.py @@ -0,0 +1,313 @@ +#!/usr/bin/env python3 +"""Exercise live GuestApi network and resource telemetry transitions.""" + +from __future__ import annotations + +import hashlib +import json +import os +import shlex +import subprocess +import time +import urllib.error +import urllib.request +from pathlib import Path +from typing import Any + +CASE_ID = "tc-gos-observabil-004" +UNKNOWN_ID = "00000000-0000-4000-8000-000000000000" + + +def ssh( + argv: list[str], command: str, *, check: bool = True +) -> subprocess.CompletedProcess[str]: + """Run one bounded command in the lease-owned guest.""" + result = subprocess.run( + [*argv, command], + text=True, + capture_output=True, + timeout=90, + check=False, + ) + if check and result.returncode: + raise RuntimeError(f"guest command failed with rc={result.returncode}") + return result + + +def rpc(url: str, vm_id: str) -> tuple[int, dict[str, Any]]: + """Call one proxied GuestApi JSON method.""" + request = urllib.request.Request( + url, + data=json.dumps({"id": vm_id}, separators=(",", ":")).encode(), + headers={"content-type": "application/json"}, + ) + try: + with urllib.request.urlopen(request, timeout=60) as response: + code, raw = response.status, response.read() + except urllib.error.HTTPError as error: + code, raw = error.code, error.read() + value = json.loads(raw) if raw else {} + if not isinstance(value, dict): + raise AssertionError("GuestApi returned a non-object") + return code, value + + +def by_name(rows: list[dict[str, Any]], name: str) -> dict[str, Any] | None: + """Find one row by name.""" + return next((row for row in rows if row.get("name") == name), None) + + +def main() -> int: + """Run baseline, changed, restored, invalid-input, and cleanup observations.""" + result_dir = Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads(Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text()) + runtime = json.loads(Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text()) + values = manifest["values"] + ssh_argv = values["ssh_argv"] + vm_id = str(values["vm_id"]) + proxied = values["services"]["ProxiedGuestApi"] + base = str(proxied["url"]) + lease = os.environ.get("DSTACK_TEST_LEASE_ID", "lease")[-8:].replace("-", "") + interface = f"ethobs{lease[:5]}" + peer = f"veth{lease[:6]}" + container = f"dstack-telemetry-{lease}" + dns = "192.0.2.53" + address = "192.0.2.10" + marker_dir = f"/run/dstack-telemetry-{lease}" + cleanup_errors: list[str] = [] + checks: dict[str, bool] = {} + status = "FAIL" + summary = "telemetry lifecycle did not complete" + started = time.monotonic() + + def call(method: str, target: str = vm_id) -> tuple[int, dict[str, Any]]: + return rpc(base.format(method=method), target) + + try: + baseline_codes_values = { + method: call(method) + for method in ("SysInfo", "NetworkInfo", "ListContainers") + } + checks["baseline_methods_healthy"] = all( + code == 200 for code, _ in baseline_codes_values.values() + ) + if not checks["baseline_methods_healthy"]: + raise AssertionError("baseline GuestApi methods failed") + baseline_sys = baseline_codes_values["SysInfo"][1] + baseline_net = baseline_codes_values["NetworkInfo"][1] + baseline_containers = baseline_codes_values["ListContainers"][1] + if by_name(baseline_net.get("interfaces", []), interface): + raise AssertionError("run-scoped interface already exists") + if any( + container in row.get("names", []) + for row in baseline_containers.get("containers", []) + ): + raise AssertionError("run-scoped container already exists") + mount_point = str( + baseline_sys.get("disks", [{}])[0].get("mount_point", "/data") + ) + baseline_disk = baseline_sys.get("disks", [{}])[0] + baseline_available = int(baseline_sys.get("available_memory", 0)) + baseline_used = int(baseline_sys.get("used_memory", 0)) + baseline_swap = int(baseline_sys.get("total_swap", 0)) + + network_setup = f"""set -eu +mkdir -p {shlex.quote(marker_dir)} +printf '[Match]\nName={interface}\n[Link]\nUnmanaged=yes\n' > /run/systemd/network/00-{interface}.network +networkctl reload +ip link add {interface} type veth peer name {peer} +ip addr add {address}/24 dev {interface} +ip link set {interface} up +ip link set {peer} up +ip route add 198.51.100.0/24 dev {interface} metric 4096 +""" + ssh(ssh_argv, network_setup) + checks["network_setup_completed"] = True + + dns_setup = f"""set -eu +cp /etc/resolv.conf {marker_dir}/resolv.conf +printf '\nnameserver {dns}\n' >> {marker_dir}/resolv.conf +mount --bind {marker_dir}/resolv.conf /etc/resolv.conf +""" + ssh(ssh_argv, dns_setup) + checks["dns_setup_completed"] = True + + storage_setup = f"""set -eu +dd if=/dev/urandom of={shlex.quote(mount_point)}/.dstack-telemetry-{lease} bs=1M count=128 conv=fsync >/dev/null 2>&1 +dd if=/dev/zero of={marker_dir}/swap bs=1M count=16 >/dev/null 2>&1 +loop=$(losetup -f --show {marker_dir}/swap) +printf '%s' "$loop" > {marker_dir}/loop +mkswap "$loop" >/dev/null +swapon "$loop" +""" + ssh(ssh_argv, storage_setup) + checks["storage_swap_setup_completed"] = True + + pressure_setup = f"""set -eu +python3 -c 'x=bytearray(268435456); __import__("time").sleep(60)' >/dev/null 2>&1 & +echo $! > {marker_dir}/memory.pid +python3 -c 'x=0\nwhile True: x+=1' >/dev/null 2>&1 & +echo $! > {marker_dir}/load.pid +""" + ssh(ssh_argv, pressure_setup) + checks["pressure_setup_completed"] = True + + container_setup = f"""set -eu +running=$(docker ps -q | head -1) +test -n "$running" +image=$(docker inspect --format '{{{{.Config.Image}}}}' "$running") +docker create --name {container} "$image" >/dev/null +docker start {container} >/dev/null +sleep 2 +""" + ssh(ssh_argv, container_setup) + checks["container_setup_completed"] = True + ssh(ssh_argv, "sync; zpool sync 2>/dev/null || true; sleep 8") + process_probe = ssh( + ssh_argv, + f"kill -0 $(cat {marker_dir}/memory.pid) $(cat {marker_dir}/load.pid)", + check=False, + ) + checks["pressure_processes_alive"] = process_probe.returncode == 0 + changed_codes_values = { + method: call(method) + for method in ("SysInfo", "NetworkInfo", "ListContainers") + } + checks["changed_methods_healthy"] = all( + code == 200 for code, _ in changed_codes_values.values() + ) + changed_sys = changed_codes_values["SysInfo"][1] + changed_net = changed_codes_values["NetworkInfo"][1] + changed_containers = changed_codes_values["ListContainers"][1] + interface_row = by_name(changed_net.get("interfaces", []), interface) + changed_disk = next( + ( + row + for row in changed_sys.get("disks", []) + if row.get("mount_point") == baseline_disk.get("mount_point") + ), + {}, + ) + checks["network_interface_row_visible"] = interface_row is not None + checks["network_interface_address_visible"] = interface_row is not None and any( + row.get("address") == address for row in interface_row.get("addresses", []) + ) + checks["network_dns_visible"] = dns in changed_net.get("dns_servers", []) + checks["memory_transition_visible"] = ( + int(changed_sys.get("used_memory", baseline_used)) > baseline_used + or int(changed_sys.get("available_memory", baseline_available)) + < baseline_available + ) + checks["swap_transition_visible"] = ( + int(changed_sys.get("total_swap", baseline_swap)) > baseline_swap + ) + checks["disk_row_visible"] = bool(changed_disk) + checks["disk_free_space_decreased"] = bool(changed_disk) and int( + changed_disk.get("free_size", 0) + ) < int(baseline_disk.get("free_size", 0)) + checks["container_transition_visible"] = any( + container == str(name).lstrip("/") + for row in changed_containers.get("containers", []) + for name in row.get("names", []) + ) + + cleanup = f"""set +e +docker rm -f {container} >/dev/null 2>&1 +kill $(cat {marker_dir}/memory.pid) $(cat {marker_dir}/load.pid) >/dev/null 2>&1 +swapoff $(cat {marker_dir}/loop) >/dev/null 2>&1 +losetup -d $(cat {marker_dir}/loop) >/dev/null 2>&1 +umount /etc/resolv.conf >/dev/null 2>&1 +ip link del {interface} >/dev/null 2>&1 +rm -f /run/systemd/network/00-{interface}.network +networkctl reload +rm -f {shlex.quote(mount_point)}/.dstack-telemetry-{lease} +rm -rf {marker_dir} +""" + ssh(ssh_argv, cleanup) + final_codes_values = { + method: call(method) + for method in ("SysInfo", "NetworkInfo", "ListContainers") + } + final_sys = final_codes_values["SysInfo"][1] + final_net = final_codes_values["NetworkInfo"][1] + final_containers = final_codes_values["ListContainers"][1] + checks["cleanup_disappeared"] = ( + all(code == 200 for code, _ in final_codes_values.values()) + and by_name(final_net.get("interfaces", []), interface) is None + and dns not in final_net.get("dns_servers", []) + and int(final_sys.get("total_swap", -1)) == baseline_swap + and not any( + container in row.get("names", []) + for row in final_containers.get("containers", []) + ) + ) + invalid_code, invalid_body = call("SysInfo", UNKNOWN_ID) + checks["unknown_identity_rejected"] = invalid_code >= 400 and bool(invalid_body) + status = "PASS" if all(checks.values()) else "FAIL" + summary = ( + "GuestApi network, DNS, memory, disk, swap, container, cleanup, and invalid-identity telemetry passed." + if status == "PASS" + else f"Telemetry checks failed: {sorted(k for k, value in checks.items() if not value)}" + ) + except Exception as error: + summary = f"Telemetry lifecycle failed: {type(error).__name__}" + finally: + emergency = f"""set +e +docker rm -f {container} >/dev/null 2>&1 +test -f {marker_dir}/memory.pid && kill $(cat {marker_dir}/memory.pid) >/dev/null 2>&1 +test -f {marker_dir}/load.pid && kill $(cat {marker_dir}/load.pid) >/dev/null 2>&1 +test -f {marker_dir}/loop && swapoff $(cat {marker_dir}/loop) >/dev/null 2>&1 +test -f {marker_dir}/loop && losetup -d $(cat {marker_dir}/loop) >/dev/null 2>&1 +mountpoint -q /etc/resolv.conf && umount /etc/resolv.conf >/dev/null 2>&1 +ip link del {interface} >/dev/null 2>&1 +rm -f /run/systemd/network/00-{interface}.network +networkctl reload >/dev/null 2>&1 +rm -rf {marker_dir} +""" + try: + ssh(ssh_argv, emergency, check=False) + except Exception as error: + cleanup_errors.append(type(error).__name__) + + if cleanup_errors: + status = "FAIL" + artifact = result_dir / "artifacts/system-telemetry-lifecycle.json" + artifact.parent.mkdir(parents=True, exist_ok=True) + artifact.write_text( + json.dumps( + { + "candidate_commit": runtime["candidate_commit"], + "checks": checks, + "cleanup_error_count": len(cleanup_errors), + "retained_addresses_dns_container_names_paths_or_native_responses": False, + }, + indent=2, + sort_keys=True, + ) + + "\n" + ) + result = { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + {"id": f"{CASE_ID}-step-{n:02d}", "status": status, "observed": summary} + for n in range(1, 4) + ], + "evidence": [ + { + "path": "artifacts/system-telemetry-lifecycle.json", + "sha256": hashlib.sha256(artifact.read_bytes()).hexdigest(), + } + ], + "remarks": "All mutations were scoped to the lease-owned guest and removed; evidence retains booleans and counts only.", + "duration_seconds": round(time.monotonic() - started, 3), + } + (result_dir / "result.json").write_text(json.dumps(result, indent=2) + "\n") + return 0 if status == "PASS" else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-005/case.md b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-005/case.md new file mode 100644 index 000000000..4205879f3 --- /dev/null +++ b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-005/case.md @@ -0,0 +1,69 @@ + + + +# TC-GOS-OBSERVABIL-005: Guest-agent watchdog recovery + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-observabil-005](../../../../catalog/feature-audit.md#req-gos-observabil-005) +- Risks: [risk-gos-observabil-005](../../../../catalog/feature-audit.md#risk-gos-observabil-005) +- Source: `dstack/guest-agent/src/server.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify guest-agent watchdog recovery across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for guest-agent watchdog recovery. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Make the watched external endpoint unresponsive and then healthy. + +**Expected results:** + +- The watchdog detects the failure within policy, triggers the configured recovery, and stops intervening after health returns. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-005/metadata.json b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-005/metadata.json new file mode 100644 index 000000000..b4bb8fd4f --- /dev/null +++ b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-005/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-observabil-005", + "title": "Guest-agent watchdog recovery", + "priority": "P1", + "requirements": [ + "req-gos-observabil-005" + ], + "risks": [ + "risk-gos-observabil-005" + ], + "tags": [ + "guest", + "observability-and-network" + ], + "fixture": { + "profile": "network-lifecycle", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Guest-agent watchdog recovery" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 240 + } +} diff --git a/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-005/run.py b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-005/run.py new file mode 100755 index 000000000..7e91ee6e9 --- /dev/null +++ b/test-suites/cases/01-guest-os/09-observability-and-network/tc-gos-observabil-005/run.py @@ -0,0 +1,262 @@ +#!/usr/bin/env python3 +"""Verify systemd watchdog replacement and stable guest-agent recovery.""" + +from __future__ import annotations + +import hashlib +import json +import os +import shlex +import subprocess +import time +from pathlib import Path +from typing import Any + +CASE_ID = "tc-gos-observabil-005" + + +def ssh( + argv: list[str], command: str, *, check: bool = True +) -> subprocess.CompletedProcess[str]: + """Run one bounded command through the manifest-recorded SSH route.""" + result = subprocess.run( + [*argv, command], + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + timeout=120, + check=False, + ) + if check and result.returncode: + raise RuntimeError( + f"guest command failed ({result.returncode}): {command!r}; " + f"stdout={result.stdout[-800:]!r}; stderr={result.stderr[-800:]!r}" + ) + return result + + +def unit_state(argv: list[str], unit: str) -> dict[str, str]: + """Read the bounded watchdog-relevant systemd unit properties.""" + output = ssh( + argv, + f"systemctl show {shlex.quote(unit)} " + "--property=MainPID,WatchdogUSec,ActiveState,SubState,NRestarts --no-pager", + ).stdout + return dict(line.split("=", 1) for line in output.splitlines() if "=" in line) + + +def health(argv: list[str], url: str) -> dict[str, Any]: + """Call the guest-local non-secret Worker.Version endpoint.""" + raw = ssh( + argv, + "curl --silent --show-error --fail-with-body --max-time 20 " + shlex.quote(url), + ).stdout + value = json.loads(raw) + if not isinstance(value, dict) or not value.get("version"): + raise AssertionError("Worker.Version response was incomplete") + return value + + +def emit(step: str, state: str) -> None: + """Emit one live step transition.""" + print(f"STEP {CASE_ID}-{step} {state}", flush=True) + + +def duration_usec(value: str) -> int: + """Parse the bounded systemd duration formats used by WatchdogUSec.""" + units = (("min", 60_000_000), ("ms", 1_000), ("us", 1), ("s", 1_000_000)) + for suffix, multiplier in units: + if value.endswith(suffix): + return int(float(value[: -len(suffix)]) * multiplier) + return int(value) + + +def main() -> int: + """Run the watchdog failure, recovery, and stability matrix.""" + result_dir = Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads(Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text()) + values = manifest.get("values", {}) + lifecycle = values.get("watchdog_lifecycle") if isinstance(values, dict) else None + ssh_argv = values.get("ssh_argv") if isinstance(values, dict) else None + status = "PASS" + summary = "guest-agent watchdog recovery matrix passed" + steps: list[dict[str, str]] = [] + observations: dict[str, Any] = {} + stage = "fixture" + frozen = False + unit = "dstack-guest-agent.service" + + try: + if not ( + isinstance(lifecycle, dict) + and lifecycle.get("destructive_actions_allowed") is True + and values.get("destructive_actions_allowed") is True + and isinstance(ssh_argv, list) + and lifecycle.get("freeze_signal") == "STOP" + ): + status = "BLOCKED" + summary = "missing capability: guest-agent-watchdog-lifecycle" + observations["missing_capability"] = "guest-agent-watchdog-lifecycle" + else: + unit = str(lifecycle["service_unit"]) + url = str(lifecycle["health_url"]) + start = time.strftime("%Y-%m-%d %H:%M:%S UTC", time.gmtime()) + + stage = "baseline" + emit("step-01", "START") + before = unit_state(ssh_argv, unit) + before_health = health(ssh_argv, url) + pid_before = int(before.get("MainPID", "0")) + watchdog_usec = duration_usec(before.get("WatchdogUSec", "0")) + if ( + pid_before <= 1 + or watchdog_usec <= 0 + or before.get("ActiveState") != "active" + ): + raise AssertionError(f"invalid watchdog baseline: {before}") + observations["baseline"] = { + "active": True, + "main_pid_positive": True, + "watchdog_usec": watchdog_usec, + "version": before_health["version"], + "restart_count": int(before.get("NRestarts", "0")), + } + steps.append( + { + "id": f"{CASE_ID}-step-01", + "status": "PASS", + "observed": "The case-owned guest-agent was active with a positive MainPID, a nonzero systemd watchdog interval, and a healthy guest-local Worker.Version endpoint.", + } + ) + emit("step-01", "PASS") + + stage = "watchdog-replacement" + emit("step-02", "START") + ssh( + ssh_argv, + f"systemctl kill --kill-who=main --signal=STOP {shlex.quote(unit)}", + ) + frozen = True + timeout = max(90.0, watchdog_usec / 1_000_000 * 3) + deadline = time.monotonic() + timeout + after: dict[str, str] = {} + while time.monotonic() < deadline: + after = unit_state(ssh_argv, unit) + current_pid = int(after.get("MainPID", "0")) + if ( + current_pid > 1 + and current_pid != pid_before + and after.get("ActiveState") == "active" + ): + frozen = False + break + time.sleep(1) + else: + raise AssertionError(f"watchdog did not replace frozen PID: {after}") + recovered_health = health(ssh_argv, url) + pid_recovered = int(after["MainPID"]) + observations["recovery"] = { + "pid_replaced": True, + "active": True, + "version_stable": recovered_health.get("version") + == before_health.get("version"), + "restart_count": int(after.get("NRestarts", "0")), + } + if not observations["recovery"]["version_stable"]: + raise AssertionError("Worker.Version changed after watchdog recovery") + steps.append( + { + "id": f"{CASE_ID}-step-02", + "status": "PASS", + "observed": "Freezing the service main process suppressed sd_notify heartbeats; systemd replaced it with a different active MainPID and Worker.Version recovered unchanged.", + } + ) + emit("step-02", "PASS") + + stage = "recovery-stability" + emit("step-03", "START") + time.sleep(watchdog_usec / 1_000_000 + 5) + stable = unit_state(ssh_argv, unit) + if ( + stable.get("ActiveState") != "active" + or int(stable.get("MainPID", "0")) != pid_recovered + ): + raise AssertionError("watchdog continued replacing the healthy service") + health(ssh_argv, url) + invalid = ssh( + ssh_argv, + "curl --silent --output /dev/null --write-out %{http_code} " + "--max-time 20 http://127.0.0.1:8090/prpc/DstackGuest.Info", + check=False, + ) + try: + invalid_code = int(invalid.stdout.strip()) + except ValueError as error: + raise AssertionError( + "invalid-route probe returned no HTTP status" + ) from error + if 200 <= invalid_code < 300: + raise AssertionError( + "internal DstackGuest method was exposed externally" + ) + journal = ssh( + ssh_argv, + f"journalctl -u {shlex.quote(unit)} --since {shlex.quote(start)} --no-pager", + ).stdout.lower() + observations["stability"] = { + "main_pid_stable_for_additional_interval": True, + "active": True, + "invalid_route_status": invalid_code, + "journal_mentions_watchdog": "watchdog" in journal, + } + steps.append( + { + "id": f"{CASE_ID}-step-03", + "status": "PASS", + "observed": "The recovered MainPID remained stable for another watchdog interval, health remained available, and the external listener rejected an internal DstackGuest route.", + } + ) + emit("step-03", "PASS") + except Exception as error: + status = "FAIL" + summary = f"{stage}: {error}" + observations["error_type"] = type(error).__name__ + observations["error"] = str(error) + finally: + if isinstance(ssh_argv, list) and isinstance(lifecycle, dict): + if frozen: + ssh( + ssh_argv, + f"systemctl kill --kill-who=main --signal=CONT {shlex.quote(unit)}", + check=False, + ) + ssh(ssh_argv, f"systemctl start {shlex.quote(unit)}", check=False) + + artifact = { + "case_id": CASE_ID, + "status": status, + "environment": "HARDWARE", + "observations": observations, + } + artifact_path = result_dir / "artifacts/watchdog-recovery.json" + artifact_path.parent.mkdir(parents=True, exist_ok=True) + artifact_path.write_text(json.dumps(artifact, indent=2) + "\n") + result = { + "schema_version": "1.0", + "case_id": CASE_ID, + "status": status, + "summary": summary, + "steps": steps, + "evidence": [ + { + "path": "artifacts/watchdog-recovery.json", + "sha256": hashlib.sha256(artifact_path.read_bytes()).hexdigest(), + } + ], + } + (result_dir / "result.json").write_text(json.dumps(result, indent=2) + "\n") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/10-platform-services/metadata.json b/test-suites/cases/01-guest-os/10-platform-services/metadata.json new file mode 100644 index 000000000..ac14dc6dd --- /dev/null +++ b/test-suites/cases/01-guest-os/10-platform-services/metadata.json @@ -0,0 +1,4 @@ +{ + "id": "section-guest-os-platform-services", + "title": "Platform Services and Image Integrity" +} diff --git a/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-001/case.md b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-001/case.md new file mode 100644 index 000000000..28e4f04ad --- /dev/null +++ b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-001/case.md @@ -0,0 +1,79 @@ + + + +# TC-GOS-PLATFORM-001: Local key provider PCCS selection and collateral lifecycle + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-gos-platform-001](../../../../catalog/feature-audit.md#req-gos-platform-001) +- Risks: [risk-gos-platform-001](../../../../catalog/feature-audit.md#risk-gos-platform-001) +- Source: `dstack/local-key-provider/src` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the SGX local key provider uses its configured PCCS for TDX quote collateral, handles cached and refreshed collateral correctly, and fails closed across dependency interruption and restart. + +## Preconditions + +1. A lease-owned SGX local-key-provider instance is configured through a lease-owned PCCS proxy or an isolated PCCS cache seeded for the hardware under test. +2. The fixture exposes controls for PCCS availability, cache freshness/expiry, provider restart, and redacted evidence capture without mutating shared host services. +3. TPM guest key provisioning is outside this case: `key_provider=tpm` is an independent Guest/VMM path and is not a mode of local-key-provider. + +## Test Data + +Use run-scoped identities and sentinel secrets that can be detected by hash without being retained in evidence. + +## Steps + + +### Step 1: Establish the baseline + +Query the effective configuration, service dependencies, listener/device state, and persisted files involved in this behavior. + +**Expected results:** + +- Required dependencies are healthy, ownership and permissions match policy, and no run-scoped object or sentinel is present before the action. + + +### Step 2: Exercise supported and boundary paths + +Submit a valid physical-TDX quote to the lease-owned SGX local-key-provider through a fresh local PCCS cache, repeat with the PCCS dependency unavailable while cached collateral remains valid, then force collateral refresh. Separately configure a public PCCS endpoint and record whether the platform registration policy permits it. + +**Expected results:** + +- The valid request succeeds through the configured local PCCS; valid cached collateral supports the documented offline interval; stale or expired collateral requires refresh; and the provider never silently falls back to an unconfigured public service. A public PCCS rejection caused by missing platform registration is reported as an expected deployment prerequisite rather than as TPM behavior. + + +### Step 3: Exercise failure and recovery + +Inject one invalid input and one dependency interruption appropriate to the behavior, restore the dependency, and repeat the valid operation. + +**Expected results:** + +- Failure is bounded, fails closed, produces actionable redacted diagnostics, leaves no partial trusted state, and the repeated valid operation succeeds exactly once after recovery. + + +### Step 4: Verify isolation and persistence + +Restart the affected service or VM when permitted, re-query state, and check adjacent app/instance/node identities. + +**Expected results:** + +- Documented state persists, transient state disappears, adjacent identities are unchanged, and no private key, credential, or plaintext sentinel appears in APIs, metrics, dashboards, journals, or artifacts. + +## Postconditions + +Remove run-scoped state, undo fault injection, and verify services and devices returned to their recorded baseline. diff --git a/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-001/metadata.json b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-001/metadata.json new file mode 100644 index 000000000..ff13e0ff0 --- /dev/null +++ b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-001/metadata.json @@ -0,0 +1,37 @@ +{ + "id": "tc-gos-platform-001", + "title": "Local key provider PCCS selection and collateral lifecycle", + "priority": "P0", + "requirements": [ + "req-gos-platform-001" + ], + "risks": [ + "risk-gos-platform-001" + ], + "tags": [ + "gos", + "platform-services-and-image-integrity" + ], + "fixture": { + "profile": "guest-lifecycle", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "Local key provider PCCS selection and collateral lifecycle", + "Local key provider sealing and identity isolation" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 1200 + } +} diff --git a/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-001/run.py b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-001/run.py new file mode 100755 index 000000000..05d03daff --- /dev/null +++ b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-001/run.py @@ -0,0 +1,249 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise physical local-provider sealing plus controlled PCCS collateral policy.""" + +from __future__ import annotations + +import hashlib +import json +import os +import re +import shutil +import subprocess +import tempfile +import time +from pathlib import Path +from typing import Any + +CASE_ID = "tc-gos-platform-001" +PHYSICAL_CASE_ID = "tc-gos-platform-002" +COLLATERAL_TEST = "tdx_quote_collateral_and_tcb_matrix" +TEST_RE = re.compile(r"test result: ok\. 1 passed; 0 failed") + + +def atomic_json(path: Path, value: Any) -> None: + """Write one JSON document atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as out: + json.dump(value, out, indent=2, sort_keys=True) + out.write("\n") + temporary = Path(out.name) + temporary.replace(path) + + +def main() -> int: + """Run the combined physical-provider and controlled-collateral matrix.""" + if os.environ.get("DSTACK_TEST_CASE_ID") != CASE_ID: + raise RuntimeError("unsupported case id") + started = time.monotonic() + result_dir = Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + artifacts = result_dir / "artifacts" + artifacts.mkdir(parents=True, exist_ok=True) + runtime = json.loads(Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text()) + repository = Path(runtime["repository"]) + physical_dir = result_dir / "physical-provider" + physical_dir.mkdir() + physical_log = artifacts / "physical-provider-controller.log" + collateral_log = artifacts / "collateral-policy.log" + physical_log.write_text("") + collateral_log.write_text("") + status = "FAIL" + failure = "" + observation: dict[str, Any] = {} + + try: + physical_env = { + **os.environ, + "DSTACK_TEST_CASE_ID": PHYSICAL_CASE_ID, + "DSTACK_TEST_RESULT_DIR": str(physical_dir), + } + physical = subprocess.run( + [ + "python3", + str( + repository + / "test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-002/run.py" + ), + ], + env=physical_env, + text=True, + capture_output=True, + timeout=900, + check=False, + ) + physical_log.write_text(physical.stdout + physical.stderr) + physical_result = json.loads((physical_dir / "result.json").read_text()) + if physical.returncode or physical_result.get("status") != "PASS": + raise RuntimeError( + f"physical local-provider matrix failed rc={physical.returncode}: " + f"{physical_result.get('summary')}" + ) + observation["physical_provider"] = { + "status": "PASS", + "environment": "PHYSICAL_TDX_GUEST_AND_HOST_SGX_GRAMINE_PROVIDER", + "stable_equivalent_identity": True, + "adjacent_identity_isolated": True, + "tampered_quote_rejected": True, + "invalid_frame_rejected": True, + "provider_quote_present": True, + "vm_restart_recovered": True, + } + + environment = { + **os.environ, + "CARGO_TARGET_DIR": str(runtime["cargo_target_dir"]), + } + collateral = subprocess.run( + [ + "cargo", + "test", + "-p", + "mock-attestation", + COLLATERAL_TEST, + "--lib", + "--", + "--nocapture", + ], + cwd=repository / "dstack", + env=environment, + text=True, + capture_output=True, + timeout=300, + check=False, + ) + collateral_output = collateral.stdout + collateral.stderr + collateral_log.write_text(collateral_output) + if collateral.returncode or not TEST_RE.search(collateral_output): + raise RuntimeError( + f"controlled PCCS/QVL matrix failed rc={collateral.returncode}" + ) + observation["controlled_collateral"] = { + "status": "PASS", + "test": COLLATERAL_TEST, + "configured_pccs_selected": True, + "rows": [ + "current", + "outdated-tcb", + "revoked", + "expired", + "signature-invalid", + "malformed", + "tampered-quote", + "network-outage", + "post-outage-recovery", + ], + "public_fallback_used": False, + "simulation_boundary": "mock-signed TDX PKI; no physical-origin claim", + } + status = "PASS" + except Exception as error: # noqa: BLE001 + failure = str(error) + + # Preserve only already-redacted physical artifacts; never copy fixture inputs. + physical_artifacts = physical_dir / "artifacts" + if physical_artifacts.is_dir(): + shutil.copytree( + physical_artifacts, + artifacts / "physical", + dirs_exist_ok=True, + ignore=shutil.ignore_patterns("*.key", "*.crt", "*.pem"), + ) + observation.update( + { + "status": status, + "failure": failure, + "pccs_configuration_sources": [ + "PCCS_URL passthrough in the Gramine manifest", + "PCCS_URL default/override in the provider deployment", + ], + "tpm_substitution_used": False, + "shared_provider_mutated": False, + "duration_seconds": round(time.monotonic() - started, 3), + } + ) + matrix_path = artifacts / "pccs-collateral-matrix.json" + atomic_json(matrix_path, observation) + artifact_rows = [ + { + "path": "artifacts/pccs-collateral-matrix.json", + "step_id": f"{CASE_ID}-step-02", + "name": "PCCS collateral lifecycle matrix", + "description": "Redacted physical-provider and controlled collateral observations.", + }, + { + "path": "artifacts/physical-provider-controller.log", + "step_id": f"{CASE_ID}-step-02", + "name": "Physical provider controller log", + "description": "Bounded controller diagnostics without quote or key material.", + }, + { + "path": "artifacts/collateral-policy.log", + "step_id": f"{CASE_ID}-step-03", + "name": "Controlled collateral policy log", + "description": "Native production-QVL test output for collateral status, mutation, outage, and recovery.", + }, + ] + atomic_json(artifacts / "manifest.json", {"artifacts": artifact_rows}) + summary = ( + "Physical local-provider and controlled PCCS collateral lifecycle passed" + if status == "PASS" + else f"PCCS collateral lifecycle failed: {failure}" + ) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{CASE_ID}-step-01", + "status": status, + "observed": "A physical TDX guest and host-managed SGX/Gramine provider were available; the provider was treated as read-only shared hardware." + if status == "PASS" + else failure, + }, + { + "id": f"{CASE_ID}-step-02", + "status": status, + "observed": "Physical quote provisioning passed stable identity, adjacent isolation, tamper/frame rejection, provider quote, VM restart, and cleanup rows." + if status == "PASS" + else failure, + }, + { + "id": f"{CASE_ID}-step-03", + "status": status, + "observed": "The configured production CollateralClient/QVL path passed current, TCB status, expiry, signature, malformed, tampered, outage, no-fallback, and recovery rows." + if status == "PASS" + else failure, + }, + { + "id": f"{CASE_ID}-step-04", + "status": status, + "observed": "The guest restart preserved provider-derived identity, the peer stayed isolated, no TPM substitution occurred, and case-owned resources were released." + if status == "PASS" + else failure, + }, + ], + "artifacts": artifact_rows, + "evidence": [ + { + "path": row["path"], + "sha256": hashlib.sha256( + (result_dir / row["path"]).read_bytes() + ).hexdigest(), + } + for row in artifact_rows + ], + "remarks": "Hardware proves the physical TDX-to-SGX provisioning path. Destructive PCCS cache-age/outage rows use a case-owned mock-signed TDX PKI through the production QVL client and do not claim physical origin. The host-managed SGX enclave is not restarted or reconfigured by this case.", + }, + ) + return 0 if status == "PASS" else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-002/case.md b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-002/case.md new file mode 100644 index 000000000..38e2698ce --- /dev/null +++ b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-002/case.md @@ -0,0 +1,79 @@ + + + +# TC-GOS-PLATFORM-002: Local key provider sealing and identity isolation + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-gos-platform-002](../../../../catalog/feature-audit.md#req-gos-platform-002) +- Risks: [risk-gos-platform-002](../../../../catalog/feature-audit.md#risk-gos-platform-002) +- Source: `dstack/local-key-provider/src` +- Prepared helper: `cases/01-guest-os/10-platform-services/tc-gos-platform-002/run.py` consumes the lease-owned primary/peer TDX guests and the configured local-key-provider endpoint; it never persists quotes, private keys, decrypted keys, or ciphertext. + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify local key provider sealing and identity isolation with explicit success, boundary, failure, restart, and isolation observations. + +## Preconditions + +1. The target runs in an isolated environment with effective configuration and synchronized evidence capture. +2. Baseline service, file, process, device, listener, and secret-redaction state has been recorded. + +## Test Data + +Use run-scoped identities and sentinel secrets that can be detected by hash without being retained in evidence. + +## Steps + + +### Step 1: Establish the baseline + +Query the effective configuration, service dependencies, listener/device state, and persisted files involved in this behavior. + +**Expected results:** + +- Required dependencies are healthy, ownership and permissions match policy, and no run-scoped object or sentinel is present before the action. + + +### Step 2: Exercise supported and boundary paths + +Generate report data from ephemeral X25519 public keys, request physical TDX quotes from two lease-owned guests with different app identities, and submit each quote to the configured SGX local-key-provider. Repeat the primary request and submit a tampered quote. + +**Expected results:** + +- Each response contains a provider quote and a sealed key decryptable only by the matching ephemeral private key. The decrypted primary key is stable for the same measured guest identity, the peer identity derives a different key, and a tampered quote returns no key. + + +### Step 3: Exercise failure and recovery + +Send invalid length framing and a structurally valid but tampered quote to the lease-visible provider endpoint, then repeat a valid request. + +**Expected results:** + +- Both invalid requests fail closed without key material, the provider remains available, and a repeated valid request succeeds. Diagnostics and stored evidence contain no quote, private key, decrypted key, ciphertext, or credential. + + +### Step 4: Verify isolation and persistence + +Restart the affected service or VM when permitted, re-query state, and check adjacent app/instance/node identities. + +**Expected results:** + +- Documented state persists, transient state disappears, adjacent identities are unchanged, and no private key, credential, or plaintext sentinel appears in APIs, metrics, dashboards, journals, or artifacts. + +## Postconditions + +Remove run-scoped state, undo fault injection, and verify services and devices returned to their recorded baseline. diff --git a/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-002/metadata.json b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-002/metadata.json new file mode 100644 index 000000000..5063a837d --- /dev/null +++ b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-002/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-platform-002", + "title": "Local key provider sealing and identity isolation", + "priority": "P0", + "requirements": [ + "req-gos-platform-002" + ], + "risks": [ + "risk-gos-platform-002" + ], + "tags": [ + "gos", + "platform-services-and-image-integrity" + ], + "fixture": { + "profile": "guest-readonly", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Local key provider sealing and identity isolation" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 900 + } +} diff --git a/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-002/run.py b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-002/run.py new file mode 100755 index 000000000..652d54937 --- /dev/null +++ b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-002/run.py @@ -0,0 +1,301 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Validate physical TDX quote sealing against the SGX local key provider.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import socket +import struct +import subprocess +import tempfile +import time +import urllib.error +import urllib.request +from typing import Any + +from nacl.public import PrivateKey, SealedBox + +CASE_ID = "tc-gos-platform-002" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Atomically write one JSON evidence document.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as f: + json.dump(value, f, indent=2, sort_keys=True) + f.write("\n") + temporary = pathlib.Path(f.name) + temporary.replace(path) + + +def rpc(url: str, method: str, body: dict[str, Any]) -> dict[str, Any]: + """Call one JSON guest RPC without retaining sensitive response data.""" + request = urllib.request.Request( + url.replace("{method}", method), + data=json.dumps(body, separators=(",", ":")).encode(), + headers={"content-type": "application/json"}, + ) + with urllib.request.urlopen(request, timeout=90) as response: + value = json.load(response) + if not isinstance(value, dict): + raise RuntimeError("guest RPC returned a non-object") + return value + + +def provider_request(host: str, port: int, quote: bytes) -> dict[str, Any]: + """Send one framed in-memory quote to the lease-visible provider.""" + payload = json.dumps({"quote": list(quote)}, separators=(",", ":")).encode() + with socket.create_connection((host, port), timeout=90) as stream: + stream.settimeout(90) + stream.sendall(struct.pack(">I", len(payload)) + payload) + header = stream.recv(4) + if len(header) != 4: + raise RuntimeError("provider closed before response header") + expected = struct.unpack(">I", header)[0] + response = bytearray() + while len(response) < expected: + part = stream.recv(expected - len(response)) + if not part: + raise RuntimeError("provider closed before complete response") + response.extend(part) + value = json.loads(response) + if not isinstance(value, dict): + raise RuntimeError("provider returned a non-object") + return value + + +def derive(tappd_url: str, host: str, port: int) -> tuple[bytes, dict[str, Any]]: + """Derive a key in memory and return it with non-sensitive observations.""" + private_key = PrivateKey.generate() + report_data = bytes(private_key.public_key) + bytes(32) + quote_attempts = 0 + while True: + quote_attempts += 1 + try: + quote_value = rpc(tappd_url, "RawQuote", {"report_data": report_data.hex()}) + break + except (ConnectionError, OSError, TimeoutError, urllib.error.URLError): + if quote_attempts >= 15: + raise + time.sleep(2) + quote = bytes.fromhex(str(quote_value["quote"])) + attempts = 0 + while True: + attempts += 1 + try: + response = provider_request(host, port, quote) + break + except (OSError, RuntimeError, json.JSONDecodeError): + if attempts >= 15: + raise + time.sleep(2) + ciphertext = bytes(response["encrypted_key"]) + provider_quote = bytes(response["provider_quote"]) + plaintext = SealedBox(private_key).decrypt(ciphertext) + observation = { + "tdx_quote_length": len(quote), + "encrypted_key_length": len(ciphertext), + "provider_quote_length": len(provider_quote), + "provider_quote_present": bool(provider_quote), + "decryption_succeeded": len(plaintext) == 32, + "quote_rpc_attempts": quote_attempts, + "provider_request_attempts": attempts, + } + return plaintext, observation + + +def main() -> int: + """Run the physical local-provider sealing and isolation matrix.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE_ID: + raise RuntimeError(f"unsupported case id: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + values = manifest["values"] + primary_url = values["services"]["Tappd"]["url"] + provider = values["services"]["LocalKeyProvider"] + peer = values["local_provider_peer"] + host, port = str(provider["host"]), int(provider["port"]) + observations: dict[str, Any] = {} + failures: list[str] = [] + stage = "primary_first" + + try: + primary_key_1, observations["primary_first"] = derive(primary_url, host, port) + stage = "primary_repeat" + primary_key_2, observations["primary_repeat"] = derive(primary_url, host, port) + stage = "peer_identity" + peer_key, observations["peer"] = derive(peer["tappd_url"], host, port) + observations["same_identity_stable"] = primary_key_1 == primary_key_2 + observations["peer_identity_isolated"] = primary_key_1 != peer_key + if primary_key_1 != primary_key_2: + failures.append( + "primary derived key changed across equivalent valid quotes" + ) + if primary_key_1 == peer_key: + failures.append("different app identities derived the same key") + + # Alter one byte in a fresh valid quote and prove that no response key is returned. + stage = "tampered_quote" + private_key = PrivateKey.generate() + report_data = bytes(private_key.public_key) + bytes(32) + quote = bytearray( + bytes.fromhex( + str( + rpc(primary_url, "RawQuote", {"report_data": report_data.hex()})[ + "quote" + ] + ) + ) + ) + quote[len(quote) // 2] ^= 1 + stage = "invalid_frame" + try: + tampered = provider_request(host, port, bytes(quote)) + observations["tampered_quote_rejected"] = not bool( + tampered.get("encrypted_key") + ) + except (OSError, ValueError, RuntimeError, json.JSONDecodeError): + observations["tampered_quote_rejected"] = True + if not observations["tampered_quote_rejected"]: + failures.append("tampered quote returned encrypted key material") + + # Invalid frame length must be bounded; a subsequent valid request proves recovery. + try: + with socket.create_connection((host, port), timeout=10) as stream: + stream.sendall(struct.pack(">I", 0)) + stream.shutdown(socket.SHUT_WR) + invalid_reply = stream.recv(32) + observations["invalid_frame_rejected"] = len(invalid_reply) == 0 + except OSError: + observations["invalid_frame_rejected"] = True + if not observations["invalid_frame_rejected"]: + failures.append("invalid zero-length frame was not rejected") + stage = "post_error_recovery" + recovered_key, observations["post_error_recovery"] = derive( + primary_url, host, port + ) + observations["post_error_key_stable"] = recovered_key == primary_key_1 + if recovered_key != primary_key_1: + failures.append( + "valid request after invalid input did not recover stable key" + ) + + # Restart only the lease-owned primary VM, never the host/provider. + stage = "vm_restart" + cli = [str(item) for item in values["vmm_cli_argv"]] + vm_id = str(values["vm_id"]) + subprocess.run( + [*cli, "stop", vm_id], + check=True, + capture_output=True, + text=True, + timeout=180, + ) + subprocess.run( + [*cli, "start", vm_id], + check=True, + capture_output=True, + text=True, + timeout=180, + ) + for _ in range(120): + status = subprocess.run( + [*cli, "info", "--json", vm_id], + check=True, + capture_output=True, + text=True, + timeout=30, + ) + info = json.loads(status.stdout) + if info.get("boot_progress") == "done" and info.get("status") == "running": + break + time.sleep(5) + else: + raise RuntimeError( + "lease-owned primary VM did not become ready after restart" + ) + stage = "after_vm_restart" + restarted_key, observations["after_vm_restart"] = derive( + primary_url, host, port + ) + observations["restart_key_stable"] = restarted_key == primary_key_1 + if restarted_key != primary_key_1: + failures.append("derived key changed after lease-owned VM restart") + except ( + Exception + ) as error: # Result captures only the error class/message, never key material. + failures.append(f"{stage}: {type(error).__name__}: {error}") + observations["failed_stage"] = stage + + artifact = { + "path": "artifacts/local-provider-sealing-observations.json", + "step_id": f"{case_id}-step-02", + "name": "Local provider sealing observations", + "description": "Lengths and boolean assertions proving physical quote acceptance, same-identity stability, cross-identity isolation, invalid-input rejection, recovery, and VM-restart persistence without retaining quotes or key material.", + } + observations["sensitive_values_persisted"] = False + observations["observation_sha256"] = hashlib.sha256( + json.dumps(observations, sort_keys=True).encode() + ).hexdigest() + atomic_json(result_dir / artifact["path"], observations) + atomic_json(result_dir / "artifacts" / "manifest.json", {"artifacts": [artifact]}) + status = "PASS" if not failures else "FAIL" + steps = [ + { + "id": f"{case_id}-step-01", + "status": status, + "observed": "Lease-owned primary and peer hardware guests plus the configured SGX local-key-provider endpoint were available." + if not failures + else "Fixture or baseline operation failed; see redacted summary.", + }, + { + "id": f"{case_id}-step-02", + "status": status, + "observed": "Same-identity stability, peer identity isolation, sealed-box recipient binding, and tamper rejection passed." + if not failures + else "One or more sealing or identity assertions failed.", + }, + { + "id": f"{case_id}-step-03", + "status": status, + "observed": "Invalid framing and tampered evidence failed closed; the next valid request recovered." + if not failures + else "Failure/recovery assertions did not all pass.", + }, + { + "id": f"{case_id}-step-04", + "status": status, + "observed": "Lease-owned VM restart preserved the identity-scoped derived key and peer isolation." + if not failures + else "Restart/isolation assertions did not all pass.", + }, + ] + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": "Physical SGX local-key-provider sealing, identity isolation, rejection, recovery, and lease-owned VM restart checks passed." + if not failures + else "; ".join(failures)[:800], + "steps": steps, + "artifacts": [artifact], + "remarks": "No quote, private key, decrypted key, ciphertext, or credential was persisted. The physical host and shared provider were not restarted.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-003/case.md b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-003/case.md new file mode 100644 index 000000000..487460343 --- /dev/null +++ b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-003/case.md @@ -0,0 +1,78 @@ + + + +# TC-GOS-PLATFORM-003: Host-shared mount and unmount command + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-platform-003](../../../../catalog/feature-audit.md#req-gos-platform-003) +- Risks: [risk-gos-platform-003](../../../../catalog/feature-audit.md#risk-gos-platform-003) +- Source: `dstack/dstack-util/src/host_shared.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify host-shared mount and unmount command with explicit success, boundary, failure, restart, and isolation observations. + +## Preconditions + +1. The target runs in an isolated environment with effective configuration and synchronized evidence capture. +2. Baseline service, file, process, device, listener, and secret-redaction state has been recorded. + +## Test Data + +Use run-scoped identities and sentinel secrets that can be detected by hash without being retained in evidence. + +## Steps + + +### Step 1: Establish the baseline + +Query the effective configuration, service dependencies, listener/device state, and persisted files involved in this behavior. + +**Expected results:** + +- Required dependencies are healthy, ownership and permissions match policy, and no run-scoped object or sentinel is present before the action. + + +### Step 2: Exercise supported and boundary paths + +Exercise dstack-util host-shared mount/unmount with labeled disk, 9p fallback, already-mounted, absent, read-only, and cleanup paths. + +**Expected results:** + +- The correct source mounts read-only once, fallback is logged, unmount is idempotent, and failure never leaves a writable or leaked mount. + + +### Step 3: Exercise failure and recovery + +Inject one invalid input and one dependency interruption appropriate to the behavior, restore the dependency, and repeat the valid operation. + +**Expected results:** + +- Failure is bounded, fails closed, produces actionable redacted diagnostics, leaves no partial trusted state, and the repeated valid operation succeeds exactly once after recovery. + + +### Step 4: Verify isolation and persistence + +Restart the affected service or VM when permitted, re-query state, and check adjacent app/instance/node identities. + +**Expected results:** + +- Documented state persists, transient state disappears, adjacent identities are unchanged, and no private key, credential, or plaintext sentinel appears in APIs, metrics, dashboards, journals, or artifacts. + +## Postconditions + +Remove run-scoped state, undo fault injection, and verify services and devices returned to their recorded baseline. diff --git a/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-003/metadata.json b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-003/metadata.json new file mode 100644 index 000000000..4459fe121 --- /dev/null +++ b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-003/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-platform-003", + "title": "Host-shared mount and unmount command", + "priority": "P1", + "requirements": [ + "req-gos-platform-003" + ], + "risks": [ + "risk-gos-platform-003" + ], + "tags": [ + "gos", + "platform-services-and-image-integrity" + ], + "fixture": { + "profile": "storage-lifecycle", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Host-shared mount and unmount command" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-003/run.py b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-003/run.py new file mode 100755 index 000000000..8f5ce6d23 --- /dev/null +++ b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-003/run.py @@ -0,0 +1,182 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise labeled-disk priority, 9p fallback, faults, recovery, and cleanup.""" + +from __future__ import annotations + +import json +import os +import pathlib +import subprocess +import tempfile +from typing import Any + +CASE_ID = "tc-gos-platform-003" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile("w", dir=path.parent, delete=False) as output: + json.dump(value, output, indent=2, sort_keys=True) + output.write("\n") + temporary = pathlib.Path(output.name) + temporary.replace(path) + + +def run( + argv: list[str], *, data: bytes | None = None, timeout: int = 180 +) -> subprocess.CompletedProcess[bytes]: + """Run one bounded command.""" + return subprocess.run( + argv, input=data, capture_output=True, timeout=timeout, check=False + ) + + +def vm_ids(argv: list[str]) -> list[str]: + """Return the stable VMM inventory identifiers.""" + completed = run(argv, timeout=30) + if completed.returncode: + raise RuntimeError("failed to observe adjacent VM inventory") + value = json.loads(completed.stdout) + rows = value if isinstance(value, list) else value.get("vms", []) + return sorted( + str(row.get("id")) for row in rows if isinstance(row, dict) and row.get("id") + ) + + +def main() -> int: + """Run the host-shared source lifecycle.""" + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + artifacts = result_dir / "artifacts" + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + values = manifest.get("values") or {} + ssh = [str(value) for value in values.get("ssh_argv") or []] + list_vms = [str(value) for value in values.get("list_vms_argv") or []] + status = "PASS" + summary = "Host-shared labeled-disk and 9p lifecycle passed." + evidence: dict[str, Any] = {} + try: + if ( + not ssh + or not list_vms + or values.get("destructive_actions_allowed") is not True + ): + raise RuntimeError("fixture omitted lease-owned guest controls") + before = vm_ids(list_vms) + script = ( + pathlib.Path(str(runtime["repository"])) + / "test-suites/shared/automation/host-shared-lifecycle.sh" + ) + installed = run( + [*ssh, "install -m 0755 /dev/stdin /run/dstack-test-host-shared-lifecycle"], + data=script.read_bytes(), + ) + if installed.returncode: + raise RuntimeError("failed to install host-shared lifecycle") + executed = run([*ssh, "/run/dstack-test-host-shared-lifecycle"], timeout=240) + artifacts.mkdir(parents=True, exist_ok=True) + (artifacts / "host-shared-lifecycle.log").write_bytes( + executed.stdout + executed.stderr + ) + rows = [ + row + for row in executed.stdout.decode(errors="replace").splitlines() + if row.startswith("{") + ] + if executed.returncode or not rows: + tail = (executed.stdout + executed.stderr).decode(errors="replace")[-2000:] + raise RuntimeError( + f"host-shared lifecycle rc={executed.returncode}: {tail}" + ) + evidence = json.loads(rows[-1]) + required = ( + "disk_source", + "disk_read_only", + "invalid_disk_fallback_9p", + "nine_p_content_hash_matched", + "duplicate_unmount_rejected", + "dependency_fault_rejected", + "dependency_recovery", + "invalid_target_rejected", + "mount_count_restored", + ) + if evidence.get("checks", 0) < 24 or not all( + evidence.get(key) is True for key in required + ): + raise RuntimeError("host-shared evidence omitted a required row") + after = vm_ids(list_vms) + if before != after or str(values.get("vm_id")) not in after: + raise RuntimeError("adjacent VM inventory changed") + evidence["adjacent_vm_inventory_stable"] = True + evidence["inventory_size"] = len(after) + except ( + KeyError, + OSError, + RuntimeError, + subprocess.SubprocessError, + ValueError, + ) as error: + status = "FAIL" + summary = f"{type(error).__name__}: {error}" + artifact_entries = [ + { + "path": "artifacts/host-shared-lifecycle.json", + "step_id": f"{CASE_ID}-step-01", + "name": "Host-shared lifecycle matrix", + "description": "Boolean and count evidence for labeled-disk priority, read-only policy, 9p fallback, dependency faults, recovery, isolation, and cleanup.", + }, + { + "path": "artifacts/host-shared-lifecycle.log", + "step_id": f"{CASE_ID}-step-02", + "name": "Host-shared native log", + "description": "Native bounded lifecycle output; shared configuration content is represented only by an in-guest equality check.", + }, + ] + atomic_json(artifacts / "host-shared-lifecycle.json", evidence) + atomic_json(artifacts / "manifest.json", {"artifacts": artifact_entries}) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{CASE_ID}-step-01", + "status": status, + "observed": "A DSTACKSHR loop disk took priority and mounted read-only." + if status == "PASS" + else summary, + }, + { + "id": f"{CASE_ID}-step-02", + "status": status, + "observed": "An invalid labeled disk fell back to the case-owned 9p source; injected mount failure was atomic and recovered." + if status == "PASS" + else summary, + }, + { + "id": f"{CASE_ID}-step-03", + "status": status, + "observed": "Duplicate unmount and invalid target failed closed; loops, mounts, files, and adjacent VM inventory returned to baseline." + if status == "PASS" + else summary, + }, + ], + "artifacts": artifact_entries, + "remarks": "No host-shared file content is persisted; the harness records only equality booleans and counts.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-005/case.md b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-005/case.md new file mode 100644 index 000000000..da74c4857 --- /dev/null +++ b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-005/case.md @@ -0,0 +1,82 @@ + + + +# TC-GOS-PLATFORM-005: Guest kernel and userspace hardening + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-gos-platform-005](../../../../catalog/feature-audit.md#req-gos-platform-005) +- Risks: [risk-gos-platform-005](../../../../catalog/feature-audit.md#risk-gos-platform-005) +- Source: `os/common/rootfs/sysctl.d/99-dstack.conf` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify guest kernel and userspace hardening with explicit success, boundary, failure, restart, and isolation observations. + +## Preconditions + +1. The target runs in an isolated environment with effective configuration and synchronized evidence capture. +2. Baseline service, file, process, device, listener, and secret-redaction state has been recorded. + +## Test Data + +Use run-scoped identities and sentinel secrets that can be detected by hash without being retained in evidence. + +## Steps + + +### Step 1: Establish the baseline + +Query the effective configuration, service dependencies, listener/device state, and persisted files involved in this behavior. + +**Expected results:** + +- Required dependencies are healthy, ownership and permissions match policy, and no run-scoped object or sentinel is present before the action. + + +### Step 2: Exercise supported and boundary paths + +Audit kernel config, sysctl, mounts, capabilities, device nodes, SSH/accounts, network discovery, and writable executable paths. + +**Expected results:** + +- The image exposes only required devices/services, applies hardening settings, has no default credential, and application containers cannot modify measured/privileged host state. + + +### Step 3: Exercise failure and recovery + +Inject one invalid input and one dependency interruption appropriate to the behavior, restore the dependency, and repeat the valid operation. + +**Expected results:** + +- Failure is bounded, fails closed, produces actionable redacted diagnostics, leaves no partial trusted state, and the repeated valid operation succeeds exactly once after recovery. + + +### Step 4: Verify isolation and persistence + +Restart the affected service or VM when permitted, re-query state, and check adjacent app/instance/node identities. + +**Expected results:** + +- Documented state persists, transient state disappears, adjacent identities are unchanged, and no private key, credential, or plaintext sentinel appears in APIs, metrics, dashboards, journals, or artifacts. + +## Post-baseline regression matrix + +For both Yocto and mkosi images, assert the effective SELinux kernel gates plus nftables bridge/CHECKSUM capabilities and shipped modules. Start an Incus-compatible bridge workload, verify rule programming and xtables-lock handling, and fail closed by dropping the WireGuard configuration when rules cannot be applied. + +## Postconditions + +Remove run-scoped state, undo fault injection, and verify services and devices returned to their recorded baseline. diff --git a/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-005/metadata.json b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-005/metadata.json new file mode 100644 index 000000000..3aef94012 --- /dev/null +++ b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-005/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-platform-005", + "title": "Guest kernel and userspace hardening", + "priority": "P0", + "requirements": [ + "req-gos-platform-005" + ], + "risks": [ + "risk-gos-platform-005" + ], + "tags": [ + "gos", + "platform-services-and-image-integrity" + ], + "fixture": { + "profile": "guest-readonly", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Guest kernel and userspace hardening" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-005/run.py b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-005/run.py new file mode 100755 index 000000000..6d68f2c0d --- /dev/null +++ b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-005/run.py @@ -0,0 +1,364 @@ +#!/usr/bin/env python3 +"""Verify declared guest hardening and a real non-privileged workload boundary.""" + +from __future__ import annotations + +import hashlib +import json +import os +import shlex +import subprocess +import time +from pathlib import Path +from typing import Any + +CASE_ID = "tc-gos-platform-005" + + +def ssh( + argv: list[str], command: str, *, check: bool = True, timeout: int = 90 +) -> subprocess.CompletedProcess[str]: + """Run one bounded command through a fixture-recorded SSH route.""" + result = subprocess.run( + [*argv, command], + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + timeout=timeout, + check=False, + ) + if check and result.returncode: + raise RuntimeError( + f"guest command failed ({result.returncode}): {command!r}; " + f"stdout={result.stdout[-600:]!r}; stderr={result.stderr[-600:]!r}" + ) + return result + + +def emit(step: str, state: str) -> None: + """Emit a live case-step transition.""" + print(f"STEP {CASE_ID}-{step} {state}", flush=True) + + +def target_container(argv: list[str]) -> str: + """Resolve the unique measured boundary container.""" + ids = ssh( + argv, "docker ps -aq --filter label=com.docker.compose.service=boundary-target" + ).stdout.split() + if len(ids) != 1: + raise AssertionError( + f"expected one boundary-target container, found {len(ids)}" + ) + return ids[0] + + +def inspect(argv: list[str], container: str) -> dict[str, Any]: + """Return one Docker container inspection object.""" + value = json.loads(ssh(argv, f"docker inspect {shlex.quote(container)}").stdout) + if not isinstance(value, list) or len(value) != 1 or not isinstance(value[0], dict): + raise AssertionError("docker inspect returned an unexpected shape") + return value[0] + + +def main() -> int: + """Execute the declared hardening, recovery, and isolation matrix.""" + result_dir = Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads(Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text()) + values = manifest.get("values", {}) + fixture = ( + values.get("guest_hardening_lifecycle") if isinstance(values, dict) else None + ) + status = "PASS" + summary = "Declared guest hardening and non-privileged workload boundary passed" + steps: list[dict[str, str]] = [] + observations: dict[str, Any] = {} + cleanup = {"marker_removed": False, "docker_recovered": False} + stage = "fixture" + primary_ssh: list[str] = [] + adjacent_ssh: list[str] = [] + marker = "" + policy_hashes = "" + + try: + if not ( + isinstance(fixture, dict) + and fixture.get("destructive_actions_allowed") is True + and isinstance(fixture.get("primary"), dict) + and isinstance(fixture.get("adjacent"), dict) + ): + status = "BLOCKED" + summary = "missing capability: guest-hardening-boundary-lifecycle" + steps.append( + { + "id": f"{CASE_ID}-step-01", + "status": "BLOCKED", + "observed": "The case-scoped hardening fixture was not declared.", + } + ) + else: + primary = fixture["primary"] + adjacent = fixture["adjacent"] + primary_ssh = [str(x) for x in primary["ssh_argv"]] + adjacent_ssh = [str(x) for x in adjacent["ssh_argv"]] + marker_hash = hashlib.sha256( + str(manifest.get("lease_id", "")).encode() + ).hexdigest() + marker = f"/tmp/dstack-hardening-{marker_hash[:12]}" + + stage = "baseline" + emit("step-01", "START") + conntrack = int( + ssh( + primary_ssh, "sysctl -n net.netfilter.nf_conntrack_max" + ).stdout.strip() + ) + if conntrack != int( + fixture["declared_policy"]["net.netfilter.nf_conntrack_max"] + ): + raise AssertionError(f"nf_conntrack_max={conntrack}") + sshd = ssh( + primary_ssh, + "grep -Ei '^(PasswordAuthentication|PermitRootLogin)[[:space:]]' /etc/ssh/sshd_config.d/10-dstack.conf | tr A-Z a-z", + ).stdout.lower() + if "passwordauthentication no" not in sshd or not any( + x in sshd + for x in ( + "permitrootlogin prohibit-password", + "permitrootlogin without-password", + ) + ): + raise AssertionError( + "effective SSH password policy differed from the image declaration" + ) + unlocked = ssh( + primary_ssh, "awk -F: '$2 !~ /^[!*]/ {print $1}' /etc/shadow" + ).stdout.split() + if unlocked: + raise AssertionError( + "one or more local accounts had an unlocked password" + ) + for unit in ( + "docker.service", + "sshd.service", + "dstack-guest-agent.service", + ): + ssh(primary_ssh, f"systemctl is-active --quiet {shlex.quote(unit)}") + measured_paths = [str(x) for x in fixture["measured_readonly_paths"]] + if not measured_paths: + raise AssertionError("no measured host policy paths were declared") + quoted_paths = " ".join(shlex.quote(x) for x in measured_paths) + policy_hashes = ssh(primary_ssh, f"sha256sum {quoted_paths}").stdout + if len(policy_hashes.splitlines()) != len(measured_paths): + raise AssertionError("host policy path measurement was incomplete") + ssh( + primary_ssh, + "test ! -e /dev/kvm && " + "zgrep -qx CONFIG_STRICT_DEVMEM=y /proc/config.gz && " + "zgrep -qx CONFIG_IO_STRICT_DEVMEM=y /proc/config.gz", + ) + observations["baseline"] = { + "declared_conntrack_exact": True, + "ssh_password_auth_disabled": True, + "password_accounts_locked": True, + "required_services_active": True, + "host_policy_paths_measured": True, + "host_kvm_absent_and_devmem_strict": True, + } + steps.append( + { + "id": f"{CASE_ID}-step-01", + "status": "PASS", + "observed": "The effective conntrack, SSH/account, service, read-only mount, and device baseline matched the checked-in release policy.", + } + ) + emit("step-01", "PASS") + + stage = "workload-boundary" + emit("step-02", "START") + container = target_container(primary_ssh) + cfg = inspect(primary_ssh, container).get("HostConfig", {}) + cap_drop = [str(x).upper() for x in cfg.get("CapDrop") or []] + security = [str(x).lower() for x in cfg.get("SecurityOpt") or []] + if cfg.get("Privileged") is not False or cfg.get("NetworkMode") != "none": + raise AssertionError("workload gained privileged or network access") + if cfg.get("PidMode") not in ("", None) or "ALL" not in cap_drop: + raise AssertionError( + "workload gained host PID namespace or capabilities" + ) + if not any("no-new-privileges" in x for x in security): + raise AssertionError("no-new-privileges was absent") + denied = ssh( + primary_ssh, + f"docker exec {shlex.quote(container)} sh -c " + "'printf blocked > /proc/sys/kernel/hostname'", + check=False, + ) + if denied.returncode == 0: + raise AssertionError("container modified its kernel hostname sysctl") + container_path_checks = " && ".join( + f"test ! -e {shlex.quote(path)}" for path in measured_paths + ) + container_checks = ( + "test ! -e /dev/kvm && test ! -e /dev/mem && " + "test ! -e /run/systemd/system && " + container_path_checks + ) + ssh( + primary_ssh, + f"docker exec {shlex.quote(container)} sh -c {shlex.quote(container_checks)}", + ) + observations["boundary"] = { + "unprivileged": True, + "network_none": True, + "host_pid_absent": True, + "all_capabilities_dropped": True, + "no_new_privileges": True, + "sysctl_write_rejected": True, + "host_devices_systemd_and_policy_paths_absent": True, + } + steps.append( + { + "id": f"{CASE_ID}-step-02", + "status": "PASS", + "observed": "The measured application container lacked host network/PID, privilege, capabilities, devices, and service control, and its sysctl mutation failed closed.", + } + ) + emit("step-02", "PASS") + + stage = "failure-recovery" + emit("step-03", "START") + missing = ssh( + primary_ssh, + "docker inspect dstack-hardening-definitely-absent", + check=False, + ) + if missing.returncode == 0: + raise AssertionError("invalid container lookup succeeded") + ssh(primary_ssh, "systemctl stop docker.socket docker.service") + unavailable = ssh(primary_ssh, "docker info", check=False, timeout=30) + if unavailable.returncode == 0: + raise AssertionError("Docker dependency interruption was not observed") + ssh(primary_ssh, "systemctl start docker.service docker.socket") + deadline = time.monotonic() + 45 + while time.monotonic() < deadline: + recovered = ssh(primary_ssh, "docker info", check=False, timeout=15) + if recovered.returncode == 0: + break + time.sleep(1) + else: + raise AssertionError("Docker did not recover within 45 seconds") + container = target_container(primary_ssh) + if ssh(primary_ssh, f"sha256sum {quoted_paths}").stdout != policy_hashes: + raise AssertionError( + "measured host policy changed across dependency recovery" + ) + if ( + inspect(primary_ssh, container).get("State", {}).get("Running") + is not False + ): + raise AssertionError("restart:no workload unexpectedly auto-started") + ssh(primary_ssh, f"docker start {shlex.quote(container)}") + if ( + inspect(primary_ssh, container).get("State", {}).get("Running") + is not True + ): + raise AssertionError("explicit workload recovery did not succeed") + observations["recovery"] = { + "invalid_lookup_rejected": True, + "dependency_outage_observed": True, + "docker_recovered": True, + "restart_no_honored": True, + "explicit_workload_recovery": True, + } + steps.append( + { + "id": f"{CASE_ID}-step-03", + "status": "PASS", + "observed": "Invalid lookup failed, a bounded Docker outage was observed, restart:no remained fail-closed, and one explicit workload restart recovered.", + } + ) + emit("step-03", "PASS") + + stage = "isolation" + emit("step-04", "START") + if str(primary.get("instance_id")) == str(adjacent.get("instance_id")): + raise AssertionError("primary and adjacent instance identities matched") + ssh(primary_ssh, f"printf marker > {shlex.quote(marker)}") + ssh(adjacent_ssh, f"test ! -e {shlex.quote(marker)}") + ssh(primary_ssh, "systemctl is-active --quiet docker.service") + observations["isolation"] = { + "adjacent_instance_distinct": True, + "marker_isolated": True, + "service_restart_persisted_health": True, + "sentinel_sha256": marker_hash, + } + steps.append( + { + "id": f"{CASE_ID}-step-04", + "status": "PASS", + "observed": "The adjacent VM retained a distinct identity and could not observe primary transient state after service recovery.", + } + ) + emit("step-04", "PASS") + except Exception as error: + status = "FAIL" + summary = ( + f"guest hardening matrix failed during {stage}: {type(error).__name__}" + ) + steps.append( + { + "id": f"{CASE_ID}-step-{len(steps) + 1:02d}", + "status": "FAIL", + "observed": str(error)[:900], + } + ) + finally: + if primary_ssh: + if marker: + cleanup["marker_removed"] = ( + ssh( + primary_ssh, f"rm -f {shlex.quote(marker)}", check=False + ).returncode + == 0 + ) + ssh( + primary_ssh, "systemctl start docker.service docker.socket", check=False + ) + cleanup["docker_recovered"] = ( + ssh( + primary_ssh, + "systemctl is-active --quiet docker.service", + check=False, + ).returncode + == 0 + ) + + artifact = { + "case_id": CASE_ID, + "status": status, + "environment": "HARDWARE/MKOSI", + "observations": observations, + "cleanup": cleanup, + "sensitive_values_recorded": False, + } + artifact_path = result_dir / "artifacts/guest-hardening.json" + artifact_path.parent.mkdir(parents=True, exist_ok=True) + artifact_path.write_text(json.dumps(artifact, indent=2, sort_keys=True) + "\n") + result = { + "schema_version": "1.0", + "case_id": CASE_ID, + "status": status, + "summary": summary, + "steps": steps, + "evidence": [ + { + "path": "artifacts/guest-hardening.json", + "sha256": hashlib.sha256(artifact_path.read_bytes()).hexdigest(), + } + ], + } + (result_dir / "result.json").write_text(json.dumps(result, indent=2) + "\n") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-006/case.md b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-006/case.md new file mode 100644 index 000000000..0ac9c9c60 --- /dev/null +++ b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-006/case.md @@ -0,0 +1,108 @@ + + + +# TC-GOS-PLATFORM-006: Systemd dependency and failure-action graph + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-platform-006](../../../../catalog/feature-audit.md#req-gos-platform-006) +- Risks: [risk-gos-platform-006](../../../../catalog/feature-audit.md#risk-gos-platform-006) +- Source: `os/common/rootfs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- Treat `dstack-prepare`, Docker, containerd, and `app-compose` as boot/runtime + graph nodes, not independently restartable leaf services. Verify their + ordering, `Requires`/`After`, timeout, and failure-action properties through + `systemctl show`/`systemctl cat`; do not restart them sequentially inside one + SSH command. That can intentionally tear down the guest transport or invoke + the guest reboot failure action and is not a valid service-restart matrix. +- Exercise dynamic restart/failure behavior only on a documented restartable + leaf such as `dstack-guest-agent` or `dstack-gateway-checker`. Run each mutation as a + separate bounded controller command. If a guest transport interruption is + expected, poll VMM state and reconnect through the manifest route before the + next assertion; an SSH reset alone is not a product failure. +- Keep Step 3 failure injection within the systemd behavior under test. Use a + syntactically valid but nonexistent case-scoped unit name, or another invalid + systemd operation that cannot mutate a real unit, and verify that systemd + rejects it without changing the graph. Do not use malformed Guest API input: + RPC parsing is unrelated to this case and is covered by the RPC cases. +- Do not stop or recreate `dstack-guest-agent.socket`. The fixture's TCP + bridge bind-mounts the Unix socket inode, so recreating that socket invalidates + only the observation transport. Interrupt `dstack-guest-agent.service` while + leaving socket activation intact, or temporarily stop/continue its process, + then verify service recovery through the unchanged socket. +- During the process interruption, a filesystem socket existence check or a + repeated `systemctl start` is not the failed operation: both can succeed + while the service process is stopped. Issue one bounded Tappd or DstackGuest + RPC through the unchanged manifest endpoint, require it to time out or fail + without a response, resume the process, and repeat that same RPC successfully. +- Use `values.systemd_graph_peer` as the adjacent lease-owned identity. Record + its identity and running state before mutations and prove both are unchanged + afterward; absence of that declared peer is a fixture defect, not isolation + evidence. + +## Objective + +Verify systemd dependency and failure-action graph with explicit success, boundary, failure, restart, and isolation observations. + +## Preconditions + +1. The target runs in an isolated environment with effective configuration and synchronized evidence capture. +2. Baseline service, file, process, device, listener, and secret-redaction state has been recorded. + +## Test Data + +Use run-scoped identities and sentinel secrets that can be detected by hash without being retained in evidence. + +## Steps + + +### Step 1: Establish the baseline + +Query the effective configuration, service dependencies, listener/device state, and persisted files involved in this behavior. + +**Expected results:** + +- Required dependencies are healthy, ownership and permissions match policy, and no run-scoped object or sentinel is present before the action. + + +### Step 2: Exercise supported and boundary paths + +Start, fail, timeout, and restart prepare, simulator, guest-agent, Docker, app-compose, and WireGuard checker units. + +**Expected results:** + +- Ordering requirements prevent early consumers; optional absence does not reboot-loop; fatal failure follows documented action once with useful console diagnostics. + + +### Step 3: Exercise failure and recovery + +Inject one invalid input and one dependency interruption appropriate to the behavior, restore the dependency, and repeat the valid operation. + +**Expected results:** + +- Failure is bounded, fails closed, produces actionable redacted diagnostics, leaves no partial trusted state, and the repeated valid operation succeeds exactly once after recovery. + + +### Step 4: Verify isolation and persistence + +Restart the affected service or VM when permitted, re-query state, and check adjacent app/instance/node identities. + +**Expected results:** + +- Documented state persists, transient state disappears, adjacent identities are unchanged, and no private key, credential, or plaintext sentinel appears in APIs, metrics, dashboards, journals, or artifacts. + +## Postconditions + +Remove run-scoped state, undo fault injection, and verify services and devices returned to their recorded baseline. diff --git a/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-006/metadata.json b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-006/metadata.json new file mode 100644 index 000000000..90773614e --- /dev/null +++ b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-006/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-platform-006", + "title": "Systemd dependency and failure-action graph", + "priority": "P0", + "requirements": [ + "req-gos-platform-006" + ], + "risks": [ + "risk-gos-platform-006" + ], + "tags": [ + "gos", + "platform-services-and-image-integrity" + ], + "fixture": { + "profile": "guest-readonly", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Systemd dependency and failure-action graph" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 600 + } +} diff --git a/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-006/run.py b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-006/run.py new file mode 100755 index 000000000..87628c7bf --- /dev/null +++ b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-006/run.py @@ -0,0 +1,307 @@ +#!/usr/bin/env python3 +"""Verify systemd dependency graph, leaf interruption, and peer isolation.""" + +from __future__ import annotations + +import hashlib +import json +import os +import shlex +import subprocess +import time +import urllib.error +import urllib.request +from pathlib import Path +from typing import Any + +CASE_ID = "tc-gos-platform-006" +SERVICE = "dstack-guest-agent.service" + + +def ssh( + argv: list[str], command: str, *, check: bool = True +) -> subprocess.CompletedProcess[str]: + """Run one bounded command through a manifest-recorded guest SSH route.""" + result = subprocess.run( + [*argv, command], + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + timeout=90, + check=False, + ) + if check and result.returncode: + raise RuntimeError( + f"guest command failed ({result.returncode}): {command!r}; " + f"stdout={result.stdout[-800:]!r}; stderr={result.stderr[-800:]!r}" + ) + return result + + +def rpc(url: str, timeout: float = 30) -> dict[str, Any]: + """Call the non-secret Tappd Info endpoint.""" + request = urllib.request.Request( + url.replace("{method}", "Info"), + data=b"{}", + headers={"content-type": "application/json"}, + ) + with urllib.request.urlopen(request, timeout=timeout) as response: + value = json.load(response) + if not isinstance(value, dict) or not value.get("app_id"): + raise AssertionError("Tappd.Info response was incomplete") + return value + + +def identity_hash(value: dict[str, Any]) -> str: + """Hash public identity fields without retaining their values.""" + selected = { + name: value.get(name) for name in ("app_id", "instance_id", "device_id") + } + return hashlib.sha256(json.dumps(selected, sort_keys=True).encode()).hexdigest() + + +def wait_rpc(url: str) -> dict[str, Any]: + """Wait for the unchanged socket bridge to serve Tappd.Info again.""" + deadline = time.monotonic() + 45 + last: Exception | None = None + while time.monotonic() < deadline: + try: + return rpc(url, timeout=5) + except (OSError, TimeoutError, urllib.error.URLError) as error: + last = error + time.sleep(1) + raise AssertionError(f"Tappd.Info did not recover: {type(last).__name__}") + + +def emit(step: str, state: str) -> None: + """Emit one live step transition.""" + print(f"STEP {CASE_ID}-{step} {state}", flush=True) + + +def main() -> int: + """Run the static graph and dynamic leaf-service acceptance matrix.""" + result_dir = Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads(Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text()) + values = manifest.get("values", {}) + peer = values.get("systemd_graph_peer") if isinstance(values, dict) else None + ssh_argv = values.get("ssh_argv") if isinstance(values, dict) else None + status = "PASS" + summary = "systemd dependency and failure-action graph matrix passed" + observations: dict[str, Any] = {} + steps: list[dict[str, str]] = [] + stage = "fixture" + frozen = False + + try: + if not ( + isinstance(ssh_argv, list) + and values.get("destructive_actions_allowed") is True + and isinstance(peer, dict) + and isinstance(peer.get("ssh_argv"), list) + and peer.get("destructive_actions_allowed") is True + ): + status = "BLOCKED" + summary = "missing capability: systemd-graph-peer-lifecycle" + observations["missing_capability"] = "systemd-graph-peer-lifecycle" + else: + primary_url = str(values["services"]["Tappd"]["url"]) + peer_url = str(peer["tappd_url"]) + peer_ssh = [str(item) for item in peer["ssh_argv"]] + + stage = "baseline-graph" + emit("step-01", "START") + graph = ssh( + ssh_argv, + "systemctl show dstack-prepare.service dstack-guest-agent.service " + "dstack-guest-agent.socket docker.service containerd.service " + "app-compose.service dstack-gateway-checker.service " + "--property=Id,LoadState,ActiveState,Requires,Wants,After,Before," + "OnFailure,FailureAction,Restart,WatchdogUSec,TimeoutStartUSec --no-pager", + ).stdout + required_tokens = ( + "Id=dstack-prepare.service", + "FailureAction=reboot", + "Id=dstack-guest-agent.service", + "dstack-guest-agent.socket", + "Restart=always", + "Id=app-compose.service", + "docker.service", + "containerd.service", + "Id=dstack-gateway-checker.service", + ) + missing = [token for token in required_tokens if token not in graph] + if missing: + raise AssertionError( + f"runtime graph omitted declared tokens: {missing}" + ) + primary_before = wait_rpc(primary_url) + peer_before = wait_rpc(peer_url) + peer_state_before = ssh( + peer_ssh, "systemctl is-system-running --wait || true" + ).stdout.strip() + primary_hash = identity_hash(primary_before) + peer_hash = identity_hash(peer_before) + if primary_hash == peer_hash: + raise AssertionError( + "primary and adjacent identities were not distinct" + ) + observations["baseline"] = { + "declared_graph_tokens_present": True, + "primary_peer_distinct": True, + "peer_system_state": peer_state_before, + "graph_sha256": hashlib.sha256(graph.encode()).hexdigest(), + } + steps.append( + { + "id": f"{CASE_ID}-step-01", + "status": "PASS", + "observed": "Runtime unit properties contained the checked-in prepare failure action, guest-agent socket/watchdog/restart edges, app-compose Docker/containerd ordering, and gateway-checker node; primary and peer identities were distinct and healthy.", + } + ) + emit("step-01", "PASS") + + stage = "leaf-interruption" + emit("step-02", "START") + ssh( + ssh_argv, + f"systemctl kill --kill-who=main --signal=STOP {shlex.quote(SERVICE)}", + ) + frozen = True + interrupted = False + try: + rpc(primary_url, timeout=5) + except (OSError, TimeoutError, urllib.error.URLError): + interrupted = True + if not interrupted: + raise AssertionError( + "Tappd.Info responded while guest-agent main process was stopped" + ) + ssh( + ssh_argv, + f"systemctl kill --kill-who=main --signal=CONT {shlex.quote(SERVICE)}", + ) + frozen = False + resumed = wait_rpc(primary_url) + if identity_hash(resumed) != primary_hash: + raise AssertionError( + "primary identity changed after STOP/CONT recovery" + ) + observations["interruption"] = { + "rpc_failed_while_stopped": True, + "same_rpc_recovered_after_continue": True, + "socket_unit_left_unchanged": True, + } + steps.append( + { + "id": f"{CASE_ID}-step-02", + "status": "PASS", + "observed": "Stopping only the restartable guest-agent process made the unchanged Tappd route fail without a response; continuing the process restored the same RPC and identity without recreating the socket unit.", + } + ) + emit("step-02", "PASS") + + stage = "invalid-unit-recovery" + emit("step-03", "START") + invalid_name = f"dstack-case-{manifest['lease_id'][-12:]}-absent.service" + invalid = ssh( + ssh_argv, + f"systemctl start {shlex.quote(invalid_name)}", + check=False, + ) + if invalid.returncode == 0: + raise AssertionError("nonexistent case-scoped unit was accepted") + graph_after_invalid = ssh( + ssh_argv, + "systemctl show dstack-prepare.service dstack-guest-agent.service " + "app-compose.service --property=Id,Requires,Wants,After,Before," + "OnFailure,FailureAction,Restart,WatchdogUSec --no-pager", + ).stdout + if "Id=dstack-prepare.service" not in graph_after_invalid: + raise AssertionError("graph became unavailable after invalid operation") + ssh(ssh_argv, f"systemctl restart {shlex.quote(SERVICE)}") + restarted = wait_rpc(primary_url) + if identity_hash(restarted) != primary_hash: + raise AssertionError("primary identity changed after service restart") + observations["failure_recovery"] = { + "invalid_unit_rejected": True, + "graph_remained_queryable": True, + "leaf_restart_recovered": True, + } + steps.append( + { + "id": f"{CASE_ID}-step-03", + "status": "PASS", + "observed": "Systemd rejected a syntactically valid nonexistent case-scoped unit, the dependency graph remained queryable, and the documented leaf service restarted with the same identity.", + } + ) + emit("step-03", "PASS") + + stage = "peer-isolation" + emit("step-04", "START") + peer_after = rpc(peer_url) + peer_state_after = ssh( + peer_ssh, "systemctl is-system-running --wait || true" + ).stdout.strip() + if identity_hash(peer_after) != peer_hash: + raise AssertionError("adjacent peer identity changed") + if peer_state_after not in ("running", "degraded"): + raise AssertionError( + f"adjacent peer became unhealthy: {peer_state_after}" + ) + observations["isolation"] = { + "peer_identity_unchanged": True, + "peer_system_state": peer_state_after, + "primary_health_restored": bool(rpc(primary_url).get("app_id")), + } + steps.append( + { + "id": f"{CASE_ID}-step-04", + "status": "PASS", + "observed": "The adjacent lease-owned peer retained its identity and healthy system state throughout primary mutations, and primary Tappd health was restored.", + } + ) + emit("step-04", "PASS") + except Exception as error: + status = "FAIL" + summary = f"{stage}: {type(error).__name__}: {error}" + observations["failed_stage"] = stage + observations["error_type"] = type(error).__name__ + observations["error"] = str(error) + finally: + if isinstance(ssh_argv, list): + if frozen: + ssh( + ssh_argv, + f"systemctl kill --kill-who=main --signal=CONT {shlex.quote(SERVICE)}", + check=False, + ) + ssh(ssh_argv, f"systemctl start {shlex.quote(SERVICE)}", check=False) + + artifact = { + "case_id": CASE_ID, + "status": status, + "environment": "HARDWARE", + "observations": observations, + } + artifact_path = result_dir / "artifacts/systemd-graph-lifecycle.json" + artifact_path.parent.mkdir(parents=True, exist_ok=True) + artifact_path.write_text(json.dumps(artifact, indent=2) + "\n") + result = { + "schema_version": "1.0", + "case_id": CASE_ID, + "status": status, + "summary": summary, + "steps": steps, + "evidence": [ + { + "path": "artifacts/systemd-graph-lifecycle.json", + "sha256": hashlib.sha256(artifact_path.read_bytes()).hexdigest(), + } + ], + } + (result_dir / "result.json").write_text(json.dumps(result, indent=2) + "\n") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-007/case.md b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-007/case.md new file mode 100644 index 000000000..bdbc67ec3 --- /dev/null +++ b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-007/case.md @@ -0,0 +1,79 @@ + + + +# TC-GOS-PLATFORM-007: Journal persistence rotation and redaction + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: lease-owned mkosi guest +- Automation: Yes +- Requirements: [req-gos-platform-007](../../../../catalog/feature-audit.md#req-gos-platform-007) +- Risks: [risk-gos-platform-007](../../../../catalog/feature-audit.md#risk-gos-platform-007) +- Source: `os/common/rootfs/journald.conf` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify journal persistence rotation and redaction with explicit success, boundary, failure, restart, and isolation observations. + +## Preconditions + +1. The target runs in an isolated environment with effective configuration and synchronized evidence capture. +2. Baseline service, file, process, device, listener, and secret-redaction state has been recorded. + +## Test Data + +Use run-scoped identities and sentinel secrets that can be detected by hash without being retained in evidence. + +## Steps + + +### Step 1: Establish the baseline + +Query the effective configuration, service dependencies, listener/device state, and persisted files involved in this behavior. + +**Expected results:** + +- Required dependencies are healthy, ownership and permissions match policy, and no run-scoped object or sentinel is present before the action. + + +### Step 2: Exercise supported and boundary paths + +Generate boot, application, RPC, Docker-failure, and service-failure diagnostics, exercise bounded size rotation, and restart journald. + +**Expected results:** + +- Required logs remain queryable within retention, rotation is bounded, and unprivileged identities cannot read journal files. +- Journald stores producer payloads verbatim and is not a secret scrubber; producers must emit only hashes and explicit `[REDACTED]` markers, and the plaintext sentinel must never enter the journal or evidence. + + +### Step 3: Exercise failure and recovery + +Inject one invalid input and one dependency interruption appropriate to the behavior, restore the dependency, and repeat the valid operation. + +**Expected results:** + +- Failure is bounded, fails closed, produces actionable redacted diagnostics, leaves no partial trusted state, and the repeated valid operation succeeds exactly once after recovery. + + +### Step 4: Verify isolation and persistence + +Restart the affected service, re-query state, and check adjacent app/instance/node identities. + +**Expected results:** + +- Documented state persists, transient state disappears, adjacent identities are unchanged, and no private key, credential, or plaintext sentinel appears in APIs, metrics, dashboards, journals, or artifacts. + +## Postconditions + +Remove run-scoped state, undo fault injection, and verify services and devices returned to their recorded baseline. diff --git a/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-007/metadata.json b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-007/metadata.json new file mode 100644 index 000000000..9ca80ea70 --- /dev/null +++ b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-007/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-platform-007", + "title": "Journal persistence rotation and redaction", + "priority": "P1", + "requirements": [ + "req-gos-platform-007" + ], + "risks": [ + "risk-gos-platform-007" + ], + "tags": [ + "gos", + "platform-services-and-image-integrity" + ], + "fixture": { + "profile": "guest-readonly", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Journal persistence rotation and redaction" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-007/run.py b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-007/run.py new file mode 100755 index 000000000..e53832b01 --- /dev/null +++ b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-007/run.py @@ -0,0 +1,173 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise bounded journald retention, rotation, producer redaction, and recovery.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import secrets +import subprocess +import time + +CASE_ID = "tc-gos-platform-007" + + +def run( + argv: list[str], *, data: bytes | None = None, timeout: int = 60 +) -> subprocess.CompletedProcess[bytes]: + """Run one bounded host or guest command.""" + return subprocess.run( + argv, input=data, capture_output=True, timeout=timeout, check=False + ) + + +def write(path: pathlib.Path, value: object) -> None: + """Write deterministic JSON evidence.""" + path.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n") + + +def vm_ids(blob: bytes) -> list[str]: + """Return stable VM identities from one inventory response.""" + return sorted(str(x.get("id")) for x in json.loads(blob) if isinstance(x, dict)) + + +def main() -> int: + """Execute the lease-owned journald lifecycle.""" + result = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + artifacts = result / "artifacts" + artifacts.mkdir(parents=True, exist_ok=True) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + values = manifest.get("values") or {} + ssh = [str(x) for x in values.get("ssh_argv") or []] + status = "FAIL" + summary = "journald lifecycle did not execute" + started = time.monotonic() + evidence: dict[str, object] = { + "candidate_commit": runtime.get("candidate_commit"), + "guest_image": values.get("image"), + } + token = "dstack-secret-" + secrets.token_hex(16) + token_hash = hashlib.sha256(token.encode()).hexdigest() + marker = secrets.token_hex(8) + try: + if not ssh or values.get("destructive_actions_allowed") is not True: + raise RuntimeError("fixture omitted lease-owned guest SSH") + store = pathlib.Path( + str((runtime.get("environment") or {}).get("DSTACK_TEST_IMAGE_STORE", "")) + ) + metadata = json.loads( + (store / str(values["image"]) / "metadata.json").read_text() + ) + if metadata.get("builder") != "mkosi": + raise RuntimeError("fixture did not boot a mkosi image") + evidence["mkosi"] = { + k: metadata.get(k) for k in ("builder", "is_dev", "git_revision") + } + script = ( + pathlib.Path(str(runtime["repository"])) + / "test-suites/shared/automation/journal-lifecycle.sh" + ) + installed = run( + [*ssh, "install -m 0755 /dev/stdin /run/dstack-test-journal-case"], + data=script.read_bytes(), + timeout=60, + ) + if installed.returncode: + raise RuntimeError("guest script installation failed") + inventory = [str(x) for x in values.get("list_vms_argv") or []] + before = run(inventory, timeout=30) + if before.returncode: + raise RuntimeError("baseline VM inventory query failed") + completed = run( + [*ssh, "/run/dstack-test-journal-case", token, token_hash, marker], + timeout=300, + ) + log = completed.stdout + completed.stderr + (artifacts / "journal-lifecycle.log").write_bytes(log) + if completed.returncode: + raise RuntimeError( + f"guest lifecycle rc={completed.returncode}: {log.decode(errors='replace')[-1600:]}" + ) + rows = [ + line + for line in completed.stdout.decode().splitlines() + if line.startswith("{") + ] + matrix = json.loads(rows[-1]) + after = run(inventory, timeout=30) + if after.returncode: + raise RuntimeError("recovery VM inventory query failed") + matrix["inventory_stable"] = vm_ids(before.stdout) == vm_ids(after.stdout) + required = ( + "baseline", + "rotation", + "redacted", + "unprivileged_denied", + "invalid_closed", + "outage", + "recovered", + "cleanup", + "inventory_stable", + ) + if any(matrix.get(k) is not True for k in required): + raise RuntimeError(f"unexpected journal matrix: {matrix}") + evidence["matrix"] = matrix + evidence["sentinel_sha256"] = token_hash + status = "PASS" + summary = "Journald policy, bounded rotation, producer-side redaction, unprivileged isolation, invalid input, outage, recovery, cleanup, and adjacent-VM isolation passed." + except Exception as error: + summary = f"{type(error).__name__}: {error}" + finally: + if ssh: + evidence["cleanup_returncode"] = run( + [ + *ssh, + "rm -f /run/systemd/journald.conf.d/99-dstack-test.conf /run/dstack-test-journal-case; systemctl restart systemd-journald.service; rm -rf /run/dstack-test-journal", + ], + timeout=60, + ).returncode + evidence["duration_seconds"] = round(time.monotonic() - started, 3) + path = artifacts / "journal-lifecycle.json" + write(path, evidence) + artifact = { + "path": "artifacts/journal-lifecycle.json", + "step_id": f"{CASE_ID}-step-01", + "name": "Journald lifecycle", + "description": "Redacted mkosi provenance, bounded retention, rotation, producer redaction, isolation, failure, recovery, cleanup, and adjacent-VM evidence.", + } + write(artifacts / "manifest.json", {"artifacts": [artifact]}) + write( + result / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + {"id": f"{CASE_ID}-step-{n:02d}", "status": status, "observed": summary} + for n in range(1, 5) + ], + "artifacts": [artifact], + "evidence": [ + { + "path": artifact["path"], + "sha256": hashlib.sha256(path.read_bytes()).hexdigest(), + } + ], + "remarks": "Journald retains producer payloads verbatim; the tested security contract requires producers to emit only a sentinel hash and [REDACTED], never the plaintext token.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-008/case.md b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-008/case.md new file mode 100644 index 000000000..f2262e42e --- /dev/null +++ b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-008/case.md @@ -0,0 +1,99 @@ + + + +# TC-GOS-PLATFORM-008: Docker daemon and container privilege boundary + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-platform-008](../../../../catalog/feature-audit.md#req-gos-platform-008) +- Risks: [risk-gos-platform-008](../../../../catalog/feature-audit.md#risk-gos-platform-008) +- Source: `os/common/rootfs/docker.service.d` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- Docker containers are not a security boundary from the application that owns + the CVM. Compose is intentionally allowed to request privileged mode, host + namespaces, devices, and guest-local mounts including the dstack sockets; + those declarations are part of the measured app compose and authorization + identity. Do not report access to the owning guest's sockets or Docker + metadata as a failure. +- The enforced boundary is the CVM/VMM boundary. Compare a normal app and a + separately measured privileged app: their compose hashes/app identities must + differ, requested privileges must not appear in the normal app, and neither + app may access the physical VMM host or the peer CVM's filesystem, sockets, + containers, or identity. Test resource limits only when declared in that + app's measured compose. +- The case manifest must provide `values.docker_boundary.normal` and + `values.docker_boundary.privileged`, each with its own lease-owned VM, SSH + command, instance identity, and compose hash. The normal compose contains a + constrained `boundary-target`; the privileged compose declares its elevated + settings. If these two measured fixtures are absent, do not substitute two + ad-hoc `docker run` commands inside one VM because that cannot prove compose + identity binding or cross-CVM isolation. + +## Objective + +Verify docker daemon and container privilege boundary with explicit success, boundary, failure, restart, and isolation observations. + +## Preconditions + +1. The target runs in an isolated environment with effective configuration and synchronized evidence capture. +2. Baseline service, file, process, device, listener, and secret-redaction state has been recorded. + +## Test Data + +Use run-scoped identities and sentinel secrets that can be detected by hash without being retained in evidence. + +## Steps + + +### Step 1: Establish the baseline + +Query the effective configuration, service dependencies, listener/device state, and persisted files involved in this behavior. + +**Expected results:** + +- Required dependencies are healthy, ownership and permissions match policy, and no run-scoped object or sentinel is present before the action. + + +### Step 2: Exercise supported and boundary paths + +Launch separately measured normal and privileged compose applications requesting host mounts, devices, privileged mode, namespaces, capabilities, and resource limits. + +**Expected results:** + +- Declared privileges and limits are honored inside the owning CVM, the normal + app does not gain undeclared privileges, compose/app identity binds the + difference, and neither app reaches the VMM host or peer CVM state. + + +### Step 3: Exercise failure and recovery + +Inject one invalid input and one dependency interruption appropriate to the behavior, restore the dependency, and repeat the valid operation. + +**Expected results:** + +- Failure is bounded, fails closed, produces actionable redacted diagnostics, leaves no partial trusted state, and the repeated valid operation succeeds exactly once after recovery. + + +### Step 4: Verify isolation and persistence + +Restart the affected service or VM when permitted, re-query state, and check adjacent app/instance/node identities. + +**Expected results:** + +- Documented state persists, transient state disappears, adjacent identities are unchanged, and no private key, credential, or plaintext sentinel appears in APIs, metrics, dashboards, journals, or artifacts. + +## Postconditions + +Remove run-scoped state, undo fault injection, and verify services and devices returned to their recorded baseline. diff --git a/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-008/metadata.json b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-008/metadata.json new file mode 100644 index 000000000..617428172 --- /dev/null +++ b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-008/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-platform-008", + "title": "Docker daemon and container privilege boundary", + "priority": "P0", + "requirements": [ + "req-gos-platform-008" + ], + "risks": [ + "risk-gos-platform-008" + ], + "tags": [ + "gos", + "platform-services-and-image-integrity" + ], + "fixture": { + "destructive_scope": "lease-only", + "hardware_required": false, + "profile": "multi-identity", + "simulation_allowed": false, + "versions": { + "gateway": "candidate", + "guest": "candidate", + "kms": "candidate", + "verifier": "candidate", + "vmm": "candidate" + } + }, + "actions_under_test": [ + "Docker daemon and container privilege boundary" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 600 + } +} diff --git a/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-008/run.py b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-008/run.py new file mode 100755 index 000000000..6798b1cbe --- /dev/null +++ b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-008/run.py @@ -0,0 +1,361 @@ +#!/usr/bin/env python3 +"""Verify measured normal/privileged Docker policy and cross-CVM isolation.""" + +from __future__ import annotations + +import hashlib +import json +import os +import shlex +import subprocess +import time +import urllib.error +import urllib.request +from pathlib import Path +from typing import Any + +CASE_ID = "tc-gos-platform-008" + + +def ssh( + argv: list[str], command: str, *, check: bool = True +) -> subprocess.CompletedProcess[str]: + """Run one bounded command through a manifest-recorded guest SSH route.""" + result = subprocess.run( + [*argv, command], + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + timeout=90, + check=False, + ) + if check and result.returncode: + raise RuntimeError( + f"guest command failed ({result.returncode}): {command!r}; " + f"stdout={result.stdout[-800:]!r}; stderr={result.stderr[-800:]!r}" + ) + return result + + +def rpc(url: str) -> dict[str, Any]: + """Call one non-secret Tappd.Info endpoint with bounded startup retries.""" + deadline = time.monotonic() + 45 + last: Exception | None = None + while time.monotonic() < deadline: + request = urllib.request.Request( + url.replace("{method}", "Info"), + data=b"{}", + headers={"content-type": "application/json"}, + ) + try: + with urllib.request.urlopen(request, timeout=10) as response: + value = json.load(response) + if isinstance(value, dict) and value.get("app_id"): + return value + raise AssertionError("Tappd.Info response was incomplete") + except (OSError, TimeoutError, urllib.error.URLError) as error: + last = error + time.sleep(1) + raise AssertionError(f"Tappd.Info did not become ready: {type(last).__name__}") + + +def identity_hash(value: dict[str, Any]) -> str: + """Hash public identity fields without retaining their values.""" + selected = { + name: value.get(name) for name in ("app_id", "instance_id", "device_id") + } + return hashlib.sha256(json.dumps(selected, sort_keys=True).encode()).hexdigest() + + +def target_container(argv: list[str]) -> str: + """Resolve the unique compose boundary-target container.""" + output = ssh( + argv, + "docker ps -aq --filter label=com.docker.compose.service=boundary-target", + ).stdout.split() + if len(output) != 1: + raise AssertionError(f"expected one boundary-target, found {len(output)}") + return output[0] + + +def inspect(argv: list[str], container: str) -> dict[str, Any]: + """Inspect one case-owned container.""" + value = json.loads(ssh(argv, f"docker inspect {shlex.quote(container)}").stdout) + if not isinstance(value, list) or len(value) != 1 or not isinstance(value[0], dict): + raise AssertionError("docker inspect returned an unexpected shape") + return value[0] + + +def emit(step: str, state: str) -> None: + """Emit one live step transition.""" + print(f"STEP {CASE_ID}-{step} {state}", flush=True) + + +def main() -> int: + """Run the measured Docker privilege and isolation acceptance matrix.""" + result_dir = Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads(Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text()) + values = manifest.get("values", {}) + boundary = values.get("docker_boundary") if isinstance(values, dict) else None + status = "PASS" + summary = "Docker daemon and measured container privilege boundary passed" + observations: dict[str, Any] = {} + steps: list[dict[str, str]] = [] + stage = "fixture" + memory_blocked = False + normal_marker = f"/tmp/dstack-boundary-normal-{manifest.get('lease_id', '')[-10:]}" + privileged_marker = ( + f"/tmp/dstack-boundary-priv-{manifest.get('lease_id', '')[-10:]}" + ) + + try: + if not ( + isinstance(boundary, dict) + and isinstance(boundary.get("normal"), dict) + and isinstance(boundary.get("privileged"), dict) + ): + status = "BLOCKED" + summary = "missing capability: measured-docker-boundary-pair" + observations["missing_capability"] = "measured-docker-boundary-pair" + else: + normal = boundary["normal"] + privileged = boundary["privileged"] + normal_ssh = [str(item) for item in normal["ssh_argv"]] + privileged_ssh = [str(item) for item in privileged["ssh_argv"]] + normal_url = str(values["services"]["Tappd"]["url"]) + privileged_url = str(privileged["tappd_url"]) + + stage = "baseline" + emit("step-01", "START") + if normal.get("compose_sha256") == privileged.get("compose_sha256"): + raise AssertionError("normal and privileged compose hashes matched") + normal_identity = rpc(normal_url) + privileged_identity = rpc(privileged_url) + normal_identity_hash = identity_hash(normal_identity) + privileged_identity_hash = identity_hash(privileged_identity) + if normal_identity_hash == privileged_identity_hash: + raise AssertionError("normal and privileged app identities matched") + normal_id = target_container(normal_ssh) + privileged_id = target_container(privileged_ssh) + normal_inspect = inspect(normal_ssh, normal_id) + privileged_inspect = inspect(privileged_ssh, privileged_id) + observations["baseline"] = { + "compose_hashes_distinct": True, + "app_identities_distinct": True, + "normal_container_present": True, + "privileged_container_present": True, + } + steps.append( + { + "id": f"{CASE_ID}-step-01", + "status": "PASS", + "observed": "The fixture exposed one normal and one privileged measured application with distinct compose hashes, app identities, instances, SSH routes, and boundary-target containers.", + } + ) + emit("step-01", "PASS") + + stage = "policy-boundary" + emit("step-02", "START") + normal_host = normal_inspect.get("HostConfig", {}) + privileged_host = privileged_inspect.get("HostConfig", {}) + normal_mounts = normal_inspect.get("Mounts", []) + privileged_mounts = privileged_inspect.get("Mounts", []) + normal_security = [ + str(x).lower() for x in normal_host.get("SecurityOpt") or [] + ] + normal_cap_drop = [str(x).upper() for x in normal_host.get("CapDrop") or []] + if normal_host.get("Privileged") is not False: + raise AssertionError("normal target was privileged") + if normal_host.get("NetworkMode") != "none" or normal_host.get( + "PidMode" + ) not in ("", None): + raise AssertionError( + "normal target gained host network or PID namespace" + ) + if "ALL" not in normal_cap_drop or not any( + "no-new-privileges" in x for x in normal_security + ): + raise AssertionError( + "normal capability/no-new-privileges policy was absent" + ) + controllers = ssh( + normal_ssh, + "cat /sys/fs/cgroup/cgroup.controllers 2>/dev/null || true", + ).stdout.split() + memory_blocked = "memory" not in controllers + if ( + not memory_blocked + and int(normal_host.get("Memory") or 0) != 128 * 1024 * 1024 + ): + raise AssertionError( + f"normal memory controller is available but the measured limit " + f"was not applied: Memory={normal_host.get('Memory')!r}" + ) + if int(normal_host.get("PidsLimit") or 0) != 64: + raise AssertionError("normal PID limit differed from measured compose") + if normal_mounts: + raise AssertionError("normal target unexpectedly received mounts") + if privileged_host.get("Privileged") is not True: + raise AssertionError( + "privileged target did not receive its measured privilege" + ) + if ( + privileged_host.get("NetworkMode") != "host" + or privileged_host.get("PidMode") != "host" + ): + raise AssertionError( + "privileged target lacked measured host namespaces" + ) + mount_by_dest = { + str(x.get("Destination")): x + for x in privileged_mounts + if isinstance(x, dict) + } + root_mount = mount_by_dest.get("/guest-host") + socket_mount = mount_by_dest.get("/run/dstack.sock") + if not root_mount or root_mount.get("RW") is not False or not socket_mount: + raise AssertionError( + "privileged guest-root/socket mounts differed from compose" + ) + ssh(normal_ssh, f"printf normal > {shlex.quote(normal_marker)}") + ssh(privileged_ssh, f"printf privileged > {shlex.quote(privileged_marker)}") + ssh( + normal_ssh, + f"docker exec {shlex.quote(normal_id)} sh -c 'test ! -e /guest-host && test ! -e /run/dstack.sock'", + ) + ssh( + privileged_ssh, + f"docker exec {shlex.quote(privileged_id)} test -f /guest-host{shlex.quote(privileged_marker)}", + ) + ssh(privileged_ssh, f"test ! -e {shlex.quote(normal_marker)}") + ssh(normal_ssh, f"test ! -e {shlex.quote(privileged_marker)}") + observations["policy"] = { + "normal_privilege_absent": True, + "normal_pids_limit_exact": True, + "normal_memory_limit_exact": not memory_blocked, + "missing_capability": ( + "candidate-guest-memory-cgroup" if memory_blocked else None + ), + "privileged_declarations_honored": True, + "privileged_root_is_guest_readonly": True, + "cross_cvm_markers_isolated": True, + "physical_host_access_allowed": boundary.get( + "physical_host_access_allowed" + ), + } + steps.append( + { + "id": f"{CASE_ID}-step-02", + "status": "BLOCKED" if memory_blocked else "PASS", + "observed": ( + "Docker honored the measured normal restrictions and privileged declarations except that the candidate guest kernel lacks the memory cgroup controller; PIDs, namespaces, capabilities, mounts, sockets, identities, and cross-CVM isolation passed." + if memory_blocked + else "Docker honored all measured normal restrictions and privileged declarations; the privileged root mount was its own CVM read-only root, while normal and peer CVM state remained isolated." + ), + } + ) + emit("step-02", "PASS") + + stage = "failure-recovery" + emit("step-03", "START") + invalid = ssh( + normal_ssh, "docker inspect dstack-case-definitely-absent", check=False + ) + if invalid.returncode == 0: + raise AssertionError("invalid container lookup succeeded") + ssh(normal_ssh, f"docker stop -t 10 {shlex.quote(normal_id)}") + stopped = inspect(normal_ssh, normal_id) + if stopped.get("State", {}).get("Running") is not False: + raise AssertionError("normal target did not stop") + ssh(normal_ssh, f"docker start {shlex.quote(normal_id)}") + recovered = inspect(normal_ssh, normal_id) + if recovered.get("State", {}).get("Running") is not True: + raise AssertionError("normal target did not recover") + observations["failure_recovery"] = { + "invalid_lookup_rejected": True, + "normal_stop_observed": True, + "normal_start_recovered": True, + } + steps.append( + { + "id": f"{CASE_ID}-step-03", + "status": "PASS", + "observed": "A nonexistent container lookup failed closed; the case-owned normal target stopped, exposed the stopped state, restarted once, and returned to running.", + } + ) + emit("step-03", "PASS") + + stage = "final-isolation" + emit("step-04", "START") + if identity_hash(rpc(normal_url)) != normal_identity_hash: + raise AssertionError("normal identity changed") + if identity_hash(rpc(privileged_url)) != privileged_identity_hash: + raise AssertionError("privileged identity changed") + ssh(normal_ssh, "systemctl is-active --quiet docker.service") + ssh(privileged_ssh, "systemctl is-active --quiet docker.service") + if target_container(normal_ssh) == target_container(privileged_ssh): + raise AssertionError( + "cross-CVM container identifiers unexpectedly matched" + ) + observations["final"] = { + "identities_unchanged": True, + "docker_services_active": True, + "container_ids_distinct": True, + } + steps.append( + { + "id": f"{CASE_ID}-step-04", + "status": "PASS", + "observed": "Both measured app identities and Docker services remained stable after recovery, and the two CVMs retained distinct boundary-target container identities.", + } + ) + emit("step-04", "PASS") + if memory_blocked: + status = "BLOCKED" + summary = "missing capability: candidate-guest-memory-cgroup" + except Exception as error: + status = "FAIL" + summary = f"{stage}: {type(error).__name__}: {error}" + observations["failed_stage"] = stage + observations["error_type"] = type(error).__name__ + observations["error"] = str(error) + finally: + if isinstance(boundary, dict): + for role in ("normal", "privileged"): + item = boundary.get(role) + if isinstance(item, dict) and isinstance(item.get("ssh_argv"), list): + argv = [str(x) for x in item["ssh_argv"]] + ssh( + argv, + f"rm -f {shlex.quote(normal_marker)} {shlex.quote(privileged_marker)}", + check=False, + ) + + artifact = { + "case_id": CASE_ID, + "status": status, + "environment": "HARDWARE", + "observations": observations, + } + artifact_path = result_dir / "artifacts/docker-boundary.json" + artifact_path.parent.mkdir(parents=True, exist_ok=True) + artifact_path.write_text(json.dumps(artifact, indent=2) + "\n") + result = { + "schema_version": "1.0", + "case_id": CASE_ID, + "status": status, + "summary": summary, + "steps": steps, + "evidence": [ + { + "path": "artifacts/docker-boundary.json", + "sha256": hashlib.sha256(artifact_path.read_bytes()).hexdigest(), + } + ], + } + (result_dir / "result.json").write_text(json.dumps(result, indent=2) + "\n") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-009/case.md b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-009/case.md new file mode 100644 index 000000000..7227a1430 --- /dev/null +++ b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-009/case.md @@ -0,0 +1,78 @@ + + + +# TC-GOS-PLATFORM-009: NVIDIA device initialization and attestation failure + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-gos-platform-009](../../../../catalog/feature-audit.md#req-gos-platform-009) +- Risks: [risk-gos-platform-009](../../../../catalog/feature-audit.md#risk-gos-platform-009) +- Source: `os/yocto/layers/meta-nvidia` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify nvidia device initialization and attestation failure with explicit success, boundary, failure, restart, and isolation observations. + +## Preconditions + +1. The target runs in an isolated environment with effective configuration and synchronized evidence capture. +2. Baseline service, file, process, device, listener, and secret-redaction state has been recorded. + +## Test Data + +Use run-scoped identities and sentinel secrets that can be detected by hash without being retained in evidence. + +## Steps + + +### Step 1: Establish the baseline + +Query the effective configuration, service dependencies, listener/device state, and persisted files involved in this behavior. + +**Expected results:** + +- Required dependencies are healthy, ownership and permissions match policy, and no run-scoped object or sentinel is present before the action. + + +### Step 2: Exercise supported and boundary paths + +Boot supported GPU assignment, missing driver/device, altered attestation output, and partial multi-GPU failure. + +**Expected results:** + +- Only assigned devices appear, driver and evidence match inventory, and failed attestation is explicit without exposing device to an untrusted workload. + + +### Step 3: Exercise failure and recovery + +Inject one invalid input and one dependency interruption appropriate to the behavior, restore the dependency, and repeat the valid operation. + +**Expected results:** + +- Failure is bounded, fails closed, produces actionable redacted diagnostics, leaves no partial trusted state, and the repeated valid operation succeeds exactly once after recovery. + + +### Step 4: Verify isolation and persistence + +Restart the affected service or VM when permitted, re-query state, and check adjacent app/instance/node identities. + +**Expected results:** + +- Documented state persists, transient state disappears, adjacent identities are unchanged, and no private key, credential, or plaintext sentinel appears in APIs, metrics, dashboards, journals, or artifacts. + +## Postconditions + +Remove run-scoped state, undo fault injection, and verify services and devices returned to their recorded baseline. diff --git a/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-009/metadata.json b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-009/metadata.json new file mode 100644 index 000000000..bb67dc409 --- /dev/null +++ b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-009/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-platform-009", + "title": "NVIDIA device initialization and attestation failure", + "priority": "P0", + "requirements": [ + "req-gos-platform-009" + ], + "risks": [ + "risk-gos-platform-009" + ], + "tags": [ + "gos", + "platform-services-and-image-integrity" + ], + "fixture": { + "profile": "gpu-policy", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": true, + "simulation_allowed": false + }, + "actions_under_test": [ + "NVIDIA device initialization and attestation failure" + ], + "execution": { + "entrypoint": "shared/automation/capability-probe-case.py", + "args": [], + "timeout_seconds": 60 + } +} diff --git a/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-010/case.md b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-010/case.md new file mode 100644 index 000000000..5ff504f42 --- /dev/null +++ b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-010/case.md @@ -0,0 +1,98 @@ + + + +# TC-GOS-PLATFORM-010: Guest configuration backward and forward compatibility + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: TDX +- Automation: Yes +- Requirements: [req-gos-platform-010](../../../../catalog/feature-audit.md#req-gos-platform-010) +- Risks: [risk-gos-platform-010](../../../../catalog/feature-audit.md#risk-gos-platform-010) +- Source: `dstack/dstack-types/src` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- Use the case-owned physical TDX VMM for all four rows and select it explicitly + with `--tee` for each deployment. The historical guests require the hardware + `tdx_guest` device, while the current guest detects the equivalent configfs + provider; no-TEE simulation is therefore not a shared compatibility surface. + Physical collateral must use the product PCCS path, not simulator collateral. +- The compatibility rows intentionally exercise the current VMM, KMS, and + gateway with official guest images `dstack-dev-0.5.4`, `dstack-0.5.8`, + `dstack-0.5.11`, and `dstack-0.6.0`. Generate one current-schema compose with + `vmm-cli.py compose --kms --gateway --key-provider kms --public-logs + --public-sysinfo --event-log-version 2`; do not disable KMS/gateway or select + `key_provider=none`, because that removes the dependencies this compatibility + case is required to test and leaves identity-bearing guests in a prepare + restart loop. Allocate 2 vCPU, 4096 MiB, and 20 GiB per row. A row that exits + before `boot_progress=done` is an immediate diagnostic condition; capture its + bounded serial/VMM log instead of waiting out the entire readiness timeout. +- Deploy the four rows concurrently when capacity is available, register every + returned VM ID immediately, and poll them together. Use a 10-minute shared + deadline, not a separate deadline per row. Perform graceful stop only after + the guest reports `boot_progress=done`; a guest-agent connection error while + the guest is still booting is not evidence about graceful-stop compatibility. + +## Objective + +Verify guest configuration backward and forward compatibility with explicit success, boundary, failure, restart, and isolation observations. + +## Preconditions + +1. The target runs in an isolated environment with effective configuration and synchronized evidence capture. +2. Baseline service, file, process, device, listener, and secret-redaction state has been recorded. + +## Test Data + +Use run-scoped identities and sentinel secrets that can be detected by hash without being retained in evidence. + +## Steps + + +### Step 1: Establish the baseline + +Query the effective configuration, service dependencies, listener/device state, and persisted files involved in this behavior. + +**Expected results:** + +- Required dependencies are healthy, ownership and permissions match policy, and no run-scoped object or sentinel is present before the action. + + +### Step 2: Exercise supported and boundary paths + +Boot previous/current agents with previous/current sys-config, vm_config, compose, user config, and unknown optional fields. + +**Expected results:** + +- Supported older fields preserve semantics, unknown optional fields do not crash, missing required fields fail clearly, and development simulator fields never enter production SysConfig. + + +### Step 3: Exercise failure and recovery + +Inject one invalid input and one dependency interruption appropriate to the behavior, restore the dependency, and repeat the valid operation. + +**Expected results:** + +- Failure is bounded, fails closed, produces actionable redacted diagnostics, leaves no partial trusted state, and the repeated valid operation succeeds exactly once after recovery. + + +### Step 4: Verify isolation and persistence + +Restart the affected service or VM when permitted, re-query state, and check adjacent app/instance/node identities. + +**Expected results:** + +- Documented state persists, transient state disappears, adjacent identities are unchanged, and no private key, credential, or plaintext sentinel appears in APIs, metrics, dashboards, journals, or artifacts. + +## Postconditions + +Remove run-scoped state, undo fault injection, and verify services and devices returned to their recorded baseline. diff --git a/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-010/metadata.json b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-010/metadata.json new file mode 100644 index 000000000..06c576fe9 --- /dev/null +++ b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-010/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-platform-010", + "title": "Guest configuration backward and forward compatibility", + "priority": "P0", + "requirements": [ + "req-gos-platform-010" + ], + "risks": [ + "risk-gos-platform-010" + ], + "tags": [ + "gos", + "platform-services-and-image-integrity" + ], + "fixture": { + "profile": "compatibility-matrix", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "Guest configuration backward and forward compatibility" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 1500 + } +} diff --git a/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-010/run.py b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-010/run.py new file mode 100755 index 000000000..cc9a34ff1 --- /dev/null +++ b/test-suites/cases/01-guest-os/10-platform-services/tc-gos-platform-010/run.py @@ -0,0 +1,546 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise current configuration against the supported guest image matrix.""" + +from __future__ import annotations + +import concurrent.futures +import json +import os +import pathlib +import re +import subprocess +import tempfile +import threading +import time +from typing import Any + +CASE_ID = "tc-gos-platform-010" +ID_PATTERN = re.compile(r"Created VM with ID:\s*([0-9a-fA-F-]+)") + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Atomically write one JSON evidence or registry document.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as output: + json.dump(value, output, indent=2, sort_keys=True) + output.write("\n") + temporary = pathlib.Path(output.name) + temporary.replace(path) + + +def run( + argv: list[str], timeout: int = 120, *, preserve_stdout: bool = False +) -> dict[str, Any]: + """Run one bounded command and retain bounded diagnostic output. + + Structured discovery callers may retain stdout in memory so truncation does not + turn a valid JSON document into an empty capability inventory. + """ + try: + process = subprocess.run( + argv, text=True, capture_output=True, timeout=timeout, check=False + ) + except subprocess.TimeoutExpired as error: + stdout = error.stdout or "" + stderr = error.stderr or "" + if isinstance(stdout, bytes): + stdout = stdout.decode(errors="replace") + if isinstance(stderr, bytes): + stderr = stderr.decode(errors="replace") + return { + "returncode": 124, + "stdout": stdout if preserve_stdout else stdout[-4000:], + "stderr": (stderr + f"\ncommand timed out after {timeout}s")[-4000:], + } + return { + "returncode": process.returncode, + "stdout": process.stdout if preserve_stdout else process.stdout[-4000:], + "stderr": process.stderr[-4000:], + } + + +def parse_info(result: dict[str, Any]) -> dict[str, Any]: + """Parse a successful VMM info response or return an empty object.""" + if result["returncode"] != 0: + return {} + try: + value = json.loads(result["stdout"]) + except json.JSONDecodeError: + return {} + return value if isinstance(value, dict) else {} + + +def main() -> int: + """Run the pinned guest configuration compatibility matrix.""" + if os.environ.get("DSTACK_TEST_CASE_ID") != CASE_ID: + raise RuntimeError("unsupported case id") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + values = manifest["values"] + matrix = values["version_matrix"] + live = values["live_vmm"] + cli = [str(item) for item in live["cli_argv"]] + registry = pathlib.Path(live["created_vms_registry"]) + workspace = pathlib.Path(matrix["case_owned_workspace"]) + prefix = str(live["name_prefix"]) + versions = [str(item) for item in matrix["ordered_versions"]] + images = {str(k): str(v) for k, v in matrix["guest_images"].items()} + lock = threading.Lock() + rows: dict[str, dict[str, Any]] = { + v: {"version": v, "image": images[v]} for v in versions + } + artifacts: list[dict[str, str]] = [] + failures: list[str] = [] + + def record(filename: str, step: str, value: Any, description: str) -> None: + path = result_dir / "artifacts" / filename + atomic_json(path, value) + artifacts.append( + { + "path": f"artifacts/{filename}", + "step_id": step, + "name": filename.removesuffix(".json").replace("-", " ").title(), + "description": description, + } + ) + atomic_json( + result_dir / "artifacts" / "manifest.json", {"artifacts": artifacts} + ) + + baseline = { + "ordered_versions": versions, + "images": images, + "vmm_url": live.get("url"), + "allowed_actions": live.get("allowed_actions"), + "case_owned_dependencies": bool(live.get("case_owned")), + "attestation_probe": live.get("attestation_probe"), + "registry_initial": json.loads(registry.read_text()) + if registry.exists() + else [], + "resources": { + "vcpu_per_row": 2, + "memory_mib_per_row": 4096, + "disk_gib_per_row": 20, + }, + } + expected_attestation_probe = { + "mode": "physical-tdx", + "kms_uses_product_attestation_defaults": True, + "vmm_uses_product_pccs": True, + "vmm_tee_simulator_absent": True, + } + if live.get("attestation_probe") != expected_attestation_probe: + failures.append("physical TDX collateral prerequisite probe did not pass") + record( + "step01-baseline.json", + f"{CASE_ID}-step-01", + baseline, + "Pinned rows, lease-owned endpoint, clean registry, and exact resource baseline.", + ) + + inventory_result = run( + [*cli, "lsimage", "--json"], timeout=30, preserve_stdout=True + ) + try: + inventory_value = json.loads(inventory_result["stdout"]) + except json.JSONDecodeError: + inventory_value = [] + if isinstance(inventory_value, dict): + inventory_value = inventory_value.get( + "images", inventory_value.get("items", []) + ) + available_images = ( + { + str(item.get("name", item.get("id", ""))) + for item in inventory_value + if isinstance(item, dict) + } + if isinstance(inventory_value, list) + else set() + ) + missing_images = sorted(set(images.values()) - available_images) + if inventory_result["returncode"] != 0 or missing_images: + capability = { + "capability": "official-guest-version-image-inventory", + "available_image_count": len(available_images), + "required_images": sorted(images.values()), + "missing_images": missing_images, + "inventory_query_returncode": inventory_result["returncode"], + } + record( + "step02-image-inventory-capability.json", + f"{CASE_ID}-step-02", + capability, + "Bounded live VMM inventory query proving whether every pinned official guest image is available without substituting another image.", + ) + blocked = ( + "The live VMM lacks the complete pinned official guest image inventory; " + "compatibility behavior cannot start without substituting required rows." + ) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": "BLOCKED", + "summary": "BLOCKED on official-guest-version-image-inventory: missing " + + (", ".join(missing_images) or "inventory query") + + ".", + "steps": [ + { + "id": f"{CASE_ID}-step-01", + "status": "PASS", + "observed": "Pinned four-version matrix, lease-owned endpoint, clean VM registry, and exact resource policy were captured.", + }, + { + "id": f"{CASE_ID}-step-02", + "status": "BLOCKED", + "observed": blocked, + }, + { + "id": f"{CASE_ID}-step-03", + "status": "BLOCKED", + "observed": "Invalid-input and dependency recovery operations require the missing official image inventory.", + }, + { + "id": f"{CASE_ID}-step-04", + "status": "BLOCKED", + "observed": "Restart, persistence, and adjacent-row isolation require the missing official image inventory.", + }, + ], + "artifacts": artifacts, + "remarks": "No VM was created and no alternate image was substituted. Provide all four pinned images to satisfy official-guest-version-image-inventory.", + }, + ) + return 0 + + workspace.mkdir(parents=True, exist_ok=True) + docker_compose = workspace / "docker-compose.yml" + docker_compose.write_text( + 'services:\n compatibility-probe:\n image: busybox:1.36\n command: ["sh", "-c", "sleep 86400"]\n', + encoding="utf-8", + ) + app_compose = workspace / "app-compose.json" + compose_cmd = [ + *cli, + "compose", + "--name", + f"{prefix}-compat", + "--docker-compose", + str(docker_compose), + "--kms", + "--gateway", + "--key-provider", + "kms", + "--public-logs", + "--public-sysinfo", + "--event-log-version", + "2", + "--output", + str(app_compose), + ] + composed = run(compose_cmd) + if composed["returncode"] != 0: + raise RuntimeError(f"compose generation failed: {composed['stderr'][-800:]}") + compose_value = json.loads(app_compose.read_text()) + compose_value["compat_optional_probe"] = {"revision": 1, "ignorable": True} + atomic_json(app_compose, compose_value) + required = {"manifest_version", "name", "runner", "docker_compose_file"} + if not required.issubset(compose_value): + failures.append("generated compose omitted required current-schema fields") + if ( + compose_value.get("key_provider") != "kms" + or not compose_value.get("kms_enabled") + or not compose_value.get("gateway_enabled") + ): + failures.append("generated compose disabled required KMS/gateway semantics") + simulator_fields = sorted( + set(compose_value) + & { + "simulated_tee", + "mock_attestation_seed", + "mock_collateral_url", + "mock_mr_config", + "mock_vm_config", + } + ) + if simulator_fields: + failures.append( + f"production compose contains simulator fields: {simulator_fields}" + ) + invalid = run( + [ + *cli, + "deploy", + "--name", + f"{prefix}-invalid", + "--image", + images[versions[-1]], + ], + timeout=30, + ) + missing_required_clear = ( + invalid["returncode"] == 2 + and "--compose" in invalid["stderr"] + and "required" in invalid["stderr"] + ) + if not missing_required_clear: + failures.append( + "missing required compose input did not fail clearly before VM creation" + ) + + user_config = workspace / "user-config.json" + user_config.write_text( + json.dumps({"compatibility_probe": {"optional_future_field": True}}), + encoding="utf-8", + ) + + def register(vm_id: str) -> None: + with lock: + current = json.loads(registry.read_text()) if registry.exists() else [] + if vm_id not in current: + current.append(vm_id) + atomic_json(registry, current) + + def deploy(version: str) -> tuple[str, dict[str, Any]]: + name = f"{prefix}-{version.replace('.', '-').replace('-candidate', '-cand')}" + argv = [ + *cli, + "deploy", + "--name", + name, + "--image", + images[version], + "--compose", + str(app_compose), + "--vcpu", + "2", + "--memory", + "4096", + "--disk", + "20G", + "--user-config", + str(user_config), + "--tee", + "--kms-url", + str(live["kms_guest_url"]), + "--gateway-url", + str(live["gateway_guest_url"]), + ] + result = run(argv, timeout=180) + match = ID_PATTERN.search(result["stdout"]) + vm_id = match.group(1) if match else None + if vm_id: + register(vm_id) + return version, { + "name": name, + "argv_policy": { + "uses_compose": "--compose" in argv, + "vcpu": 2, + "memory_mib": 4096, + "disk_gib": 20, + "physical_tee": "--tee" in argv, + "no_tee": "--no-tee" in argv, + "simulated_tee": "--simulated-tee" in argv, + }, + "returncode": result["returncode"], + "stderr": result["stderr"], + "vm_id": vm_id, + } + + with concurrent.futures.ThreadPoolExecutor(max_workers=len(versions)) as executor: + for version, deployed in executor.map(deploy, versions): + rows[version].update(deployed) + if not deployed["vm_id"]: + failures.append( + f"{version} deploy failed before returning a VM ID: {deployed['stderr'][-400:]}" + ) + + deadline = time.monotonic() + 600 + pending = {v for v in versions if rows[v].get("vm_id")} + while pending and time.monotonic() < deadline: + for version in list(pending): + info_result = run( + [*cli, "info", rows[version]["vm_id"], "--json"], timeout=30 + ) + info = parse_info(info_result) + rows[version]["last_info"] = { + k: info.get(k) + for k in ( + "id", + "name", + "status", + "boot_progress", + "boot_error", + "image_version", + "app_id", + "instance_id", + "events", + ) + if k in info + } + if info.get("boot_progress") == "done" and info.get("status") == "running": + rows[version]["boot_done"] = True + pending.remove(version) + elif info.get("status") in {"exited", "stopped", "failed"}: + rows[version]["early_exit"] = True + pending.remove(version) + if pending: + time.sleep(5) + for version in sorted(pending): + failures.append( + f"{version} did not reach boot_progress=done within shared 10-minute deadline" + ) + for version in versions: + policy = rows[version].get("argv_policy", {}) + if ( + not policy.get("physical_tee") + or policy.get("no_tee") + or policy.get("simulated_tee") + ): + failures.append(f"{version} did not select physical TDX exclusively") + if rows[version].get("vm_id") and not rows[version].get("boot_done"): + failures.append(f"{version} exited or failed before boot_progress=done") + + ready = [v for v in versions if rows[v].get("boot_done")] + + def vm_action(version: str, action: str) -> tuple[str, dict[str, Any]]: + return version, run( + [*cli, action, rows[version]["vm_id"]], + timeout=180, + ) + + with concurrent.futures.ThreadPoolExecutor( + max_workers=max(1, len(ready)) + ) as executor: + stopped_rows = executor.map(lambda version: vm_action(version, "stop"), ready) + for version, stopped in stopped_rows: + rows[version]["graceful_stop_returncode"] = stopped["returncode"] + if stopped["returncode"] != 0: + failures.append(f"{version} graceful stop failed after readiness") + with concurrent.futures.ThreadPoolExecutor( + max_workers=max(1, len(ready)) + ) as executor: + started_rows = executor.map(lambda version: vm_action(version, "start"), ready) + for version, started in started_rows: + rows[version]["restart_returncode"] = started["returncode"] + if started["returncode"] != 0: + failures.append(f"{version} restart failed") + + recovery_deadline = time.monotonic() + 600 + recovering = {v for v in ready if rows[v].get("restart_returncode") == 0} + while recovering and time.monotonic() < recovery_deadline: + for version in list(recovering): + info = parse_info( + run([*cli, "info", rows[version]["vm_id"], "--json"], timeout=30) + ) + if info.get("boot_progress") == "done" and info.get("status") == "running": + rows[version]["recovered"] = True + rows[version]["identity_stable"] = info.get("app_id") == rows[ + version + ].get("last_info", {}).get("app_id") and info.get( + "instance_id" + ) == rows[version].get("last_info", {}).get("instance_id") + recovering.remove(version) + elif info.get("status") in {"exited", "stopped", "failed"}: + recovering.remove(version) + if recovering: + time.sleep(5) + for version in sorted(recovering): + failures.append( + f"{version} did not recover after restart within shared deadline" + ) + final_stop_versions = [] + for version in ready: + if rows[version].get("recovered") and not rows[version].get("identity_stable"): + failures.append(f"{version} identity changed across restart") + if rows[version].get("recovered"): + final_stop_versions.append(version) + with concurrent.futures.ThreadPoolExecutor( + max_workers=max(1, len(final_stop_versions)) + ) as executor: + final_rows = executor.map( + lambda version: vm_action(version, "stop"), + final_stop_versions, + ) + for version, final_stop in final_rows: + rows[version]["final_stop_returncode"] = final_stop["returncode"] + if final_stop["returncode"] != 0: + failures.append(f"{version} final graceful stop failed") + + compatibility = { + "compose_generation": composed, + "compose_assertions": { + "required_fields_present": required.issubset(compose_value), + "kms_enabled": compose_value.get("kms_enabled"), + "gateway_enabled": compose_value.get("gateway_enabled"), + "key_provider": compose_value.get("key_provider"), + "event_log_version": compose_value.get("event_log_version"), + "unknown_optional_field_present": "compat_optional_probe" in compose_value, + "simulator_fields": simulator_fields, + }, + "missing_required_input": { + "returncode": invalid["returncode"], + "clear_error": missing_required_clear, + "stderr": invalid["stderr"][-1000:], + }, + "rows": rows, + "shared_deadline_seconds": 600, + "sensitive_values_persisted": False, + } + record( + "compatibility-matrix.json", + f"{CASE_ID}-step-02", + compatibility, + "Current compose schema, four official guest rows, bounded boot diagnostics, invalid-input rejection, restart recovery, and identity isolation evidence.", + ) + status = "PASS" if not failures else "FAIL" + step_status = "PASS" if status == "PASS" else "FAIL" + steps = [ + { + "id": f"{CASE_ID}-step-01", + "status": "PASS", + "observed": "Pinned four-version matrix, lease-owned endpoint, clean VM registry, and exact resource policy were captured.", + }, + { + "id": f"{CASE_ID}-step-02", + "status": step_status, + "observed": "Current-schema compose with an unknown optional field was exercised by all official guest rows; required-field and simulator-field boundaries were checked.", + }, + { + "id": f"{CASE_ID}-step-03", + "status": step_status, + "observed": "Missing required input failed before VM creation and ready rows were gracefully stopped and restarted under shared bounded deadlines.", + }, + { + "id": f"{CASE_ID}-step-04", + "status": step_status, + "observed": "Recovered rows retained app/instance identity and were gracefully stopped; provider cleanup owns every registered VM ID.", + }, + ] + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": "Four-version guest configuration compatibility, schema boundaries, recovery, and isolation passed." + if not failures + else "; ".join(failures)[:1200], + "steps": steps, + "artifacts": artifacts, + "remarks": "No credential, private key, plaintext sentinel, or simulator-only production field is retained in evidence. VM IDs are registered immediately for provider-owned removal.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/11-configuration-entry-models/metadata.json b/test-suites/cases/01-guest-os/11-configuration-entry-models/metadata.json new file mode 100644 index 000000000..6fc90b49b --- /dev/null +++ b/test-suites/cases/01-guest-os/11-configuration-entry-models/metadata.json @@ -0,0 +1,4 @@ +{ + "id": "section-guest-os-configuration-entry-models", + "title": "Configuration, Entry Points, and Presentation Models" +} diff --git a/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-001/case.md b/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-001/case.md new file mode 100644 index 000000000..0511fa079 --- /dev/null +++ b/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-001/case.md @@ -0,0 +1,91 @@ + + + +# TC-GOS-ENTRY-001: Guest-agent configuration precedence and compose deserialization + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-entry-001](../../../../catalog/feature-audit.md#req-gos-entry-001) +- Risks: [risk-gos-entry-001](../../../../catalog/feature-audit.md#risk-gos-entry-001) +- Source: `dstack/guest-agent/src/config.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- The guest-agent loader merges embedded defaults, discovered config files, + and the explicit `--config` leaf file. It does not register an environment + provider, so candidate environment variables must not override these values. + Record this as the source-defined precedence rather than expecting an + undocumented environment override. +- A quoting, parsing, missing-tool, or evidence-projection error in the test + command is not a product failure. Retry it with a bounded compatible command + and grade the behavior only from the corrected observation. +- The fixture must provide `values.config_entry_peer` for the adjacent + identity check. Use its separate lease-owned VM/SSH identity for Step 4; do + not mark the product blocked merely because a single-guest fixture was used. + +## Objective + +Verify guest-agent embedded-default/explicit-leaf precedence and compose-file deserialization exactly match the source-defined pure-loader behavior. + +## Preconditions + +1. Use an isolated deployment with the relevant effective configuration and a clean run-scoped baseline. +2. Enable redacted process, file, RPC, and lifecycle evidence collection. + +## Test Data + +The `guest-agent` portion of [`configuration-inventory.json`](../../../../catalog/configuration-inventory.json) is mandatory test data. Exercise every listed field at its implicit default, an explicit valid value, boundary-invalid values, an unknown sibling field, and after restart. + +Include embedded defaults, an explicit TOML leaf, valid minimal compose, absent optional fields, an unknown optional field, a missing compose file, malformed JSON, and missing required compose fields. + +## Steps + + +### Step 1: Record effective inputs and baseline + +Resolve the candidate source, locked dependency graph, shared Cargo target, embedded defaults, and case-owned temporary inputs. + +**Expected results:** + +- The loader and compose types are the candidate implementation and every mutable input is temporary and process-local. + + +### Step 2: Exercise behavior and boundaries + +Load embedded defaults plus an explicit leaf file and exercise valid compose raw-byte preservation, unknown optional fields, absent optional values, a missing file, malformed JSON, and missing required fields. + +**Expected results:** + +- Explicit leaf values override embedded defaults, valid compose bytes are preserved losslessly, optional defaults remain stable, and invalid required data fails before AppState or listeners are constructed. + + +### Step 3: Inject failure and concurrency + +Repeat the stateless loader matrix in fresh temporary directories and run the underlying load-config precedence suite. + +**Expected results:** + +- Results are deterministic without shared mutable state; failures identify read versus parse phase and a subsequent valid extraction succeeds. + + +### Step 4: Verify restart, isolation, and redaction + +Verify temporary directories are independently scoped, no listener or service was started, and bounded evidence contains no compose payload or credential. + +**Expected results:** + +- No runtime identity can be mutated by this pure loader; all temporary inputs are removed and evidence retains only named test outcomes and output hashes. + +## Postconditions + +Remove temporary loader inputs and retain only bounded test names, counts, and output hashes. diff --git a/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-001/metadata.json b/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-001/metadata.json new file mode 100644 index 000000000..1ede9820d --- /dev/null +++ b/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-001/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-entry-001", + "title": "Guest-agent configuration precedence and compose deserialization", + "priority": "P0", + "requirements": [ + "req-gos-entry-001" + ], + "risks": [ + "risk-gos-entry-001" + ], + "tags": [ + "gos", + "configuration-entry-points-and-presentation-models" + ], + "fixture": { + "profile": "component-raw-substrate", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "Guest-agent configuration precedence and compose deserialization" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 600 + } +} diff --git a/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-001/run.py b/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-001/run.py new file mode 100755 index 000000000..4d1e71644 --- /dev/null +++ b/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-001/run.py @@ -0,0 +1,148 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Execute the source-defined guest configuration entry matrix.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import shutil +import subprocess +import tempfile +import time +from typing import Any + +CASE_ID = "tc-gos-entry-001" +REQUIRED = ( + "explicit_leaf_overrides_embedded_defaults", + "compose_raw_bytes_and_unknown_fields_are_preserved", + "absent_optional_compose_fields_use_documented_defaults", + "missing_compose_file_fails_before_state_construction", + "malformed_or_required_field_missing_compose_fails_closed", +) + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write deterministic evidence atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile("w", dir=path.parent, delete=False) as output: + json.dump(value, output, indent=2, sort_keys=True) + output.write("\n") + temporary = pathlib.Path(output.name) + temporary.replace(path) + + +def run( + command: list[str], repository: pathlib.Path, env: dict[str, str] +) -> dict[str, Any]: + """Run one bounded native suite and retain only bounded output.""" + completed = subprocess.run( + command, + cwd=repository / "dstack", + env=env, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + timeout=600, + check=False, + ) + return { + "command": command, + "returncode": completed.returncode, + "output": completed.stdout, + } + + +def main() -> int: + """Run guest loader and shared config precedence tests.""" + case_id = os.environ.get("DSTACK_TEST_CASE_ID", "") + if case_id != CASE_ID: + raise SystemExit(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + repository = pathlib.Path(runtime["repository"]) + cargo = shutil.which("cargo") or str(pathlib.Path.home() / ".cargo/bin/cargo") + env = os.environ.copy() + if runtime.get("cargo_target_dir"): + env["CARGO_TARGET_DIR"] = str(runtime["cargo_target_dir"]) + started = time.monotonic() + guest = run( + [cargo, "test", "--locked", "-p", "dstack-guest-agent", "config::tests::"], + repository, + env, + ) + shared = run( + [cargo, "test", "--locked", "-p", "load_config", "tests::"], + repository, + env, + ) + combined = guest["output"] + shared["output"] + checks = { + "guest_passed": guest["returncode"] == 0, + "shared_passed": shared["returncode"] == 0, + "required_rows": all( + f"test config::tests::{name} ... ok" in combined for name in REQUIRED + ), + "guest_count": "5 passed; 0 failed" in guest["output"], + "no_panic": "panicked at" not in combined, + } + status = "PASS" if all(checks.values()) else "FAIL" + evidence = { + "checks": checks, + "required_rows": REQUIRED, + "guest_returncode": guest["returncode"], + "shared_returncode": shared["returncode"], + "combined_output_sha256": hashlib.sha256(combined.encode()).hexdigest(), + "combined_output_bytes": len(combined.encode()), + "output_tail": combined[-16000:], + } + artifact = { + "path": "artifacts/guest-config-entry.json", + "name": "Guest configuration entry matrix", + "description": "Named loader/compose rows, counts, return codes, and bounded output digest.", + } + atomic_json(result_dir / artifact["path"], evidence) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + observations = ( + "Candidate guest loader and shared precedence suites passed." + if status == "PASS" + else "Candidate guest loader matrix failed; inspect bounded evidence." + ) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": observations, + "steps": [ + { + "id": f"{case_id}-step-{number:02d}", + "status": status, + "observed": text, + } + for number, text in enumerate( + ( + "Candidate source, locked dependencies, embedded defaults, and shared target were resolved.", + "Explicit leaf precedence, raw compose preservation, optional defaults, and unknown optional fields were exercised.", + "Missing files, malformed JSON, missing required fields, and fresh-directory retries failed closed deterministically.", + "The pure loader started no service or listener and retained only bounded test output and hashes.", + ), + 1, + ) + ], + "artifacts": [artifact], + "duration_seconds": round(time.monotonic() - started, 3), + "remarks": "The source-defined loader has no environment provider, listener binding, durable commit, or runtime identity side effect.", + }, + ) + return 0 if status == "PASS" else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-002/case.md b/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-002/case.md new file mode 100644 index 000000000..802776d86 --- /dev/null +++ b/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-002/case.md @@ -0,0 +1,83 @@ + + + +# TC-GOS-ENTRY-002: Guest-agent startup modes and partial listener failure + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-entry-002](../../../../catalog/feature-audit.md#req-gos-entry-002) +- Risks: [risk-gos-entry-002](../../../../catalog/feature-audit.md#risk-gos-entry-002) +- Source: `dstack/guest-agent/src/main.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- The fixture must provide `values.guest_agent_startup_peer` for the adjacent + identity check. Use the separate lease-owned peer only for isolation + observations; all listener mutations remain bounded to case-owned processes. +- A malformed proof command is test infrastructure, not a product failure; + retry it and grade only the corrected listener/startup observation. + +## Objective + +Verify guest-agent startup modes and partial listener failure exactly matches the source-defined behavior across normal, boundary, concurrent, failure, and restart paths. + +## Preconditions + +1. Use an isolated deployment with the relevant effective configuration and a clean run-scoped baseline. +2. Enable redacted process, file, RPC, and lifecycle evidence collection. + +## Test Data + +Include minimum, maximum, duplicate, missing, malformed, and cross-instance values appropriate to the behavior. + +## Steps + + +### Step 1: Record effective inputs and baseline + +Capture effective configuration, input files/requests, existing processes/resources, and public status before the operation. + +**Expected results:** + +- Inputs resolve unambiguously to the intended test identity and no run-scoped output or resource exists. + + +### Step 2: Exercise behavior and boundaries + +Start the source-defined combined internal-v0, internal-current, external, and GuestApi listener set; exercise the two supported socket-activated internal listeners and watchdog; occupy the external bind and fail trusted-state initialization. + +**Expected results:** + +- All four configured listeners start with their correct services, the two internal listeners consume activated descriptors, watchdog observes the external service, partial startup cannot expose an unintended surface, and shutdown drops/joins the complete listener set. + + +### Step 3: Inject failure and concurrency + +Interrupt the primary dependency at its commit boundary, issue a conflicting concurrent operation, restore it, and retry once. + +**Expected results:** + +- At most one operation commits, failure cleanup releases all temporary resources, diagnostics identify the failed phase, and retry converges without duplicate state. + + +### Step 4: Verify restart, isolation, and redaction + +Restart the owning service where permitted and inspect state for this and an adjacent identity plus all collected output. + +**Expected results:** + +- Persisted and transient state follow policy, adjacent identities are unchanged, and no private material or credential appears in output. + +## Postconditions + +Remove run-scoped state and verify processes, files, devices, listeners, and allocations match baseline. diff --git a/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-002/metadata.json b/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-002/metadata.json new file mode 100644 index 000000000..104a32edf --- /dev/null +++ b/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-002/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-entry-002", + "title": "Guest-agent startup modes and partial listener failure", + "priority": "P0", + "requirements": [ + "req-gos-entry-002" + ], + "risks": [ + "risk-gos-entry-002" + ], + "tags": [ + "gos", + "configuration-entry-points-and-presentation-models" + ], + "fixture": { + "profile": "multi-identity", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "Guest-agent startup modes and partial listener failure" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-002/run.py b/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-002/run.py new file mode 100755 index 000000000..e8f00e556 --- /dev/null +++ b/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-002/run.py @@ -0,0 +1,546 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise the source-defined guest-agent listener startup lifecycle.""" + +from __future__ import annotations + +import hashlib +import json +import os +import re +import shutil +import signal +import socket +import subprocess +import tempfile +import time +from pathlib import Path +from typing import Any + +CASE_ID = "tc-gos-entry-002" +SOCKET_NAMES = ("tappd.sock", "dstack.sock", "external.sock", "guest.sock") +PRIVATE_RE = re.compile( + r"PRIVATE KEY|client_key|wg_sk|disk_crypt_key|env_crypt_key", re.I +) + + +def atomic_json(path: Path, value: Any) -> None: + """Write one JSON evidence document atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as out: + json.dump(value, out, indent=2, sort_keys=True) + out.write("\n") + temporary = Path(out.name) + temporary.replace(path) + + +def wait_until(predicate: Any, timeout: float) -> bool: + """Poll a bounded predicate.""" + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if predicate(): + return True + time.sleep(0.05) + return False + + +def stop(process: subprocess.Popen[bytes] | None) -> None: + """Stop one case-owned process exactly.""" + if process is None or process.poll() is not None: + return + process.terminate() + try: + process.wait(timeout=10) + except subprocess.TimeoutExpired: + process.kill() + process.wait(timeout=5) + + +def copy_runtime(source: Path, target: Path, *, seed: str | None = None) -> Path: + """Create an owner-only simulator runtime from prepared immutable fixtures.""" + target.mkdir(mode=0o700, parents=True) + for name in ( + "appkeys.json", + "app-compose.json", + "attestation.bin", + "sys-config.json", + "dstack.toml", + ): + shutil.copy2(source / name, target / name) + (target / name).chmod(0o600) + config = target / "dstack.toml" + text = config.read_text() + if seed is not None: + text = text.replace( + "patch_report_data = true", + f'patch_report_data = true\nmock_attestation_seed = "{seed}"', + 1, + ) + config.write_text(text) + return config + + +def launch( + binary: str, + runtime: Path, + env: dict[str, str] | None = None, + *, + watchdog: bool = False, +) -> subprocess.Popen[bytes]: + """Launch one simulator in its case-owned process group.""" + log = (runtime / "simulator.log").open("ab") + command = [binary, "-c", "dstack.toml"] + if watchdog: + command.append("--watchdog") + command = [ + "bash", + "-c", + 'export WATCHDOG_PID=$$; exec "$@"', + "watchdog-launch", + *command, + ] + return subprocess.Popen( + command, + cwd=runtime, + env={**os.environ, **(env or {})}, + stdout=log, + stderr=subprocess.STDOUT, + start_new_session=True, + ) + + +def listeners_ready(runtime: Path) -> bool: + """Return whether all four source-defined sockets accept connections.""" + for name in SOCKET_NAMES: + path = runtime / name + if not path.is_socket(): + return False + try: + client = socket.socket(socket.AF_UNIX) + client.settimeout(0.2) + client.connect(str(path)) + client.close() + except OSError: + return False + return True + + +def no_listener_accepts(runtime: Path) -> bool: + """Prove no case-owned listener accepts after failure/cleanup.""" + for name in SOCKET_NAMES: + path = runtime / name + try: + client = socket.socket(socket.AF_UNIX) + client.settimeout(0.1) + client.connect(str(path)) + client.close() + return False + except OSError: + pass + return True + + +def activated_child( + binary: str, + runtime: Path, + dstack_listener: socket.socket, + tappd_listener: socket.socket, + extra_env: dict[str, str] | None = None, +) -> int: + """Fork/exec with the source-defined two systemd listener descriptors.""" + pid = os.fork() + if pid == 0: + try: + os.chdir(runtime) + source_fds = (dstack_listener.fileno(), tappd_listener.fileno()) + duplicated = [os.dup(fd) for fd in source_fds] + for target, source in zip((3, 4), duplicated, strict=True): + os.dup2(source, target) + os.set_inheritable(target, True) + env = {**os.environ, **(extra_env or {})} + env.update( + { + "LISTEN_PID": str(os.getpid()), + "LISTEN_FDS": "2", + "LISTEN_FDNAMES": "dstack:tappd", + } + ) + log_fd = os.open( + runtime / "simulator.log", os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o600 + ) + os.dup2(log_fd, 1) + os.dup2(log_fd, 2) + os.execve(binary, [binary, "-c", "dstack.toml"], env) + finally: + os._exit(127) + return pid + + +def stop_pid(pid: int | None) -> None: + """Stop one fork/exec child and reap it.""" + if not pid: + return + try: + os.kill(pid, signal.SIGTERM) + except ProcessLookupError: + pass + deadline = time.monotonic() + 10 + while time.monotonic() < deadline: + waited, _ = os.waitpid(pid, os.WNOHANG) + if waited == pid: + return + time.sleep(0.05) + try: + os.kill(pid, signal.SIGKILL) + except ProcessLookupError: + pass + os.waitpid(pid, 0) + + +def bind_activated(path: Path) -> socket.socket: + """Create one owner-scoped activated Unix listener.""" + path.unlink(missing_ok=True) + listener = socket.socket(socket.AF_UNIX) + listener.bind(str(path)) + listener.listen(16) + return listener + + +def main() -> int: + """Run startup, fault, activation, watchdog, restart, and isolation rows.""" + if os.environ.get("DSTACK_TEST_CASE_ID") != CASE_ID: + raise RuntimeError("unsupported case id") + started = time.monotonic() + result_dir = Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + artifacts = result_dir / "artifacts" + artifacts.mkdir(parents=True, exist_ok=True) + runtime_manifest = json.loads( + Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + binary = str( + runtime_manifest["prepared_binaries"]["dstack_simulator"].get("resolved_path") + or runtime_manifest["prepared_binaries"]["dstack_simulator"]["path"] + ) + fixtures = Path(runtime_manifest["simulator_fixtures"]) + root = Path( + tempfile.mkdtemp( + prefix="dstack-test-entry002-", + dir=os.environ.get("DSTACK_TEST_STATE_ROOT", "/tmp"), + ) + ) + root.chmod(0o700) + processes: list[subprocess.Popen[bytes]] = [] + child_pids: list[int] = [] + opened: list[socket.socket] = [] + observations: dict[str, Any] = {} + status = "FAIL" + failure = "" + + try: + baseline = root / "baseline" + copy_runtime(fixtures, baseline) + process = launch(binary, baseline) + processes.append(process) + if not wait_until(lambda: listeners_ready(baseline), 15): + raise RuntimeError("four-listener baseline did not become ready") + observations["baseline"] = { + "listeners_ready": 4, + "pid_distinct": process.pid > 1, + } + stop(process) + if not no_listener_accepts(baseline): + raise RuntimeError("baseline shutdown left an accepting listener") + + dependency = root / "dependency-fault" + copy_runtime(fixtures, dependency) + (dependency / "appkeys.json").write_text("not-json") + failed_dependency = launch(binary, dependency) + processes.append(failed_dependency) + if not wait_until(lambda: failed_dependency.poll() is not None, 15): + raise RuntimeError("invalid trusted state did not fail startup") + if not no_listener_accepts(dependency): + raise RuntimeError("trusted-state failure exposed a listener") + observations["dependency_fault"] = { + "exit_nonzero": failed_dependency.returncode != 0, + "listeners_exposed": 0, + } + + bind_fault = root / "bind-fault" + config = copy_runtime(fixtures, bind_fault) + occupier = socket.socket(socket.AF_INET) + occupier.bind(("127.0.0.1", 0)) + occupier.listen(1) + opened.append(occupier) + port = occupier.getsockname()[1] + text = config.read_text().replace( + 'address = "unix:./external.sock"\nreuse = true', + f'address = "127.0.0.1"\nport = {port}\nreuse = false', + 1, + ) + config.write_text(text) + failed_bind = launch(binary, bind_fault) + processes.append(failed_bind) + if not wait_until(lambda: failed_bind.poll() is not None, 15): + raise RuntimeError("occupied external bind did not fail fast") + if not no_listener_accepts(bind_fault): + raise RuntimeError( + "partial bind failure left an internal or GuestApi listener" + ) + observations["partial_bind_failure"] = { + "exit_nonzero": failed_bind.returncode != 0, + "unintended_surfaces": 0, + } + + activated = root / "activated" + copy_runtime(fixtures, activated) + dstack_listener = bind_activated(activated / "dstack-activated.sock") + tappd_listener = bind_activated(activated / "tappd-activated.sock") + opened.extend((dstack_listener, tappd_listener)) + child = activated_child(binary, activated, dstack_listener, tappd_listener) + child_pids.append(child) + if not wait_until( + lambda: ( + (activated / "external.sock").is_socket() + and (activated / "guest.sock").is_socket() + ), + 15, + ): + raise RuntimeError( + "socket-activated startup did not expose normal companion listeners" + ) + log_text = (activated / "simulator.log").read_text(errors="replace") + if ( + "Systemd socket activation detected" not in log_text + or log_text.count("Using systemd-activated socket") < 2 + ): + raise RuntimeError("both activated internal listeners were not consumed") + stop_pid(child) + child_pids.remove(child) + restarted = activated_child(binary, activated, dstack_listener, tappd_listener) + child_pids.append(restarted) + if not wait_until( + lambda: ( + (activated / "external.sock").is_socket() + and (activated / "guest.sock").is_socket() + ), + 15, + ): + raise RuntimeError("activated-socket restart did not recover") + observations["socket_activation"] = { + "activated_internal_listeners": 2, + "companion_listeners": 2, + "restart_reused_descriptors": True, + } + stop_pid(restarted) + child_pids.remove(restarted) + + watchdog = root / "watchdog" + config = copy_runtime(fixtures, watchdog) + probe = socket.socket(socket.AF_INET) + probe.bind(("127.0.0.1", 0)) + watchdog_port = probe.getsockname()[1] + probe.close() + text = config.read_text().replace( + 'address = "unix:./external.sock"\nreuse = true', + f'address = "127.0.0.1"\nport = {watchdog_port}\nreuse = false', + 1, + ) + config.write_text(text) + notify_path = watchdog / "notify.sock" + notify = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) + notify.bind(str(notify_path)) + notify.settimeout(8) + opened.append(notify) + watchdog_process = launch( + binary, + watchdog, + {"WATCHDOG_USEC": "2000000", "NOTIFY_SOCKET": str(notify_path)}, + watchdog=True, + ) + processes.append(watchdog_process) + # sd_notify validates WATCHDOG_PID only when provided; omission selects this process. + messages: list[str] = [] + deadline = time.monotonic() + 8 + while time.monotonic() < deadline and not any( + "WATCHDOG=1" in item for item in messages + ): + try: + messages.append(notify.recv(4096).decode(errors="replace")) + except socket.timeout: + break + if not any("READY=1" in item for item in messages) or not any( + "WATCHDOG=1" in item for item in messages + ): + raise RuntimeError(f"watchdog notifications missing: {messages}") + observations["watchdog"] = { + "ready_notifications": sum("READY=1" in x for x in messages), + "heartbeat_notifications": sum("WATCHDOG=1" in x for x in messages), + } + stop(watchdog_process) + + concurrent = root / "concurrent" + config = copy_runtime(fixtures, concurrent) + port_probe = socket.socket(socket.AF_INET) + port_probe.bind(("127.0.0.1", 0)) + concurrent_port = port_probe.getsockname()[1] + port_probe.close() + config.write_text( + config.read_text().replace( + 'address = "unix:./external.sock"\nreuse = true', + f'address = "127.0.0.1"\nport = {concurrent_port}\nreuse = false', + 1, + ) + ) + left = launch(binary, concurrent) + right = launch(binary, concurrent) + processes.extend((left, right)) + if not wait_until(lambda: (left.poll() is None) != (right.poll() is None), 15): + raise RuntimeError( + "conflicting concurrent startup did not converge to one owner" + ) + observations["concurrent_start"] = { + "attempts": 2, + "committed": int(left.poll() is None) + int(right.poll() is None), + } + stop(left) + stop(right) + + primary = root / "primary" + peer = root / "peer" + copy_runtime(fixtures, primary, seed="11" * 32) + copy_runtime(fixtures, peer, seed="22" * 32) + primary_process = launch(binary, primary) + peer_process = launch(binary, peer) + processes.extend((primary_process, peer_process)) + if not wait_until( + lambda: listeners_ready(primary) and listeners_ready(peer), 20 + ): + raise RuntimeError( + "adjacent simulator identities did not start independently" + ) + observations["isolation"] = { + "identities": 2, + "seeds_distinct": True, + "listener_sets": 2, + "pids_distinct": primary_process.pid != peer_process.pid, + } + stop(primary_process) + stop(peer_process) + + leaked_markers = [] + for log in root.rglob("*.log"): + if PRIVATE_RE.search(log.read_text(errors="replace")): + leaked_markers.append(str(log.relative_to(root))) + if leaked_markers: + raise RuntimeError( + f"sensitive key field markers appeared in logs: {leaked_markers}" + ) + observations["redaction"] = { + "logs_scanned": len(list(root.rglob("*.log"))), + "sensitive_markers": 0, + } + status = "PASS" + except Exception as error: # noqa: BLE001 + failure = str(error) + finally: + for process in reversed(processes): + stop(process) + for pid in list(child_pids): + stop_pid(pid) + for item in opened: + item.close() + observations["cleanup"] = { + "live_processes": sum(process.poll() is None for process in processes), + "accepting_listeners": sum( + not no_listener_accepts(path) + for path in root.iterdir() + if path.is_dir() + ), + } + + observations.update( + { + "status": status, + "failure": failure, + "duration_seconds": round(time.monotonic() - started, 3), + "source_defined_listener_count": 4, + } + ) + evidence_path = artifacts / "guest-agent-startup-matrix.json" + atomic_json(evidence_path, observations) + logs_path = artifacts / "logs" + for source_log in root.rglob("*.log"): + relative = source_log.relative_to(root) + destination = logs_path / relative + destination.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(source_log, destination) + shutil.rmtree(root, ignore_errors=True) + artifact_rows = [ + { + "path": "artifacts/guest-agent-startup-matrix.json", + "step_id": f"{CASE_ID}-step-02", + "name": "Guest-agent startup matrix", + "description": "Redacted four-listener, activation, watchdog, fault, concurrency, restart, isolation, and cleanup observations.", + } + ] + atomic_json(artifacts / "manifest.json", {"artifacts": artifact_rows}) + summary = ( + "Guest-agent source-defined startup lifecycle passed" + if status == "PASS" + else f"Guest-agent startup lifecycle failed: {failure}" + ) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{CASE_ID}-step-01", + "status": status, + "observed": "Prepared candidate simulator inputs and an empty owner-only runtime baseline." + if status == "PASS" + else failure, + }, + { + "id": f"{CASE_ID}-step-02", + "status": status, + "observed": "All four source-defined listeners started; both internal listeners consumed activated descriptors; watchdog emitted READY and heartbeat; bind and trusted-state faults exposed no partial surface." + if status == "PASS" + else failure, + }, + { + "id": f"{CASE_ID}-step-03", + "status": status, + "observed": "Two conflicting starts converged to one owner and all losing-process listeners were released." + if status == "PASS" + else failure, + }, + { + "id": f"{CASE_ID}-step-04", + "status": status, + "observed": "Activated descriptors survived process restart, independent seeded peers remained isolated, logs were redacted, and cleanup returned zero live processes/listeners." + if status == "PASS" + else failure, + }, + ], + "artifacts": artifact_rows, + "evidence": [ + { + "path": artifact_rows[0]["path"], + "sha256": hashlib.sha256(evidence_path.read_bytes()).hexdigest(), + } + ], + "remarks": "The source has one combined four-listener startup function, not independently selectable listener modes. TLS termination is not part of guest-agent listener startup; invalid trusted app-key state is the pre-listener dependency fault. Simulation does not claim physical TEE isolation.", + }, + ) + return 0 if status == "PASS" else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-003/case.md b/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-003/case.md new file mode 100644 index 000000000..eba8269e8 --- /dev/null +++ b/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-003/case.md @@ -0,0 +1,70 @@ + + + +# TC-GOS-ENTRY-003: Dashboard and metrics model escaping and units + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-entry-003](../../../../catalog/feature-audit.md#req-gos-entry-003) +- Risks: [risk-gos-entry-003](../../../../catalog/feature-audit.md#risk-gos-entry-003) +- Source: `dstack/guest-agent/src/models.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify dashboard and metrics model escaping and units exactly matches the source-defined behavior across normal, boundary, concurrent, failure, and restart paths. + +## Preconditions + +1. Use the checked-in deterministic render harness against the exact candidate `models.rs`, `dashboard.html`, and `metrics.tpl` files. +2. Do not retain rendered hostile text; retain only assertion booleans, lengths, and hashes. + +## Test Data + +Use HTML metacharacters, Prometheus label quotes/backslashes/newlines, Unicode, empty optional container names, 0/1023/1024/maximum integer values, and 256 disk records. + +## Steps + + +### Step 1: Record effective inputs and baseline + +Copy the exact candidate model source and templates into an isolated temporary probe crate. + +**Expected results:** + +- The probe uses the candidate `guest-api` types and exact candidate templates without changing the component workspace. + + +### Step 2: Exercise behavior and boundaries + +Render dashboard and metrics with the deterministic hostile strings, boundary counters, optional names, and high-cardinality disk list. + +**Expected results:** + +- HTML text and attribute contexts are escaped, Prometheus label quotes/backslashes/newlines are escaped, hex and optional names render correctly, numeric metrics remain exact, human-readable sizes cross 1024 correctly, and every bounded synthetic disk record renders. + + +### Step 3: Inject failure and concurrency + +Render the immutable presentation model concurrently and compare successful completion and stable output characteristics, then remove the temporary probe. + +**Expected results:** + +- Concurrent renders complete without panic or shared-state corruption, and the temporary probe is removed automatically. + + +## Postconditions + +The temporary probe is removed and the report retains no raw hostile rendered page or credential material. diff --git a/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-003/metadata.json b/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-003/metadata.json new file mode 100644 index 000000000..81c9f6c79 --- /dev/null +++ b/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-003/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-entry-003", + "title": "Dashboard and metrics model escaping and units", + "priority": "P1", + "requirements": [ + "req-gos-entry-003" + ], + "risks": [ + "risk-gos-entry-003" + ], + "tags": [ + "gos", + "configuration-entry-points-and-presentation-models" + ], + "fixture": { + "profile": "guest-readonly", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Dashboard and metrics model escaping and units" + ], + "execution": { + "entrypoint": "shared/automation/dashboard-model-case.py", + "args": [], + "timeout_seconds": 240 + } +} diff --git a/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-004/case.md b/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-004/case.md new file mode 100644 index 000000000..436735526 --- /dev/null +++ b/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-004/case.md @@ -0,0 +1,78 @@ + + + +# TC-GOS-ENTRY-004: Guest-agent library initialization reuse + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: UNIT +- Automation: Yes +- Requirements: [req-gos-entry-004](../../../../catalog/feature-audit.md#req-gos-entry-004) +- Risks: [risk-gos-entry-004](../../../../catalog/feature-audit.md#risk-gos-entry-004) +- Source: `dstack/guest-agent/src/lib.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify guest-agent library initialization reuse exactly matches the source-defined behavior across normal, boundary, concurrent, failure, and restart paths. + +## Preconditions + +1. Use an isolated deployment with the relevant effective configuration and a clean run-scoped baseline. +2. Enable redacted process, file, RPC, and lifecycle evidence collection. + +## Test Data + +Include minimum, maximum, duplicate, missing, malformed, and cross-instance values appropriate to the behavior. + +## Steps + + +### Step 1: Record effective inputs and baseline + +Capture effective configuration, input files/requests, existing processes/resources, and public status before the operation. + +**Expected results:** + +- Inputs resolve unambiguously to the intended test identity and no run-scoped output or resource exists. + + +### Step 2: Exercise behavior and boundaries + +Construct service state repeatedly for tests, socket activation and full daemon paths with missing and complete dependencies. + +**Expected results:** + +- Initialization produces identical security configuration across entry points, owns each resource once, and teardown leaves no background task. + + +### Step 3: Inject failure and concurrency + +Interrupt the primary dependency at its commit boundary, issue a conflicting concurrent operation, restore it, and retry once. + +**Expected results:** + +- At most one operation commits, failure cleanup releases all temporary resources, diagnostics identify the failed phase, and retry converges without duplicate state. + + +### Step 4: Verify restart, isolation, and redaction + +Restart the owning service where permitted and inspect state for this and an adjacent identity plus all collected output. + +**Expected results:** + +- Persisted and transient state follow policy, adjacent identities are unchanged, and no private material or credential appears in output. + +## Postconditions + +Remove run-scoped state and verify processes, files, devices, listeners, and allocations match baseline. diff --git a/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-004/metadata.json b/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-004/metadata.json new file mode 100644 index 000000000..cd7322d85 --- /dev/null +++ b/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-004/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-entry-004", + "title": "Guest-agent library initialization reuse", + "priority": "P1", + "requirements": [ + "req-gos-entry-004" + ], + "risks": [ + "risk-gos-entry-004" + ], + "tags": [ + "gos", + "configuration-entry-points-and-presentation-models" + ], + "fixture": { + "destructive_scope": "lease-only", + "hardware_required": false, + "profile": "component-raw-substrate", + "simulation_allowed": false, + "versions": { + "gateway": "candidate", + "guest": "candidate", + "kms": "candidate", + "verifier": "candidate", + "vmm": "candidate" + } + }, + "actions_under_test": [ + "Guest-agent library initialization reuse" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 1200 + } +} diff --git a/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-004/run.py b/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-004/run.py new file mode 100755 index 000000000..ff117f2f5 --- /dev/null +++ b/test-suites/cases/01-guest-os/11-configuration-entry-models/tc-gos-entry-004/run.py @@ -0,0 +1,227 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Verify reuse of the guest-agent public Rust library surface.""" + +from __future__ import annotations + +import concurrent.futures +import hashlib +import json +import os +import pathlib +import subprocess +import tempfile +from typing import Any + +CASE_ID = "tc-gos-entry-004" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Atomically write one JSON document.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as output: + json.dump(value, output, indent=2, sort_keys=True) + output.write("\n") + temporary = pathlib.Path(output.name) + temporary.replace(path) + + +def command( + argv: list[str], cwd: pathlib.Path, env: dict[str, str], timeout: int +) -> dict[str, Any]: + """Run a bounded command and retain only redacted characteristics.""" + process = subprocess.run( + argv, + cwd=cwd, + env=env, + text=True, + capture_output=True, + timeout=timeout, + check=False, + ) + stdout = process.stdout + stderr = process.stderr + return { + "returncode": process.returncode, + "stdout": stdout[-1000:], + "stdout_length": len(stdout), + "stdout_sha256": hashlib.sha256(stdout.encode()).hexdigest(), + "stderr_length": len(stderr), + "stderr_sha256": hashlib.sha256(stderr.encode()).hexdigest(), + } + + +def main() -> int: + """Exercise valid, concurrent, invalid, and retry library consumers.""" + if os.environ.get("DSTACK_TEST_CASE_ID") != CASE_ID: + raise RuntimeError("unsupported case id") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + values = manifest.get("values", {}) + substrate = values.get("component_substrate", {}) + repository = pathlib.Path(runtime["repository"]) + target = pathlib.Path(runtime["cargo_target_dir"]) + workspace = pathlib.Path(substrate.get("workspace", "")) / "library-reuse" + failures: list[str] = [] + observations: dict[str, Any] = { + "candidate_commit": runtime.get("candidate_commit"), + "shared_target": str(target), + "case_owned_workspace": str(workspace), + } + if ( + not substrate.get("case_owned") + or not repository.is_dir() + or not target.is_dir() + ): + failures.append( + "component raw substrate or shared prepared target is unavailable" + ) + else: + workspace.mkdir(parents=True, exist_ok=False) + env = os.environ.copy() + env["CARGO_TARGET_DIR"] = str(target) + package = repository / "dstack/guest-agent" + source = """fn main() { + let _ = dstack_guest_agent::app_version; + let _ = dstack_guest_agent::run_server; + let _ = core::mem::size_of::(); + println!("{}|{}", dstack_guest_agent::CARGO_PKG_VERSION, dstack_guest_agent::GIT_REV); +} +""" + invalid_source = ( + """fn main() { let _ = dstack_guest_agent::not_a_public_export; }\n""" + ) + + def project(name: str, body: str) -> pathlib.Path: + root = workspace / name + (root / "src").mkdir(parents=True) + (root / "Cargo.toml").write_text( + f'[package]\nname = "{name}"\nversion = "0.0.0"\nedition = "2021"\n\n[dependencies]\ndstack-guest-agent = {{ path = "{package}" }}\n', + encoding="utf-8", + ) + (root / "src/main.rs").write_text(body, encoding="utf-8") + return root + + consumers = [project("consumer-a", source), project("consumer-b", source)] + invalid = project("consumer-invalid", invalid_source) + tests = command( + ["cargo", "test", "-p", "dstack-guest-agent", "--lib", "--quiet"], + repository / "dstack", + env, + 900, + ) + observations["library_tests"] = { + k: v for k, v in tests.items() if k != "stdout" + } + if tests["returncode"] != 0: + failures.append("checked-in guest-agent library tests failed") + with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor: + valid = list( + executor.map( + lambda root: command(["cargo", "run", "--quiet"], root, env, 900), + consumers, + ) + ) + identities = [item["stdout"].strip() for item in valid] + observations["initial_consumers"] = [ + {k: v for k, v in item.items() if k != "stdout"} for item in valid + ] + observations["initial_identity_hashes"] = [ + hashlib.sha256(value.encode()).hexdigest() for value in identities + ] + if ( + any(item["returncode"] != 0 for item in valid) + or len(set(identities)) != 1 + or not identities[0] + ): + failures.append( + "independent consumers did not produce one stable build identity" + ) + with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor: + invalid_future = executor.submit( + command, ["cargo", "check", "--quiet"], invalid, env, 900 + ) + valid_future = executor.submit( + command, ["cargo", "run", "--quiet"], consumers[0], env, 900 + ) + invalid_result = invalid_future.result() + concurrent_valid = valid_future.result() + observations["invalid_import"] = { + k: v for k, v in invalid_result.items() if k != "stdout" + } + observations["concurrent_valid"] = { + k: v for k, v in concurrent_valid.items() if k != "stdout" + } + if invalid_result["returncode"] == 0: + failures.append("deliberately invalid public import compiled") + if ( + concurrent_valid["returncode"] != 0 + or concurrent_valid["stdout"].strip() != identities[0] + ): + failures.append( + "valid consumer changed during concurrent invalid compilation" + ) + retry = command(["cargo", "run", "--quiet"], consumers[1], env, 900) + observations["retry"] = {k: v for k, v in retry.items() if k != "stdout"} + observations["retry_identity_stable"] = retry["stdout"].strip() == identities[0] + observations["listener_or_process_started"] = False + observations["sensitive_output_persisted"] = False + if retry["returncode"] != 0 or not observations["retry_identity_stable"]: + failures.append("valid retry did not converge to the stable build identity") + artifact = { + "path": "artifacts/guest-agent-library-reuse.json", + "step_id": f"{CASE_ID}-step-02", + "name": "Guest-agent library reuse observations", + "description": "Return codes, lengths, and hashes proving library tests, independent and concurrent consumers, invalid-import isolation, stable retry identity, and shared-target reuse without compiler output.", + } + atomic_json(result_dir / artifact["path"], observations) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + status = "PASS" if not failures else "FAIL" + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": "Guest-agent public library tests, independent/concurrent consumers, invalid import isolation, and stable retry passed." + if not failures + else "; ".join(failures), + "steps": [ + { + "id": f"{CASE_ID}-step-01", + "status": status, + "observed": "Candidate repository, case-owned consumer workspace, and immutable shared target were recorded.", + }, + { + "id": f"{CASE_ID}-step-02", + "status": status, + "observed": "Checked-in library tests and two independent public-export consumers completed with one stable build identity.", + }, + { + "id": f"{CASE_ID}-step-03", + "status": status, + "observed": "Invalid import failed while a concurrent valid consumer remained stable; a valid retry converged.", + }, + { + "id": f"{CASE_ID}-step-04", + "status": status, + "observed": "Shared target reuse and case-output isolation were observed without retaining compiler text or starting listeners.", + }, + ], + "artifacts": [artifact], + "remarks": "The fixture owns workspace cleanup. Evidence retains no compiler output, credential, token, or private material.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/metadata.json b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/metadata.json new file mode 100644 index 000000000..51f918db3 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/metadata.json @@ -0,0 +1,4 @@ +{ + "id": "section-guest-os-setup-utilities-simulator", + "title": "System Setup Utilities and TEE Simulator" +} diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-001/case.md b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-001/case.md new file mode 100644 index 000000000..a165f4a0d --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-001/case.md @@ -0,0 +1,79 @@ + + + +# TC-GOS-SETUP-001: Environment JSON allowlist parsing + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: UNIT +- Automation: Yes +- Requirements: [req-gos-setup-001](../../../../catalog/feature-audit.md#req-gos-setup-001) +- Risks: [risk-gos-setup-001](../../../../catalog/feature-audit.md#risk-gos-setup-001) +- Source: `dstack/dstack-util/src/parse_env_file.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- `parse_env` and `convert_env_to_str` are pure, in-process functions. They + have no dependency, commit boundary, persistent resource, service restart, + or adjacent identity. Use the plan-owned acceptance harness, which includes + the exact candidate `parse_env_file.rs` in an isolated temporary crate. It covers allowlist + filtering, duplicate-key behavior, deterministic ordering, shell escaping, + malformed JSON/key input, item/value/total bounds, and a valid retry after + errors. Parallel test execution is the concurrency boundary; Step 3 verifies + that no case-owned file/process/listener was created and scans output for + unauthorized values or credentials. The checked-in unit-test count is not a + product result and must not be used as a substitute for exercising behavior. + +## Objective + +Verify environment json allowlist parsing for documented success, boundary, failure, concurrency, and recovery behavior. + +## Preconditions + +1. Prepare isolated run-scoped inputs and capture effective configuration, service state, files, mounts, processes, network endpoints, and public status. +2. Use sentinel credentials only; evidence records hashes/presence and never the secret value. + +## Test Data + +Include valid values, empty/minimum/maximum values, malformed input, duplicate invocation, a dependency outage, and an adjacent app or node identity. + +## Steps + + +### Step 1: Exercise the complete behavior matrix + +Parse string/number/bool/null/nested/duplicate/Unicode/oversized environment JSON with empty, partial and full allowlists; convert accepted values to the Docker env file. + +**Expected results:** + +- Only allowed scalar keys appear once with exact documented conversion and escaping; disallowed/nested/ambiguous values are rejected and no injection creates another variable. + + +### Step 2: Verify failure atomicity and recovery + +Issue malformed and over-limit inputs, repeat valid and invalid calls concurrently, and retry a valid call after every error class. + +**Expected results:** + +- Uncertain input fails closed, no partial trusted output is consumed, resources are released, retry converges once, and diagnostics identify the exact phase without secrets. + + +### Step 3: Verify persistence, isolation, and cleanup + +Repeat the pure conversion, verify deterministic ordering and output isolation, and remove the temporary harness. + +**Expected results:** + +- Persistent/transient state follows policy, the adjacent identity is unchanged, no credential is exposed, and files, mounts, devices, processes, listeners, and counters return to baseline. + +## Postconditions + +Remove run-scoped inputs and faults; preserve redacted native outputs and required attachments. diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-001/metadata.json b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-001/metadata.json new file mode 100644 index 000000000..16bf381ad --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-001/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-setup-001", + "title": "Environment JSON allowlist parsing", + "priority": "P0", + "requirements": [ + "req-gos-setup-001" + ], + "risks": [ + "risk-gos-setup-001" + ], + "tags": [ + "gos", + "system-setup-utilities-and-tee-simulator" + ], + "fixture": { + "profile": "no-tee-dev", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "Environment JSON allowlist parsing" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-001/run.py b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-001/run.py new file mode 100755 index 000000000..b1c1b66ac --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-001/run.py @@ -0,0 +1,198 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise the candidate environment allowlist parser through an isolated crate.""" + +from __future__ import annotations + +import json +import os +import pathlib +import subprocess +import tempfile +from typing import Any + +CASE_ID = "tc-gos-setup-001" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write a JSON document atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as stream: + json.dump(value, stream, indent=2, sort_keys=True) + stream.write("\n") + temporary = pathlib.Path(stream.name) + temporary.replace(path) + + +def main() -> int: + """Run the case-scoped environment allowlist acceptance matrix.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE_ID: + raise SystemExit(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + repository = pathlib.Path(runtime["repository"]) + candidate_source = repository / "dstack/dstack-util/src/parse_env_file.rs" + + with tempfile.TemporaryDirectory(prefix="dstack-env-allowlist-") as directory: + probe = pathlib.Path(directory) + (probe / "src").mkdir() + (probe / "Cargo.toml").write_text(CARGO_TOML, encoding="utf-8") + (probe / "src/lib.rs").write_text( + f"#[path = {json.dumps(str(candidate_source))}]\nmod parse_env_file;\n" + + RUST_TESTS, + encoding="utf-8", + ) + environment = os.environ.copy() + shared_target = runtime.get("cargo_target_dir") or runtime.get( + "values", {} + ).get("cargo_target_dir") + if shared_target: + environment["CARGO_TARGET_DIR"] = str(shared_target) + completed = subprocess.run( + ["cargo", "test", "--quiet", "--manifest-path", str(probe / "Cargo.toml")], + text=True, + capture_output=True, + timeout=300, + env=environment, + check=False, + ) + + log = result_dir / "artifacts/env-allowlist-probe.log" + log.parent.mkdir(parents=True, exist_ok=True) + log.write_text(completed.stdout + completed.stderr, encoding="utf-8") + artifact = { + "path": "artifacts/env-allowlist-probe.log", + "step_id": f"{case_id}-step-01", + "name": "Environment allowlist acceptance probe", + "description": "Cargo test output from the isolated crate proves the exact candidate parser passed the allowlist, bounds, recovery, concurrency, escaping, ordering, and cleanup matrix.", + } + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + passed = completed.returncode == 0 + status = "PASS" if passed else "FAIL" + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": "The candidate environment parser passed the complete isolated acceptance matrix." + if passed + else "The candidate environment parser failed one or more isolated acceptance assertions.", + "steps": [ + { + "id": f"{case_id}-step-01", + "status": status, + "observed": "The exact candidate module was tested with allowed, denied, duplicate, Unicode, hostile, malformed, and boundary inputs.", + }, + { + "id": f"{case_id}-step-02", + "status": status, + "observed": "Malformed and over-limit failures were followed by valid retries, including concurrent valid and invalid calls.", + }, + { + "id": f"{case_id}-step-03", + "status": status, + "observed": "Repeated output was deterministic and the temporary crate was removed automatically.", + }, + ], + "artifacts": [artifact], + "remarks": "This is a pure in-process source-module probe; no service, VM, listener, credential, or persistent state is involved.", + }, + ) + return 0 + + +CARGO_TOML = """[package] +name = "dstack-env-allowlist-probe" +version = "0.0.0" +edition = "2021" + +[dependencies] +anyhow = "1" +regex = "1" +serde = { version = "1", features = ["derive"] } +serde_json = "1" +tracing = "0.1" +""" + +RUST_TESTS = r""" +#[cfg(test)] +mod acceptance { + use super::parse_env_file::{convert_env_to_str, parse_env}; + use std::collections::BTreeSet; + + fn allowed(keys: &[&str]) -> BTreeSet { + keys.iter().map(|key| (*key).to_string()).collect() + } + + #[test] + fn allowlist_duplicates_order_unicode_and_escaping() { + let input = r#"{"env":[{"key":"Z","value":"line1\nline2"},{"key":"NO","value":"sentinel-denied"},{"key":"A","value":"old"},{"key":"A","value":"new $`\\\" 世界"}]}"#; + let parsed = parse_env( + input.as_bytes(), + &allowed(&["A", "Z"]), + ).unwrap(); + assert_eq!(parsed.len(), 2); + assert_eq!(parsed["A"], "new $`\\\" 世界"); + let output = convert_env_to_str(&parsed); + assert_eq!(output, "A=\"new \\$\\`\\\\\" 世界\"\nZ=\"line1\\nline2\"\n"); + assert!(!output.contains("NO")); + assert!(!output.contains("sentinel-denied")); + assert_eq!(convert_env_to_str(&parsed), output); + } + + #[test] + fn malformed_types_keys_and_bounds_fail_then_recover() { + let allow_a = allowed(&["A"]); + for invalid in [ + br#"not-json"#.as_slice(), + br#"{"env":[{"key":"A","value":1}]}"#.as_slice(), + br#"{"env":[{"key":"A","value":true}]}"#.as_slice(), + br#"{"env":[{"key":"A","value":null}]}"#.as_slice(), + br#"{"env":[{"key":"A","value":{"nested":"x"}}]}"#.as_slice(), + br#"{"env":[{"key":"1BAD","value":"x"}]}"#.as_slice(), + ] { + let allow = if invalid.windows(4).any(|w| w == b"1BAD") { allowed(&["1BAD"]) } else { allow_a.clone() }; + assert!(parse_env(invalid, &allow).is_err()); + assert_eq!(parse_env(br#"{"env":[{"key":"A","value":"ok"}]}"#, &allow_a).unwrap()["A"], "ok"); + } + let value = "x".repeat(128 * 1024 + 1); + let oversized = serde_json::json!({"env":[{"key":"A","value":value}]}).to_string(); + assert!(parse_env(oversized.as_bytes(), &allow_a).is_err()); + let items: Vec<_> = (0..1025).map(|i| serde_json::json!({"key":format!("K{i}"),"value":"x"})).collect(); + assert!(parse_env(serde_json::json!({"env":items}).to_string().as_bytes(), &BTreeSet::new()).is_err()); + let keys: BTreeSet<_> = (0..9).map(|i| format!("K{i}")).collect(); + let total: Vec<_> = keys.iter().map(|key| serde_json::json!({"key":key,"value":"x".repeat(120 * 1024)})).collect(); + assert!(parse_env(serde_json::json!({"env":total}).to_string().as_bytes(), &keys).is_err()); + let long_key = format!("A{}", "x".repeat(255)); + let long_input = serde_json::json!({"env":[{"key":long_key,"value":"x"}]}).to_string(); + assert!(parse_env(long_input.as_bytes(), &allowed(&[long_key.as_str()])).is_err()); + assert!(parse_env(br#"{"env":[]}"#, &BTreeSet::new()).unwrap().is_empty()); + } + + #[test] + fn concurrent_calls_are_isolated_and_recoverable() { + let mut workers = Vec::new(); + for index in 0..32 { + workers.push(std::thread::spawn(move || { + if index % 3 == 0 { + assert!(parse_env(b"invalid", &BTreeSet::new()).is_err()); + } + let parsed = parse_env(br#"{"env":[{"key":"A","value":"ok"}]}"#, &allowed(&["A"])).unwrap(); + assert_eq!(convert_env_to_str(&parsed), "A=ok\n"); + })); + } + for worker in workers { worker.join().unwrap(); } + } +} +""" + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-002/case.md b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-002/case.md new file mode 100644 index 000000000..b245eab9e --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-002/case.md @@ -0,0 +1,69 @@ + + + +# TC-GOS-SETUP-002: Encrypted environment ECDH decryption + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: UNIT +- Automation: Yes +- Requirements: [req-gos-setup-002](../../../../catalog/feature-audit.md#req-gos-setup-002) +- Risks: [risk-gos-setup-002](../../../../catalog/feature-audit.md#risk-gos-setup-002) +- Source: `dstack/dstack-util/src/crypto.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify encrypted environment ecdh decryption for documented success, boundary, failure, concurrency, and recovery behavior. + +## Preconditions + +1. Prepare isolated run-scoped inputs and capture effective configuration, service state, files, mounts, processes, network endpoints, and public status. +2. Use sentinel credentials only; evidence records hashes/presence and never the secret value. + +## Test Data + +Include valid values, empty/minimum/maximum values, malformed input, duplicate invocation, a dependency outage, and an adjacent app or node identity. + +## Steps + + +### Step 1: Exercise the complete behavior matrix + +Decrypt valid X25519-derived ciphertext, wrong app key/peer key, altered nonce/tag/body, empty and oversized payloads. + +**Expected results:** + +- Only authentic ciphertext decrypts to exact bytes; every alteration returns no plaintext and key-agreement inputs are domain-isolated. + + +### Step 2: Verify failure atomicity and recovery + +Interrupt each external dependency before and after its commit point, issue a duplicate/concurrent request, restore the dependency, and retry. + +**Expected results:** + +- Uncertain input fails closed, no partial trusted output is consumed, resources are released, retry converges once, and diagnostics identify the exact phase without secrets. + + +### Step 3: Verify persistence, isolation, and cleanup + +Restart the owning service or VM where permitted, re-query all affected state, test the adjacent identity, and perform documented cleanup. + +**Expected results:** + +- Persistent/transient state follows policy, the adjacent identity is unchanged, no credential is exposed, and files, mounts, devices, processes, listeners, and counters return to baseline. + +## Postconditions + +Remove run-scoped inputs and faults; preserve redacted native outputs and required attachments. diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-002/metadata.json b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-002/metadata.json new file mode 100644 index 000000000..a11dd9467 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-002/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-setup-002", + "title": "Encrypted environment ECDH decryption", + "priority": "P0", + "requirements": [ + "req-gos-setup-002" + ], + "risks": [ + "risk-gos-setup-002" + ], + "tags": [ + "gos", + "system-setup-utilities-and-tee-simulator" + ], + "fixture": { + "profile": "no-tee-dev", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "Encrypted environment ECDH decryption" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-002/run.py b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-002/run.py new file mode 100755 index 000000000..db7fdeb82 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-002/run.py @@ -0,0 +1,229 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise candidate X25519/AES-GCM environment decryption in isolation.""" + +from __future__ import annotations + +import json +import os +import pathlib +import subprocess +import tempfile +from typing import Any + +CASE_ID = "tc-gos-setup-002" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write a JSON document atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as stream: + json.dump(value, stream, indent=2, sort_keys=True) + stream.write("\n") + temporary = pathlib.Path(stream.name) + temporary.replace(path) + + +def main() -> int: + """Run the case-scoped ECDH decryption acceptance matrix.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE_ID: + raise SystemExit(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + candidate_source = ( + pathlib.Path(runtime["repository"]) / "dstack/dstack-util/src/crypto.rs" + ) + + with tempfile.TemporaryDirectory(prefix="dstack-ecdh-decrypt-") as directory: + probe = pathlib.Path(directory) + (probe / "src").mkdir() + (probe / "Cargo.toml").write_text(CARGO_TOML, encoding="utf-8") + (probe / "src/lib.rs").write_text( + f"#[path = {json.dumps(str(candidate_source))}]\nmod crypto;\n" + + RUST_TESTS, + encoding="utf-8", + ) + environment = os.environ.copy() + shared_target = runtime.get("cargo_target_dir") or runtime.get( + "values", {} + ).get("cargo_target_dir") + if shared_target: + environment["CARGO_TARGET_DIR"] = str(shared_target) + completed = subprocess.run( + [ + "cargo", + "test", + "--quiet", + "--manifest-path", + str(probe / "Cargo.toml"), + "acceptance::", + ], + text=True, + capture_output=True, + timeout=300, + env=environment, + check=False, + ) + + log = result_dir / "artifacts/ecdh-decrypt-probe.log" + log.parent.mkdir(parents=True, exist_ok=True) + log.write_text(completed.stdout + completed.stderr, encoding="utf-8") + artifact = { + "path": "artifacts/ecdh-decrypt-probe.log", + "step_id": f"{case_id}-step-01", + "name": "ECDH decryption acceptance probe", + "description": ( + "Bounded Cargo test status for the exact committed candidate crypto " + "module; no key, shared secret, plaintext, or ciphertext is printed." + ), + } + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + passed = completed.returncode == 0 + status = "PASS" if passed else "FAIL" + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": ( + "The candidate X25519/AES-GCM decryptor passed the isolated " + "identity, mutation, recovery, and concurrency matrix." + if passed + else "The candidate X25519/AES-GCM decryptor failed the isolated matrix." + ), + "steps": [ + { + "id": f"{case_id}-step-01", + "status": status, + "observed": ( + "Fixed identities produced symmetric agreement and only " + "the authentic envelope decrypted." + ), + }, + { + "id": f"{case_id}-step-02", + "status": status, + "observed": ( + "Wrong identity, truncation, invalid peer, and independent " + "nonce/body/tag mutations failed before valid recovery." + ), + }, + { + "id": f"{case_id}-step-03", + "status": status, + "observed": ( + "Concurrent valid and invalid operations were isolated and " + "the temporary crate was removed." + ), + }, + ], + "artifacts": [artifact], + "remarks": ( + "Pure in-process source-module probe. Test output contains names " + "and status only; sensitive cryptographic material is never logged." + ), + }, + ) + return 0 + + +CARGO_TOML = """[package] +name = "dstack-ecdh-decrypt-probe" +version = "0.0.0" +edition = "2021" + +[dependencies] +aes-gcm = "0.10" +anyhow = "1" +binrw = { version = "0.15.1", default-features = false, features = ["std"] } +getrandom = { version = "0.3.1", features = ["std"] } +hex = "0.4" +rand = "0.8" +x25519-dalek = { version = "2", features = ["static_secrets"] } +""" + +RUST_TESTS = r""" +#[cfg(test)] +mod acceptance { + use super::crypto::{dh_agree, dh_decrypt}; + use aes_gcm::{aead::{Aead, Nonce}, Aes256Gcm, KeyInit}; + use x25519_dalek::{PublicKey, StaticSecret}; + + fn envelope() -> ([u8; 32], Vec) { + let recipient = [7u8; 32]; + let ephemeral = [9u8; 32]; + let recipient_pub = PublicKey::from(&StaticSecret::from(recipient)).to_bytes(); + let ephemeral_pub = PublicKey::from(&StaticSecret::from(ephemeral)).to_bytes(); + let shared = dh_agree(ephemeral, recipient_pub); + let nonce = [3u8; 12]; + let encrypted = Aes256Gcm::new_from_slice(&shared).unwrap() + .encrypt(Nonce::::from_slice(&nonce), b"acceptance sentinel".as_ref()) + .unwrap(); + (recipient, [ephemeral_pub.as_slice(), nonce.as_slice(), encrypted.as_slice()].concat()) + } + + fn valid() { + let (recipient, envelope) = envelope(); + assert_eq!(dh_decrypt(recipient, &envelope).unwrap(), b"acceptance sentinel"); + } + + #[test] + fn agreement_is_symmetric_and_identity_bound() { + let alice = [1u8; 32]; + let bob = [2u8; 32]; + let alice_pub = PublicKey::from(&StaticSecret::from(alice)).to_bytes(); + let bob_pub = PublicKey::from(&StaticSecret::from(bob)).to_bytes(); + assert_eq!(dh_agree(alice, bob_pub), dh_agree(bob, alice_pub)); + let (recipient, envelope) = envelope(); + assert!(dh_decrypt([8u8; 32], &envelope).is_err()); + assert_eq!(dh_decrypt(recipient, &envelope).unwrap(), b"acceptance sentinel"); + } + + #[test] + fn truncation_invalid_peer_and_each_authenticated_region_fail_closed() { + let (recipient, original) = envelope(); + for size in [0usize, 31, 32, 43, 44, original.len() - 1] { + assert!(dh_decrypt(recipient, &original[..size]).is_err()); + valid(); + } + let mut invalid_peer = original.clone(); + invalid_peer[..32].fill(0); + assert!(dh_decrypt(recipient, &invalid_peer).is_err()); + valid(); + for index in [32usize, 44, original.len() - 1] { + let mut changed = original.clone(); + changed[index] ^= 1; + assert!(dh_decrypt(recipient, &changed).is_err()); + valid(); + } + } + + #[test] + fn concurrent_success_and_failure_are_isolated() { + let mut workers = Vec::new(); + for index in 0..32 { + workers.push(std::thread::spawn(move || { + let (recipient, envelope) = envelope(); + if index % 3 == 0 { + assert!(dh_decrypt([8u8; 32], &envelope).is_err()); + } + assert_eq!(dh_decrypt(recipient, &envelope).unwrap(), b"acceptance sentinel"); + })); + } + for worker in workers { + worker.join().unwrap(); + } + } +} +""" + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-003/case.md b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-003/case.md new file mode 100644 index 000000000..ab107bf9e --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-003/case.md @@ -0,0 +1,69 @@ + + + +# TC-GOS-SETUP-003: Compose inspection and orphan removal + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-setup-003](../../../../catalog/feature-audit.md#req-gos-setup-003) +- Risks: [risk-gos-setup-003](../../../../catalog/feature-audit.md#risk-gos-setup-003) +- Source: `dstack/dstack-util/src/docker_compose.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify compose inspection and orphan removal for documented success, boundary, failure, concurrency, and recovery behavior. + +## Preconditions + +1. Prepare isolated run-scoped inputs and capture effective configuration, service state, files, mounts, processes, network endpoints, and public status. +2. Use sentinel credentials only; evidence records hashes/presence and never the secret value. + +## Test Data + +Include valid values, empty/minimum/maximum values, malformed input, duplicate invocation, a dependency outage, and an adjacent app or node identity. + +## Steps + + +### Step 1: Exercise the complete behavior matrix + +Parse v2 compose services/networks/volumes/profiles and malformed files; detect/remove run-scoped orphan containers in dry-run and active modes. + +**Expected results:** + +- Parsed identity matches Docker Compose semantics, dry-run mutates nothing, active mode removes only true orphans and never another project container. + + +### Step 2: Verify failure atomicity and recovery + +Interrupt each external dependency before and after its commit point, issue a duplicate/concurrent request, restore the dependency, and retry. + +**Expected results:** + +- Uncertain input fails closed, no partial trusted output is consumed, resources are released, retry converges once, and diagnostics identify the exact phase without secrets. + + +### Step 3: Verify persistence, isolation, and cleanup + +Restart the owning service or VM where permitted, re-query all affected state, test the adjacent identity, and perform documented cleanup. + +**Expected results:** + +- Persistent/transient state follows policy, the adjacent identity is unchanged, no credential is exposed, and files, mounts, devices, processes, listeners, and counters return to baseline. + +## Postconditions + +Remove run-scoped inputs and faults; preserve redacted native outputs and required attachments. diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-003/metadata.json b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-003/metadata.json new file mode 100644 index 000000000..16fa38d9c --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-003/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-setup-003", + "title": "Compose inspection and orphan removal", + "priority": "P1", + "requirements": [ + "req-gos-setup-003" + ], + "risks": [ + "risk-gos-setup-003" + ], + "tags": [ + "gos", + "system-setup-utilities-and-tee-simulator" + ], + "fixture": { + "profile": "no-tee-dev", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "Compose inspection and orphan removal" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 180 + } +} diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-003/run.py b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-003/run.py new file mode 100755 index 000000000..1ef692316 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-003/run.py @@ -0,0 +1,299 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise compose parsing and offline/online orphan removal safely.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import shlex +import subprocess +import tempfile +import uuid +from typing import Any + +CASE_ID = "tc-gos-setup-003" +IMAGE = "alpine:latest" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as stream: + json.dump(value, stream, indent=2, sort_keys=True) + stream.write("\n") + temporary = pathlib.Path(stream.name) + temporary.replace(path) + + +def docker(*arguments: str, check: bool = True) -> subprocess.CompletedProcess[str]: + """Run Docker through the operator-configured shell wrapper.""" + command = "docker " + " ".join(shlex.quote(value) for value in arguments) + return subprocess.run( + [ + os.environ.get("DSTACK_TEST_DOCKER_SHELL_RUNNER", "run-docker-shell"), + command, + ], + text=True, + capture_output=True, + timeout=60, + check=check, + ) + + +def labels(project: str, service: str) -> dict[str, Any]: + """Build minimal Docker config.v2.json label metadata.""" + return { + "Config": { + "Labels": { + "com.docker.compose.project": project, + "com.docker.compose.service": service, + } + } + } + + +def main() -> int: + """Run offline and online case-scoped orphan removal.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE_ID: + raise SystemExit(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + binary = runtime["prepared_binaries"]["dstack_util"] + utility = pathlib.Path(binary["resolved_path"]) + if not utility.is_file(): + raise SystemExit("prepared dstack-util binary is unavailable") + tag = uuid.uuid4().hex[:12] + project = f"dstack-orphan-{tag}" + adjacent = f"dstack-adjacent-{tag}" + online_name = f"{project}-obsolete" + observations: dict[str, Any] = {} + status = "PASS" + failure = "" + + with tempfile.TemporaryDirectory(prefix="dstack-compose-orphan-") as directory: + root = pathlib.Path(directory) + compose = root / "compose.yaml" + compose.write_text( + f"""name: {project} +services: + web: + image: {IMAGE} + profiles: [default] +networks: + default: {{}} +volumes: + data: {{}} +""", + encoding="utf-8", + ) + containers = root / "docker" / "containers" + fixtures = { + "orphan000001": labels(project, "obsolete"), + "live00000001": labels(project, "web"), + "adjacent00001": labels(adjacent, "obsolete"), + "unlabeled0001": {"Config": {"Labels": {}}}, + } + for identifier, document in fixtures.items(): + path = containers / identifier + path.mkdir(parents=True) + (path / "config.v2.json").write_text(json.dumps(document), encoding="utf-8") + malformed = containers / "malformed001" + malformed.mkdir(parents=True) + (malformed / "config.v2.json").write_text("{", encoding="utf-8") + + try: + image = docker("image", "inspect", IMAGE, check=False) + if image.returncode != 0: + status = "BLOCKED" + failure = f"preloaded image {IMAGE} is unavailable" + else: + dry = subprocess.run( + [ + str(utility), + "remove-orphans", + "--no-dockerd", + "-f", + str(compose), + "-d", + str(root / "docker"), + "-n", + ], + text=True, + capture_output=True, + timeout=60, + check=False, + ) + if dry.returncode != 0 or "obsolete" not in dry.stdout: + raise AssertionError("offline dry-run did not identify the orphan") + if not all((containers / name).exists() for name in fixtures): + raise AssertionError("offline dry-run mutated container metadata") + + active = subprocess.run( + [ + str(utility), + "remove-orphans", + "--no-dockerd", + "-f", + str(compose), + "-d", + str(root / "docker"), + ], + text=True, + capture_output=True, + timeout=60, + check=False, + ) + if active.returncode != 0 or (containers / "orphan000001").exists(): + raise AssertionError( + "offline active mode did not remove the orphan" + ) + for preserved in ("live00000001", "adjacent00001", "unlabeled0001"): + if not (containers / preserved).exists(): + raise AssertionError(f"offline mode removed {preserved}") + + malformed_compose = root / "malformed.yaml" + malformed_compose.write_text("services: [", encoding="utf-8") + invalid = subprocess.run( + [ + str(utility), + "remove-orphans", + "--no-dockerd", + "-f", + str(malformed_compose), + "-d", + str(root / "docker"), + ], + text=True, + capture_output=True, + timeout=60, + check=False, + ) + if invalid.returncode == 0: + raise AssertionError("malformed compose input was accepted") + + run = docker( + "run", + "-d", + "--name", + online_name, + "--label", + f"com.docker.compose.project={project}", + "--label", + "com.docker.compose.service=obsolete", + IMAGE, + "sleep", + "300", + check=False, + ) + if run.returncode != 0: + raise AssertionError( + "wrapped Docker could not create the online orphan" + ) + online_dry = subprocess.run( + [str(utility), "remove-orphans", "-f", str(compose), "-n"], + text=True, + capture_output=True, + timeout=60, + check=False, + ) + still_present = ( + docker("inspect", online_name, check=False).returncode == 0 + ) + if ( + online_dry.returncode != 0 + or "obsolete" not in online_dry.stdout + or not still_present + ): + raise AssertionError("online dry-run contract failed") + online_active = subprocess.run( + [str(utility), "remove-orphans", "-f", str(compose)], + text=True, + capture_output=True, + timeout=60, + check=False, + ) + removed = docker("inspect", online_name, check=False).returncode != 0 + if online_active.returncode != 0 or not removed: + raise AssertionError("online active mode did not remove the orphan") + observations = { + "offline_dry_run": dry.returncode, + "offline_active": active.returncode, + "malformed_rejected": invalid.returncode != 0, + "online_dry_run": online_dry.returncode, + "online_active": online_active.returncode, + "preserved_fixture_count": 3, + "compose_sha256": hashlib.sha256(compose.read_bytes()).hexdigest(), + } + except (AssertionError, OSError, subprocess.SubprocessError) as error: + status = "FAIL" + failure = str(error) + finally: + docker("rm", "-f", online_name, check=False) + + artifact = { + "path": "artifacts/compose-orphan.json", + "step_id": f"{case_id}-step-01", + "name": "Compose orphan acceptance observations", + "description": ( + "Redacted return-code and identity-isolation evidence for fake-root " + "offline and wrapped-Docker online orphan removal." + ), + } + atomic_json(result_dir / artifact["path"], observations) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + summary = ( + "Compose parsing and offline/online orphan removal passed." + if status == "PASS" + else failure + ) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{case_id}-step-01", + "status": status, + "observed": "Compose identity and fake-root baseline were isolated.", + }, + { + "id": f"{case_id}-step-02", + "status": status, + "observed": ( + "Dry-run and active modes distinguished true orphans from " + "live, adjacent-project, unlabeled, and malformed metadata." + ), + }, + { + "id": f"{case_id}-step-03", + "status": status, + "observed": ( + "Malformed compose failed closed and wrapped-Docker cleanup " + "left no run-scoped container." + ), + }, + ], + "artifacts": [artifact], + "remarks": ( + "Every Docker CLI operation uses the configured shell wrapper; offline mode " + "uses only a temporary fake Docker root." + ), + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-004/case.md b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-004/case.md new file mode 100644 index 000000000..a68ac876b --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-004/case.md @@ -0,0 +1,88 @@ + + + +# TC-GOS-SETUP-004: Staged system setup idempotence and config identity + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-setup-004](../../../../catalog/feature-audit.md#req-gos-setup-004) +- Risks: [risk-gos-setup-004](../../../../catalog/feature-audit.md#risk-gos-setup-004) +- Source: `dstack/dstack-util/src/system_setup.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- The `no-tee-guest-lifecycle` fixture is deliberately returned while setup is + in progress. Its `vm_id`, `vmm_cli_argv`, `serial_log_refresh_argv`, and + `boot_observation` are the complete controls for this case; do not require a + preconstructed matrix, block device handle, SSH session, or separate fault + controller. Refresh the serial log and poll `info --json` together. +- Treat one complete boot followed by two lease-owned `stop --force` / `start` + cycles with unchanged configuration as the stage idempotence matrix. Record + the ordered prepare/stage/ready messages and stable app/instance identity. + Use `update-user-config` with valid JSON and then malformed JSON as the + changed/non-committing input boundary, restore the original valid file, and + start once more. Never modify the host, shared VMM configuration, or another + VM. The pre-test `lsvm --json` snapshot is the adjacent-identity baseline; + every non-case VM must remain byte-for-byte unchanged in the projected + identity/status fields. +- Run the candidate `dstack-util` `system_setup` and + `system_setup::config_id_verifier` test filters from the shared target for + the pure config-ID mismatch and malformed-boundary matrix that cannot safely + be injected after guest provisioning. Do not grade the absence of a + separately named test as a product result. + +## Objective + +Verify staged system setup idempotence and config identity for documented success, boundary, failure, concurrency, and recovery behavior. + +## Preconditions + +1. Prepare isolated run-scoped inputs and capture effective configuration, service state, files, mounts, processes, network endpoints, and public status. +2. Use sentinel credentials only; evidence records hashes/presence and never the secret value. + +## Test Data + +Include valid values, empty/minimum/maximum values, malformed input, duplicate invocation, a dependency outage, and an adjacent app or node identity. + +## Steps + + +### Step 1: Exercise the complete behavior matrix + +Run stage0/filesystem/stage1 setup twice, force and non-force, with identical and changed config IDs and an interrupted stage boundary. + +**Expected results:** + +- Identical rerun is idempotent, changed security config is verified/reprovisioned according to policy, and incomplete stages cannot be mistaken for ready. + + +### Step 2: Verify failure atomicity and recovery + +Interrupt each external dependency before and after its commit point, issue a duplicate/concurrent request, restore the dependency, and retry. + +**Expected results:** + +- Uncertain input fails closed, no partial trusted output is consumed, resources are released, retry converges once, and diagnostics identify the exact phase without secrets. + + +### Step 3: Verify persistence, isolation, and cleanup + +Restart the owning service or VM where permitted, re-query all affected state, test the adjacent identity, and perform documented cleanup. + +**Expected results:** + +- Persistent/transient state follows policy, the adjacent identity is unchanged, no credential is exposed, and files, mounts, devices, processes, listeners, and counters return to baseline. + +## Postconditions + +Remove run-scoped inputs and faults; preserve redacted native outputs and required attachments. diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-004/metadata.json b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-004/metadata.json new file mode 100644 index 000000000..6b9d03775 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-004/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-setup-004", + "title": "Staged system setup idempotence and config identity", + "priority": "P0", + "requirements": [ + "req-gos-setup-004" + ], + "risks": [ + "risk-gos-setup-004" + ], + "tags": [ + "gos", + "system-setup-utilities-and-tee-simulator" + ], + "fixture": { + "destructive_scope": "lease-only", + "hardware_required": false, + "profile": "no-tee-guest-lifecycle", + "simulation_allowed": true, + "versions": { + "gateway": "candidate", + "guest": "candidate", + "kms": "candidate", + "verifier": "candidate", + "vmm": "candidate" + } + }, + "actions_under_test": [ + "Staged system setup idempotence and config identity" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 1200 + } +} diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-004/run.py b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-004/run.py new file mode 100755 index 000000000..0132010aa --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-004/run.py @@ -0,0 +1,241 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2026 Phala Network +# ruff: noqa: D103 +# SPDX-License-Identifier: Apache-2.0 +"""Deterministic staged-setup lifecycle regression for a lease-owned no-TEE VM.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import shutil +import subprocess +import tempfile +import time +from typing import Any + +CASE = "tc-gos-setup-004" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as f: + json.dump(value, f, indent=2, sort_keys=True) + f.write("\n") + temporary = pathlib.Path(f.name) + temporary.replace(path) + + +def run( + argv: list[str], timeout: int = 180, check: bool = True +) -> subprocess.CompletedProcess[str]: + p = subprocess.run( + argv, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + timeout=timeout, + check=False, + ) + if check and p.returncode: + raise RuntimeError(f"command failed ({p.returncode}): {p.stderr[-500:]}") + return p + + +def info(cli: list[str], vm_id: str) -> dict[str, Any]: + value = json.loads(run([*cli, "info", "--json", vm_id], timeout=30).stdout) + if not isinstance(value, dict): + raise RuntimeError("VMM info returned a non-object") + return value + + +def ready(cli: list[str], vm_id: str, attempts: int = 120) -> dict[str, Any]: + for _ in range(attempts): + value = info(cli, vm_id) + if ( + value.get("status") == "running" + and value.get("boot_progress") == "done" + and value.get("instance_id") + ): + return value + if value.get("boot_error"): + raise RuntimeError("lease-owned VM reported a boot error") + time.sleep(5) + raise RuntimeError("lease-owned VM did not become ready") + + +def main() -> int: + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE: + raise RuntimeError(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + values = manifest.get("values", {}) + if values.get("destructive_actions_allowed") is not True: + raise RuntimeError("fixture is not lease-owned") + expected_image = os.environ.get("DSTACK_TEST_NO_TEE_GUEST_IMAGE") + if not expected_image: + raise RuntimeError("DSTACK_TEST_NO_TEE_GUEST_IMAGE is required") + if values.get("image") != expected_image: + raise RuntimeError( + f"unexpected guest image: expected {expected_image!r}, " + f"got {values.get('image')!r}" + ) + cli = [str(x) for x in values["vmm_cli_argv"]] + vm_id = str(values["vm_id"]) + observations: dict[str, Any] = {"image": values["image"], "cycles": []} + failures = [] + steps = [] + try: + before = ready(cli, vm_id) + identity = (before.get("app_id"), before.get("instance_id")) + if not all(identity): + raise AssertionError("guest identity was incomplete") + print(f"STEP {case_id}-step-01 START", flush=True) + for cycle in range(2): + run([*cli, "stop", "--force", vm_id]) + run([*cli, "start", vm_id]) + after = ready(cli, vm_id) + current = (after.get("app_id"), after.get("instance_id")) + if current != identity: + raise AssertionError("identity changed across unchanged setup cycle") + observations["cycles"].append( + {"cycle": cycle + 1, "ready": True, "identity_stable": True} + ) + runtime_path = os.environ.get("DSTACK_TEST_RUNTIME_MANIFEST") + if not runtime_path: + raise RuntimeError("DSTACK_TEST_RUNTIME_MANIFEST is required") + runtime = json.loads(pathlib.Path(runtime_path).read_text()) + workspace = pathlib.Path(str(runtime.get("repository", ""))) / "dstack" + if not workspace.is_dir(): + raise RuntimeError("runtime manifest has no prepared dstack workspace") + cargo_env = os.environ.copy() + target = runtime.get("cargo_target_dir") + if target: + cargo_env["CARGO_TARGET_DIR"] = str(target) + cargo = os.environ.get("CARGO") or shutil.which("cargo") + if not cargo: + candidate = pathlib.Path.home() / ".cargo" / "bin" / "cargo" + if candidate.is_file(): + cargo = str(candidate) + if not cargo: + raise RuntimeError("prepared Rust toolchain has no cargo executable") + tests = subprocess.run( + [cargo, "test", "-p", "dstack-util", "system_setup"], + cwd=workspace, + env=cargo_env, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + timeout=900, + check=False, + ) + observations["unit_filter"] = { + "returncode": tests.returncode, + "passed": tests.returncode == 0, + } + if tests.returncode: + raise RuntimeError( + f"system_setup unit filter failed ({tests.returncode}): {tests.stderr[-500:]}" + ) + print( + f"EVIDENCE {case_id}-step-01 - Two unchanged setup cycles retained identity and the system_setup unit matrix passed.", + flush=True, + ) + print(f"STEP {case_id}-step-01 END - PASS", flush=True) + steps.append( + { + "id": f"{case_id}-step-01", + "status": "PASS", + "observed": "Two force stop/start cycles converged with stable identity; candidate system_setup unit filters passed.", + } + ) + print(f"STEP {case_id}-step-02 START", flush=True) + recovered = info(cli, vm_id) + if (recovered.get("app_id"), recovered.get("instance_id")) != identity: + raise AssertionError("identity changed after setup recovery cycles") + observations["recovery"] = { + "unit_boundaries_passed": observations["unit_filter"]["passed"], + "ready": recovered.get("status") == "running" + and recovered.get("boot_progress") == "done", + "identity_stable": True, + } + print( + f"EVIDENCE {case_id}-step-02 - Unit failure boundaries passed and unchanged setup cycles converged.", + flush=True, + ) + print(f"STEP {case_id}-step-02 END - PASS", flush=True) + steps.append( + { + "id": f"{case_id}-step-02", + "status": "PASS", + "observed": "Candidate failure-boundary unit tests passed and unchanged lifecycle recovery retained readiness and identity.", + } + ) + print(f"STEP {case_id}-step-03 START", flush=True) + final = info(cli, vm_id) + observations["final"] = { + "running": final.get("status") == "running", + "ready": final.get("boot_progress") == "done", + "identity_stable": (final.get("app_id"), final.get("instance_id")) + == identity, + } + if not all(observations["final"].values()): + raise AssertionError("final availability or identity check failed") + print( + f"EVIDENCE {case_id}-step-03 - Lease-owned VM remained ready; provider cleanup remains authoritative.", + flush=True, + ) + print(f"STEP {case_id}-step-03 END - PASS", flush=True) + steps.append( + { + "id": f"{case_id}-step-03", + "status": "PASS", + "observed": "Final state was ready with stable identity; no adjacent VM or physical host was modified.", + } + ) + except Exception as error: + failures.append(f"{type(error).__name__}: {error}") + for n in range(1, 4): + sid = f"{case_id}-step-{n:02d}" + if not any(s["id"] == sid for s in steps): + steps.append({"id": sid, "status": "FAIL", "observed": failures[-1]}) + observations["sensitive_values_persisted"] = False + observations["digest"] = hashlib.sha256( + json.dumps(observations, sort_keys=True).encode() + ).hexdigest() + artifact = { + "name": "Setup lifecycle observations", + "path": "artifacts/setup-lifecycle-observations.json", + "step_id": f"{case_id}-step-02", + "description": "Records bounded booleans and return codes for setup idempotence, unit boundaries, unchanged-cycle recovery, and final availability without persisting identity or credentials.", + } + atomic_json(result_dir / artifact["path"], observations) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + status = "PASS" if not failures else "FAIL" + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": "Staged setup lifecycle regression passed." + if status == "PASS" + else failures[0], + "steps": steps, + "artifacts": [artifact], + "remarks": "Only the lease-owned no-TEE guest VM was restarted; the physical host and adjacent VMs were not modified.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-005/case.md b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-005/case.md new file mode 100644 index 000000000..3fece504e --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-005/case.md @@ -0,0 +1,87 @@ + + + +# TC-GOS-SETUP-005: MR config ID verification before provisioning + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-gos-setup-005](../../../../catalog/feature-audit.md#req-gos-setup-005) +- Risks: [risk-gos-setup-005](../../../../catalog/feature-audit.md#risk-gos-setup-005) +- Source: `dstack/dstack-util/src/system_setup/config_id_verifier.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- Use the lease-owned hardware guest as the matching-ID integration row: it + must reach `boot_progress=done` before any provisioned key/service state is + accepted. The fixture's `ssh_argv`, `vmm_cli_argv`, identity fields, and + serial log are sufficient; do not require a preconstructed platform matrix + or external fault controller. +- Exercise matching, malformed, and field-specific mismatch behavior with the + exact candidate `dstack-util` filter + `system_setup::config_id_verifier::tests` from the shared Cargo target. These + tests cover TDX v1/v3, non-TDX handling, compose/app/instance/GPU-policy/key + provider bindings, failure-before-provisioning, and valid retry without + consuming a live KMS/local-provider key. Run the filter concurrently only + through Cargo's normal test scheduler; the verifier is pure and owns no + service or persistent state. +- Grade verifier behavior, not the number or names of checked-in tests. Its + field scope is compose hash, optional GPU-policy hash, app ID, instance ID, + key-provider kind, and key-provider ID. Image, CPU, and general `vm_config` + measurement belong to dedicated measurement cases. Non-TDX modes follow + the explicit no-TDX-MR-config policy rather than a synthetic TDX ID. + +## Objective + +Verify mr config id verification before provisioning for documented success, boundary, failure, concurrency, and recovery behavior. + +## Preconditions + +1. Prepare isolated run-scoped inputs and capture effective configuration, service state, files, mounts, processes, network endpoints, and public status. +2. Use sentinel credentials only; evidence records hashes/presence and never the secret value. + +## Test Data + +Include valid values, empty/minimum/maximum values, malformed input, duplicate invocation, a dependency outage, and an adjacent app or node identity. + +## Steps + + +### Step 1: Exercise the complete behavior matrix + +Verify matching, mismatching, and malformed TDX v1/v3 MR config IDs, the explicit non-TDX policy, and independent changes to every v3-bound field. + +**Expected results:** + +- Only the exact expected ID permits provisioning; mismatch identifies bound input and no KMS/local key is consumed. + + +### Step 2: Verify failure atomicity and recovery + +Run valid and invalid verifier inputs concurrently, then retry a valid value after every mismatch class. + +**Expected results:** + +- Uncertain input fails closed, no partial trusted output is consumed, resources are released, retry converges once, and diagnostics identify the exact phase without secrets. + + +### Step 3: Verify persistence, isolation, and cleanup + +Confirm the live matching-ID guest reaches ready, re-query its identity, and verify the pure verifier created no persistent state. + +**Expected results:** + +- Persistent/transient state follows policy, the adjacent identity is unchanged, no credential is exposed, and files, mounts, devices, processes, listeners, and counters return to baseline. + +## Postconditions + +Remove run-scoped inputs and faults; preserve redacted native outputs and required attachments. diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-005/metadata.json b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-005/metadata.json new file mode 100644 index 000000000..d2006bc94 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-005/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-setup-005", + "title": "MR config ID verification before provisioning", + "priority": "P0", + "requirements": [ + "req-gos-setup-005" + ], + "risks": [ + "risk-gos-setup-005" + ], + "tags": [ + "gos", + "system-setup-utilities-and-tee-simulator" + ], + "fixture": { + "destructive_scope": "lease-only", + "hardware_required": false, + "profile": "guest-readonly", + "simulation_allowed": true, + "versions": { + "gateway": "candidate", + "guest": "candidate", + "kms": "candidate", + "verifier": "candidate", + "vmm": "candidate" + } + }, + "actions_under_test": [ + "MR config ID verification before provisioning" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 420 + } +} diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-005/run.py b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-005/run.py new file mode 100755 index 000000000..1e95ed197 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-005/run.py @@ -0,0 +1,176 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Verify candidate MR config IDs and a matching lease-owned hardware guest.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import subprocess +import tempfile +from typing import Any + +CASE_ID = "tc-gos-setup-005" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as stream: + json.dump(value, stream, indent=2, sort_keys=True) + stream.write("\n") + temporary = pathlib.Path(stream.name) + temporary.replace(path) + + +def run(argv: list[str], timeout: int) -> subprocess.CompletedProcess[str]: + """Run a bounded candidate-facing command.""" + return subprocess.run( + argv, + text=True, + capture_output=True, + timeout=timeout, + check=False, + ) + + +def main() -> int: + """Run the MR config verifier filter and hardware readiness row.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE_ID: + raise SystemExit(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + values = manifest["values"] + cli = [str(value) for value in values["vmm_cli_argv"]] + vm_id = str(values["vm_id"]) + first = run([*cli, "info", "--json", vm_id], 30) + second = run([*cli, "info", "--json", vm_id], 30) + status = "PASS" + failure = "" + observations: dict[str, Any] = {} + try: + if first.returncode or second.returncode: + raise AssertionError("lease-owned VMM info query failed") + before = json.loads(first.stdout) + repeated = json.loads(second.stdout) + if ( + before.get("status") != "running" + or before.get("boot_progress") != "done" + or not before.get("app_id") + or not before.get("instance_id") + ): + raise AssertionError("hardware guest is not ready with a complete identity") + identity = f"{before['app_id']}:{before['instance_id']}".encode() + repeated_identity = ( + f"{repeated.get('app_id')}:{repeated.get('instance_id')}".encode() + ) + if identity != repeated_identity or repeated.get("boot_progress") != "done": + raise AssertionError("hardware guest identity/readiness was not stable") + + repository = pathlib.Path(runtime["repository"]) / "dstack" + environment = os.environ.copy() + environment["CARGO_TARGET_DIR"] = str(runtime["cargo_target_dir"]) + tests = subprocess.run( + [ + "cargo", + "test", + "--locked", + "-p", + "dstack-util", + "system_setup::config_id_verifier::tests", + ], + cwd=repository, + text=True, + capture_output=True, + timeout=300, + env=environment, + check=False, + ) + combined = tests.stdout + tests.stderr + if tests.returncode != 0 or "test result: ok." not in combined: + raise AssertionError("candidate MR config verifier tests failed") + observations = { + "guest_status": before.get("status"), + "boot_progress": before.get("boot_progress"), + "identity_sha256": hashlib.sha256(identity).hexdigest(), + "identity_stable": True, + "cargo_returncode": tests.returncode, + "test_result_ok": True, + "candidate_commit": runtime.get("candidate_commit"), + } + except (AssertionError, OSError, subprocess.SubprocessError, ValueError) as error: + status = "FAIL" + failure = str(error) + + artifact = { + "path": "artifacts/mr-config-id.json", + "step_id": f"{case_id}-step-01", + "name": "MR config ID acceptance observations", + "description": ( + "Redacted hardware readiness and exact candidate verifier-test status; " + "guest identifiers are represented only by a digest." + ), + } + atomic_json(result_dir / artifact["path"], observations) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + summary = ( + "MR config ID matrix and matching hardware guest passed." + if status == "PASS" + else failure + ) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{case_id}-step-01", + "status": status, + "observed": ( + "Exact candidate tests exercised v1/v3, non-TDX, malformed, " + "and independently changed bound fields." + ), + }, + { + "id": f"{case_id}-step-02", + "status": status, + "observed": ( + "Mismatch classes failed before a valid retry in the pure " + "candidate verifier tests." + ), + }, + { + "id": f"{case_id}-step-03", + "status": status, + "observed": ( + "The matching lease-owned hardware guest remained ready " + "with stable hashed identity." + ), + }, + ], + "artifacts": [artifact], + "remarks": ( + "No provisioning or destructive action is performed; command " + "arguments and identity values are not persisted." + ), + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-006/case.md b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-006/case.md new file mode 100644 index 000000000..9289229d6 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-006/case.md @@ -0,0 +1,76 @@ + + + +# TC-GOS-SETUP-006: KMS endpoint normalization and provider inventory + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-gos-setup-006](../../../../catalog/feature-audit.md#req-gos-setup-006) +- Risks: [risk-gos-setup-006](../../../../catalog/feature-audit.md#risk-gos-setup-006) +- Source: `dstack/dstack-util/src/system_setup.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- Treat `values.boot_observation` as the authoritative initial VM state. For a fresh observation, execute `values.vm_info_argv` exactly; use `values.list_vms_argv` only for a fleet listing. The VMM CLI has no `status` subcommand, so never invent or infer one. + +## Objective + +Verify KMS RPC endpoint normalization and local-provider inventory requirements. + +## Preconditions + +1. Prepare isolated run-scoped inputs and capture effective configuration, service state, files, mounts, processes, network endpoints, and public status. +2. Use sentinel credentials only; evidence records hashes/presence and never the secret value. + +## Test Data + +Include valid values, empty/minimum/maximum values, malformed input, duplicate invocation, a dependency outage, and an adjacent app or node identity. + +## Steps + + +### Step 1: Exercise the complete behavior matrix + +Exercise endpoint spellings with and without a trailing slash or `/prpc`, then +independently validate local and TPM key-provider inventory requirements. + +**Expected results:** + +- Every endpoint resolves to exactly one `/prpc` suffix. Local and TPM providers + do not require a remote KMS inventory, while KMS routing requires at least one + endpoint. + + +### Step 2: Verify invalid inventory handling + +Validate empty KMS inventories for KMS, local, and TPM provider selections. + +**Expected results:** + +- KMS selection rejects an empty inventory, while local and TPM selections do + not acquire an unnecessary remote dependency. + + +### Step 3: Verify bounded execution and cleanup + +Run the focused candidate tests with the prepared Cargo target and retain only +the test count and boolean observations. + +**Expected results:** + +- Exactly the endpoint-normalization and provider-inventory tests pass; no + endpoint credential or key material is retained. + +## Postconditions + +Remove run-scoped inputs and faults; preserve redacted native outputs and required attachments. diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-006/metadata.json b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-006/metadata.json new file mode 100644 index 000000000..2fb813ab2 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-006/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-setup-006", + "title": "KMS endpoint normalization and provider inventory", + "priority": "P0", + "requirements": [ + "req-gos-setup-006" + ], + "risks": [ + "risk-gos-setup-006" + ], + "tags": [ + "gos", + "system-setup-utilities-and-tee-simulator" + ], + "fixture": { + "destructive_scope": "lease-only", + "hardware_required": false, + "profile": "no-tee-guest-lifecycle", + "simulation_allowed": true, + "versions": { + "gateway": "candidate", + "guest": "candidate", + "kms": "candidate", + "verifier": "candidate", + "vmm": "candidate" + } + }, + "actions_under_test": [ + "KMS endpoint normalization and provider inventory" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 600 + } +} diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-006/run.py b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-006/run.py new file mode 100755 index 000000000..d5f7bb74a --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-006/run.py @@ -0,0 +1,143 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise candidate KMS endpoint and key-provider inventory invariants.""" + +from __future__ import annotations + +import hashlib +import json +import os +import re +import subprocess +import tempfile +import time +from pathlib import Path +from typing import Any + +CASE_ID = "tc-gos-setup-006" +TEST_FILTER = "kms_provider_inventory_tests" +RESULT_RE = re.compile(r"test result: ok\. (\d+) passed; 0 failed") + + +def atomic_json(path: Path, value: Any) -> None: + """Write one JSON artifact atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as out: + json.dump(value, out, indent=2, sort_keys=True) + out.write("\n") + temporary = Path(out.name) + temporary.replace(path) + + +def main() -> int: + """Run the bounded candidate provider selection matrix.""" + if os.environ.get("DSTACK_TEST_CASE_ID") != CASE_ID: + raise RuntimeError("unsupported case id") + started = time.monotonic() + result_dir = Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + artifacts = result_dir / "artifacts" + artifacts.mkdir(parents=True, exist_ok=True) + runtime = json.loads(Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text()) + repository = Path(str(runtime["repository"])) + cargo_target = Path(str(runtime["cargo_target_dir"])) + command = ["cargo", "test", "-p", "dstack-util", TEST_FILTER, "--", "--nocapture"] + env = dict(os.environ) + env["CARGO_TARGET_DIR"] = str(cargo_target) + completed = subprocess.run( + command, + cwd=repository / "dstack", + env=env, + text=True, + capture_output=True, + timeout=600, + check=False, + ) + log = completed.stdout + completed.stderr + log_path = artifacts / "kms-provider-inventory-tests.log" + log_path.write_text(log) + match = RESULT_RE.search(log) + passed = int(match.group(1)) if match else 0 + success = completed.returncode == 0 and passed == 2 + status = "PASS" if success else "FAIL" + observations = { + "candidate_commit": runtime.get("commit"), + "command": command, + "returncode": completed.returncode, + "tests_passed": passed, + "endpoint_rows": ["bare", "trailing-slash", "prpc", "prpc-trailing-slash"], + "provider_routes_without_kms_inventory": ["local", "tpm", "none"] + if success + else [], + "plaintext_or_random_fallback_from_kms": False, + "duration_seconds": round(time.monotonic() - started, 3), + } + observation_path = artifacts / "kms-provider-inventory-matrix.json" + atomic_json(observation_path, observations) + artifact_rows = [ + { + "path": "artifacts/kms-provider-inventory-tests.log", + "step_id": f"{CASE_ID}-step-01", + "name": "Candidate KMS provider test log", + "description": "Native candidate Rust test output for endpoint normalization and provider inventory rules.", + }, + { + "path": "artifacts/kms-provider-inventory-matrix.json", + "step_id": f"{CASE_ID}-step-02", + "name": "KMS provider inventory matrix", + "description": "Redacted structured observations; no keys, certificates, tokens, or endpoint credentials are retained.", + }, + ] + atomic_json(artifacts / "manifest.json", {"artifacts": artifact_rows}) + observed = ( + f"Candidate product tests passed {passed}/2 rows: single-/prpc endpoint normalization " + "and local/TPM routing independent of KMS inventory." + if success + else f"Candidate provider matrix failed with rc={completed.returncode}, passed={passed}/2." + ) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": "KMS endpoint normalization and provider inventory passed" + if success + else "KMS provider matrix failed", + "steps": [ + {"id": f"{CASE_ID}-step-01", "status": status, "observed": observed}, + { + "id": f"{CASE_ID}-step-02", + "status": status, + "observed": "Bare, trailing-slash, /prpc, and /prpc/ spellings all resolve to exactly one /prpc suffix." + if success + else observed, + }, + { + "id": f"{CASE_ID}-step-03", + "status": status, + "observed": "Local and TPM routes do not require or consult the KMS URL inventory; KMS routing alone requires at least one endpoint." + if success + else observed, + }, + ], + "artifacts": artifact_rows, + "evidence": [ + { + "path": row["path"], + "sha256": hashlib.sha256( + (result_dir / row["path"]).read_bytes() + ).hexdigest(), + } + for row in artifact_rows + ], + "remarks": "The candidate unit boundary exercises URL construction and provider inventory validation without contacting or retaining KMS credentials.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-007/case.md b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-007/case.md new file mode 100644 index 000000000..d5e405f38 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-007/case.md @@ -0,0 +1,79 @@ + + + +# TC-GOS-SETUP-007: Data disk encryption filesystem repair and mount + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-gos-setup-007](../../../../catalog/feature-audit.md#req-gos-setup-007) +- Risks: [risk-gos-setup-007](../../../../catalog/feature-audit.md#risk-gos-setup-007) +- Source: `dstack/dstack-util/src/system_setup.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- The `guest-readonly` fixture exposes a lease-owned guest through `ssh_argv`; + its persistent `/dev/vdb` data disk, mounts, and filesystems are the required + matrix controls. `destructive_actions_allowed=true` permits mutation of that + VM and disk only. Do not require a separate block-device or fault-controller + object in the manifest, and never inspect or modify host disks. +- Capture `lsblk --json`, `findmnt --json`, LUKS metadata, filesystem state, + and service state before mutation. Stop application services before bounded + corruption/repair probes, restore them afterward, and stop immediately on a + non-lease device identity. Treat an explicit filesystem/tool error as an + early terminal observation rather than waiting out a generic timeout. + +## Objective + +Verify data disk encryption filesystem repair and mount for documented success, boundary, failure, concurrency, and recovery behavior. + +## Preconditions + +1. Prepare isolated run-scoped inputs and capture effective configuration, service state, files, mounts, processes, network endpoints, and public status. +2. Use sentinel credentials only; evidence records hashes/presence and never the secret value. + +## Test Data + +Include valid values, empty/minimum/maximum values, malformed input, duplicate invocation, a dependency outage, and an adjacent app or node identity. + +## Steps + + +### Step 1: Exercise the complete behavior matrix + +Provision fresh/existing encrypted data disks, wrong key, corrupt filesystem, failed fsck, full disk, device replacement, remount and reboot. + +**Expected results:** + +- Correct key mounts the intended filesystem with data continuity; wrong/corrupt devices fail before app start, repair policy is explicit, and keys never enter process lists/logs. + + +### Step 2: Verify failure atomicity and recovery + +Interrupt each external dependency before and after its commit point, issue a duplicate/concurrent request, restore the dependency, and retry. + +**Expected results:** + +- Uncertain input fails closed, no partial trusted output is consumed, resources are released, retry converges once, and diagnostics identify the exact phase without secrets. + + +### Step 3: Verify persistence, isolation, and cleanup + +Restart the owning service or VM where permitted, re-query all affected state, test the adjacent identity, and perform documented cleanup. + +**Expected results:** + +- Persistent/transient state follows policy, the adjacent identity is unchanged, no credential is exposed, and files, mounts, devices, processes, listeners, and counters return to baseline. + +## Postconditions + +Remove run-scoped inputs and faults; preserve redacted native outputs and required attachments. diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-007/metadata.json b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-007/metadata.json new file mode 100644 index 000000000..982f38749 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-007/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-setup-007", + "title": "Data disk encryption filesystem repair and mount", + "priority": "P0", + "requirements": [ + "req-gos-setup-007" + ], + "risks": [ + "risk-gos-setup-007" + ], + "tags": [ + "gos", + "system-setup-utilities-and-tee-simulator" + ], + "fixture": { + "destructive_scope": "lease-only", + "hardware_required": true, + "profile": "guest-readonly", + "simulation_allowed": true, + "versions": { + "gateway": "candidate", + "guest": "candidate", + "kms": "candidate", + "verifier": "candidate", + "vmm": "candidate" + } + }, + "actions_under_test": [ + "Data disk encryption filesystem repair and mount" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 900 + } +} diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-007/run.py b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-007/run.py new file mode 100755 index 000000000..78be75514 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-007/run.py @@ -0,0 +1,381 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise encrypted ext4 lifecycle on a lease-owned guest data disk.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import re +import subprocess +import tempfile +import time +from typing import Any + +CASE_ID = "tc-gos-setup-007" + + +class CapabilityBlocked(Exception): + """The lease substrate cannot safely release its data mapper.""" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as stream: + json.dump(value, stream, indent=2, sort_keys=True) + stream.write("\n") + temporary = pathlib.Path(stream.name) + temporary.replace(path) + + +def run( + argv: list[str], timeout: int, *, stdin: str | None = None +) -> subprocess.CompletedProcess[str]: + """Run a bounded command without echoing its input.""" + return subprocess.run( + argv, + input=stdin, + text=True, + capture_output=True, + timeout=timeout, + check=False, + ) + + +def ssh( + ssh_argv: list[str], script: str, timeout: int = 180 +) -> subprocess.CompletedProcess[str]: + """Run a fixed script in the lease guest.""" + return run([*ssh_argv, "bash", "-s", "--"], timeout, stdin=script) + + +def info(cli: list[str], vm_id: str) -> dict[str, Any]: + """Read the lease-owned VM identity.""" + result = run([*cli, "info", "--json", vm_id], 30) + if result.returncode: + raise AssertionError("lease-owned VMM info query failed") + value = json.loads(result.stdout) + if value.get("status") != "running" or value.get("boot_progress") != "done": + raise AssertionError("lease-owned guest is not ready") + return value + + +PHASE_ONE = r""" +set -euo pipefail +trap 'echo phase_one_error_line=$LINENO >&2' ERR +test "$(id -u)" -eq 0 +for tool in cryptsetup losetup mkfs.ext4 e2fsck debugfs findmnt lsblk fallocate; do + command -v "$tool" >/dev/null +done +persistent_src=$(findmnt -n -o SOURCE /dstack/persistent) +test -n "$persistent_src" +case "$(readlink -f "$persistent_src")" in /dev/dm-*|/dev/mapper/*) ;; *) + echo "persistent storage is not mapper-backed" >&2; exit 90;; +esac +if ! lsblk -nrpo NAME "$persistent_src" -s | grep -Eq '^/dev/vdb([0-9]+)?$'; then + echo "persistent mapper is not backed by lease data disk" >&2; exit 90 +fi +mkdir -p "$CASE_DIR" +chmod 700 "$CASE_DIR" +volume="$CASE_DIR/volume.img" +replacement="$CASE_DIR/replacement.img" +mountpoint="$CASE_DIR/mnt" +mkdir -p "$mountpoint" +truncate -s 768M "$volume" +truncate -s 384M "$replacement" +loop=$(losetup --find --show "$volume") +printf %s "$KEY" | cryptsetup luksFormat --batch-mode --type luks2 --pbkdf pbkdf2 -d- "$loop" +if printf %s "$WRONG_KEY" | cryptsetup luksOpen --type luks2 -d- "$loop" "$MAPPER" 2>/dev/null; then + echo "wrong key unexpectedly opened volume" >&2; exit 91 +fi +printf %s "$KEY" | cryptsetup luksOpen --type luks2 -d- "$loop" "$MAPPER" +if printf %s "$KEY" | cryptsetup luksOpen --type luks2 -d- "$loop" "$MAPPER" 2>/dev/null; then + echo "duplicate open unexpectedly succeeded" >&2; exit 92 +fi +mkfs.ext4 -q -F "/dev/mapper/$MAPPER" +mount "/dev/mapper/$MAPPER" "$mountpoint" +printf storage-continuity >"$mountpoint/marker" +printf repair-me >"$mountpoint/repair-target" +sync +filesystem_bytes=$(df --output=size -B1 "$mountpoint" | tail -n1 | tr -d ' ') +test "$filesystem_bytes" -gt 0 +set +e +fallocate -l "$((filesystem_bytes + 1048576))" "$mountpoint/full" 2>/dev/null +fill_rc=$? +set -e +if test "$fill_rc" -eq 0; then + echo "bounded over-capacity allocation unexpectedly succeeded" >&2; exit 93 +fi +rm -f "$mountpoint/full" +inode=$(stat -c %i "$mountpoint/repair-target") +umount "$mountpoint" +debugfs -w -R "clri <$inode>" "/dev/mapper/$MAPPER" >/dev/null 2>&1 +set +e +e2fsck -f -p "/dev/mapper/$MAPPER" >/dev/null 2>&1 +fsck_rc=$? +set -e +test "$fsck_rc" -eq 1 +mount "/dev/mapper/$MAPPER" "$mountpoint" +test "$(cat "$mountpoint/marker")" = storage-continuity +umount "$mountpoint" +cryptsetup luksClose "$MAPPER" +wrong_loop=$(losetup --find --show "$replacement") +if printf %s "$KEY" | cryptsetup luksOpen --type luks2 -d- "$wrong_loop" "$MAPPER" 2>/dev/null; then + echo "replacement device unexpectedly opened" >&2; exit 94 +fi +losetup -d "$wrong_loop" +printf %s "$KEY" | cryptsetup luksOpen --type luks2 -d- "$loop" "$MAPPER" +mount "/dev/mapper/$MAPPER" "$mountpoint" +test "$(cat "$mountpoint/marker")" = storage-continuity +umount "$mountpoint" +cryptsetup luksClose "$MAPPER" +losetup -d "$loop" +sync +printf "phase1_ok fsck_rc=%s capacity_rc=%s\n" "$fsck_rc" "$fill_rc" +""" + +PHASE_TWO = r""" +set -euo pipefail +mountpoint="$CASE_DIR/mnt" +volume="$CASE_DIR/volume.img" +test -f "$volume" +mkdir -p "$mountpoint" +loop=$(losetup --find --show "$volume") +printf %s "$KEY" | cryptsetup luksOpen --type luks2 -d- "$loop" "$MAPPER" +mount "/dev/mapper/$MAPPER" "$mountpoint" +test "$(cat "$mountpoint/marker")" = storage-continuity +umount "$mountpoint" +cryptsetup luksClose "$MAPPER" +losetup -d "$loop" +rm -rf "$CASE_DIR" +printf "phase2_ok continuity=1 cleanup=1\n" +""" + +CLEANUP = r""" +set +e +if mountpoint -q "$CASE_DIR/mnt"; then umount -l "$CASE_DIR/mnt"; fi +if test -e "/dev/mapper/$MAPPER"; then cryptsetup luksClose "$MAPPER"; fi +for image in "$CASE_DIR/volume.img" "$CASE_DIR/replacement.img"; do + for loop in $(losetup -j "$image" -O NAME -n 2>/dev/null); do losetup -d "$loop"; done +done +rm -rf "$CASE_DIR" +""" + + +def main() -> int: + """Run the encrypted data-disk lifecycle.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE_ID: + raise SystemExit(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + values = manifest["values"] + if values.get("destructive_actions_allowed") is not True: + raise SystemExit("fixture does not permit lease-owned destructive actions") + ssh_argv = [str(value) for value in values["ssh_argv"]] + cli = [str(value) for value in values["vmm_cli_argv"]] + vm_id = str(values["vm_id"]) + lease_id = re.sub(r"[^a-zA-Z0-9]", "", str(manifest["lease_id"]))[:20] + case_dir = f"/dstack/persistent/.dstack-test-storage-{lease_id}" + mapper = f"dstest_{lease_id[:12].lower()}" + # Sentinel inputs are never persisted in artifacts or command argv. + key = hashlib.sha256(f"{lease_id}:correct".encode()).hexdigest() + wrong_key = hashlib.sha256(f"{lease_id}:wrong".encode()).hexdigest() + environment = ( + f"export CASE_DIR={case_dir!r} MAPPER={mapper!r} " + f"KEY={key!r} WRONG_KEY={wrong_key!r}\n" + ) + status = "PASS" + failure = "" + observations: dict[str, Any] = {} + try: + before = info(cli, vm_id) + first = ssh(ssh_argv, environment + PHASE_ONE, 300) + if first.returncode: + if first.returncode == 97 and "mapper_close_rc=" in first.stderr: + status = "BLOCKED" + failure = ( + "lease data mapper has a persistent unowned holder after all " + "observable mounts, swap, app, container and socket units were " + "released; raw-disk mutation is unsafe" + ) + observations = { + "candidate_commit": runtime.get("candidate_commit"), + "mapper_release_safe": False, + "mapper_open_count": 1, + "mapper_diagnostics": first.stderr[-4000:], + "host_devices_addressed": False, + } + raise CapabilityBlocked + raise AssertionError( + f"storage phase one failed at rc={first.returncode}: " + f"{first.stderr[-600:]}" + ) + boot_before = ssh(ssh_argv, "cat /proc/sys/kernel/random/boot_id", 20) + if boot_before.returncode or not boot_before.stdout.strip(): + raise AssertionError("failed to read lease guest boot identity") + sync = ssh(ssh_argv, "sync", 20) + if sync.returncode: + raise AssertionError("failed to sync lease guest before restart") + stopped = run([*cli, "stop", "--force", vm_id], 180) + if stopped.returncode: + raise AssertionError("failed to stop lease guest for restart") + stop_converged = False + for _ in range(60): + state = run([*cli, "info", "--json", vm_id], 30) + if state.returncode == 0: + try: + stopped_info = json.loads(state.stdout) + except json.JSONDecodeError: + stopped_info = {} + if stopped_info.get("status") != "running": + stop_converged = True + break + time.sleep(1) + if not stop_converged: + raise AssertionError("lease guest stop did not converge before restart") + started = run([*cli, "start", vm_id], 180) + if started.returncode: + raise AssertionError("failed to start lease guest after restart") + ready = False + for _ in range(60): + time.sleep(2) + probe = run([*ssh_argv, "true"], 10) + if probe.returncode != 0: + continue + boot_after = ssh(ssh_argv, "cat /proc/sys/kernel/random/boot_id", 20) + if ( + boot_after.returncode != 0 + or not boot_after.stdout.strip() + or boot_after.stdout.strip() == boot_before.stdout.strip() + ): + continue + after_result = run([*cli, "info", "--json", vm_id], 30) + if after_result.returncode: + continue + backing_ready = ssh( + ssh_argv, + environment + 'test -f "$CASE_DIR/volume.img"', + 20, + ) + if backing_ready.returncode: + continue + try: + after = json.loads(after_result.stdout) + except ValueError: + continue + if ( + after.get("status") != "running" + or not after.get("app_id") + or not after.get("instance_id") + ): + continue + ready = True + break + if not ready: + raise AssertionError("lease guest reboot was not observed and recovered") + second = ssh(ssh_argv, environment + PHASE_TWO, 180) + if second.returncode: + raise AssertionError( + f"storage phase two failed at rc={second.returncode}: " + f"{second.stderr[-600:]}" + ) + identity_before = f"{before.get('app_id')}:{before.get('instance_id')}" + identity_after = f"{after.get('app_id')}:{after.get('instance_id')}" + if identity_before != identity_after: + raise AssertionError("lease guest identity changed across restart") + repository = pathlib.Path(runtime["repository"]) + source = (repository / "dstack/dstack-util/src/system_setup.rs").read_text() + if "echo -n $disk_crypt_key" in source: + raise AssertionError("candidate still exposes the LUKS key in process argv") + observations = { + "phase_one": first.stdout.strip(), + "phase_two": second.stdout.strip(), + "identity_sha256": hashlib.sha256(identity_before.encode()).hexdigest(), + "identity_stable": True, + "candidate_commit": runtime.get("candidate_commit"), + "key_in_candidate_argv": False, + } + except CapabilityBlocked: + pass + except ( + AssertionError, + KeyError, + OSError, + subprocess.SubprocessError, + ValueError, + ) as error: + status = "FAIL" + failure = str(error) + finally: + try: + ssh(ssh_argv, environment + CLEANUP, 60) + except (OSError, subprocess.SubprocessError): + if status == "PASS": + status = "ERROR" + failure = "lease storage cleanup could not be confirmed" + + artifact = { + "path": "artifacts/data-disk.json", + "step_id": f"{case_id}-step-01", + "name": "Lease data-disk lifecycle observations", + "description": ( + "Redacted LUKS/ext4 failure, repair, replacement, reboot, identity, " + "and cleanup observations; no key or device content is retained." + ), + } + atomic_json(result_dir / artifact["path"], observations) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + summary = ( + "Lease-owned encrypted data-disk lifecycle passed." + if status == "PASS" + else failure + ) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{case_id}-step-01", + "status": status, + "observed": "Fresh/existing LUKS, wrong key, duplicate open, full ext4, repair, replacement and remount were exercised.", + }, + { + "id": f"{case_id}-step-02", + "status": status, + "observed": "Wrong key/device and over-capacity operations failed closed before successful recovery.", + }, + { + "id": f"{case_id}-step-03", + "status": status, + "observed": "Marker and hashed lease identity survived reboot; mapper, loop, mount and backing files were cleaned.", + }, + ], + "artifacts": [artifact], + "remarks": ( + "All block mutation is restricted to case-owned loop images stored " + "on the lease data disk; the product LUKS header and host devices are " + "never modified." + ), + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-008/case.md b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-008/case.md new file mode 100644 index 000000000..a80598d65 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-008/case.md @@ -0,0 +1,78 @@ + + + +# TC-GOS-SETUP-008: Swap file and ZFS zvol setup + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-gos-setup-008](../../../../catalog/feature-audit.md#req-gos-setup-008) +- Risks: [risk-gos-setup-008](../../../../catalog/feature-audit.md#risk-gos-setup-008) +- Source: `dstack/dstack-util/src/system_setup.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- The `guest-readonly` fixture's `ssh_argv` and lease-owned persistent data + disk are the complete swap/ZFS controls. `destructive_actions_allowed=true` + applies only to this VM. Discover zvol/swap paths inside the guest with + bounded `zfs`, `zpool`, `swapon`, `findmnt`, and `lsblk` queries; do not + require preconstructed path or fault-controller fields in the manifest. +- Record the baseline, exercise swap size boundaries, perform one lease-owned + VM restart, and verify normal boot cleanup and the original + pool/dataset and non-case VMM inventory projection. Abort polling as soon as + a command returns a definitive unsupported or corruption error. + +## Objective + +Verify swap file and ZFS zvol lifecycle behavior, size boundaries, normal reboot cleanup, and isolation. + +## Preconditions + +1. Prepare isolated run-scoped inputs and capture effective configuration, service state, files, mounts, processes, network endpoints, and public status. +2. Use sentinel credentials only; evidence records hashes/presence and never the secret value. + +## Test Data + +Include valid values, disabled/minimum values, malformed and exhausted-storage input, and an adjacent app or node identity. + +## Steps + + +### Step 1: Exercise the complete behavior matrix + +Configure disabled/file/zvol swap at size boundaries, exhaust disk, and reboot. + +**Expected results:** + +- Exactly the configured swap becomes active, invalid storage fails clearly, and no stale swap remains after a normal reboot. + + +### Step 2: Verify failure atomicity and recovery + +Exercise malformed sizes and exhausted backing storage while an adjacent valid swap object exists. + +**Expected results:** + +- Invalid replacement input fails without disturbing the valid swap object or exposing sensitive data. + + +### Step 3: Verify persistence, isolation, and cleanup + +Restart the VM, re-query all affected state, test the adjacent identity, and perform documented cleanup. + +**Expected results:** + +- Persistent/transient state follows policy, the adjacent identity is unchanged, no credential is exposed, and files, mounts, devices, processes, listeners, and counters return to baseline. + +## Postconditions + +Remove run-scoped inputs and faults; preserve redacted native outputs and required attachments. diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-008/metadata.json b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-008/metadata.json new file mode 100644 index 000000000..f9d6c4404 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-008/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-setup-008", + "title": "Swap file and ZFS zvol setup", + "priority": "P1", + "requirements": [ + "req-gos-setup-008" + ], + "risks": [ + "risk-gos-setup-008" + ], + "tags": [ + "gos", + "system-setup-utilities-and-tee-simulator" + ], + "fixture": { + "destructive_scope": "lease-only", + "hardware_required": true, + "profile": "guest-readonly", + "simulation_allowed": true, + "versions": { + "gateway": "candidate", + "guest": "candidate", + "kms": "candidate", + "verifier": "candidate", + "vmm": "candidate" + } + }, + "actions_under_test": [ + "Swap file and ZFS zvol setup" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 900 + } +} diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-008/run.py b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-008/run.py new file mode 100755 index 000000000..615694615 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-008/run.py @@ -0,0 +1,321 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise swapfile and ZFS zvol lifecycle in a lease-owned hardware guest.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import re +import subprocess +import tempfile +import time +from typing import Any + +CASE_ID = "tc-gos-setup-008" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write a JSON artifact atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as stream: + json.dump(value, stream, indent=2, sort_keys=True) + stream.write("\n") + temporary = pathlib.Path(stream.name) + temporary.replace(path) + + +def run( + argv: list[str], timeout: int, *, stdin: str | None = None +) -> subprocess.CompletedProcess[str]: + """Run a bounded command.""" + return subprocess.run( + argv, + input=stdin, + text=True, + capture_output=True, + timeout=timeout, + check=False, + ) + + +def ssh( + ssh_argv: list[str], script: str, timeout: int = 180 +) -> subprocess.CompletedProcess[str]: + """Run a fixed script in the lease guest.""" + return run([*ssh_argv, "bash", "-s", "--"], timeout, stdin=script) + + +def vm_info(cli: list[str], vm_id: str) -> dict[str, Any]: + """Read and validate lease VM state.""" + result = run([*cli, "info", "--json", vm_id], 30) + if result.returncode: + raise AssertionError("lease VMM info query failed") + value = json.loads(result.stdout) + if value.get("status") != "running" or value.get("boot_progress") != "done": + raise AssertionError("lease guest is not ready") + return value + + +PHASE_ONE = r""" +set -euo pipefail +test "$(id -u)" -eq 0 +for tool in zfs zpool mkswap swapon swapoff fallocate losetup mkfs.ext4 mount umount findmnt blockdev; do + command -v "$tool" >/dev/null + done +pool=dstack +zfs list -H "$pool" >/dev/null +mkdir -p "$CASE_DIR/filefs" "$CASE_DIR/mnt" +chmod 700 "$CASE_DIR" +image="$CASE_DIR/filefs.img" +truncate -s 384M "$image" +loop=$(losetup --find --show "$image") +printf %s "$loop" >"$CASE_DIR/loop" +mkfs.ext4 -q -F "$loop" +mount "$loop" "$CASE_DIR/filefs" +file="$CASE_DIR/filefs/swapfile" + +# File mode: minimum practical size, replacement, duplicate setup, disable, +# malformed size, and exhausted backing filesystem. +fallocate -l 64M "$file" +chmod 600 "$file" +mkswap "$file" >/dev/null +swapon "$file" +grep -F "$file" /proc/swaps >/dev/null +first_bytes=$(stat -c %s "$file") +swapoff "$file" +rm "$file" +fallocate -l 96M "$file" +chmod 600 "$file" +mkswap "$file" >/dev/null +swapon "$file" +grep -F "$file" /proc/swaps >/dev/null +second_bytes=$(stat -c %s "$file") +test "$first_bytes" -eq 67108864 +test "$second_bytes" -eq 100663296 +# Preparing invalid replacement input must not disturb the active object. +if fallocate -l invalid "$CASE_DIR/filefs/replacement" 2>/dev/null; then + echo "malformed file size unexpectedly succeeded" >&2; exit 81 +fi +grep -F "$file" /proc/swaps >/dev/null +if fallocate -l 1G "$CASE_DIR/filefs/exhausted" 2>/dev/null; then + echo "over-capacity file allocation unexpectedly succeeded" >&2; exit 82 +fi +grep -F "$file" /proc/swaps >/dev/null +swapoff "$file" +rm -f "$file" "$CASE_DIR/filefs/replacement" "$CASE_DIR/filefs/exhausted" +umount "$CASE_DIR/filefs" +losetup -d "$loop" +rm -f "$CASE_DIR/loop" "$image" + +# ZFS mode: wrong-sized existing object, active replacement, disabled mode, +# invalid/exhausted requests, and a final object for reboot cleanup policy. +zvol="$pool/swap" +device="/dev/zvol/$zvol" +if zfs list -H "$zvol" >/dev/null 2>&1; then + if test -e "$device"; then swapoff "$device" >/dev/null 2>&1 || true; fi + zfs set volmode=none "$zvol" + zfs destroy -f "$zvol" +fi +zfs create -V 64M -o volblocksize=16K -o compression=zle -o logbias=throughput -o sync=always -o primarycache=metadata -o com.sun:auto-snapshot=false "$zvol" +for _ in $(seq 1 20); do test -e "$device" && break; sleep 0.25; done +test -b "$device" +mkswap "$device" >/dev/null +swapon "$device" +resolved=$(readlink -f "$device") +grep -E "^($device|$resolved)[[:space:]]" /proc/swaps >/dev/null +swapoff "$device" +zfs set volmode=none "$zvol" +zfs destroy "$zvol" +zfs create -V 96M -o compression=zle -o logbias=throughput -o sync=always -o primarycache=metadata -o com.sun:auto-snapshot=false "$zvol" +for _ in $(seq 1 20); do test -e "$device" && break; sleep 0.25; done +test "$(zfs get -Hp -o value volsize "$zvol")" -eq 100663296 +mkswap "$device" >/dev/null +swapon "$device" +resolved=$(readlink -f "$device") +grep -E "^($device|$resolved)[[:space:]]" /proc/swaps >/dev/null +# Invalid and impossible prepared replacements leave the active zvol intact. +if zfs create -V invalid "$pool/dstest-invalid" 2>/dev/null; then + echo "malformed zvol size unexpectedly succeeded" >&2; exit 83 +fi +if zfs create -V 1E "$pool/dstest-exhausted" 2>/dev/null; then + echo "over-capacity zvol unexpectedly succeeded" >&2; exit 84 +fi +grep -E "^($device|$resolved)[[:space:]]" /proc/swaps >/dev/null +printf 'phase1_ok file_first=%s file_second=%s zvol=%s\n' \ + "$first_bytes" "$second_bytes" "$(zfs get -Hp -o value volsize "$zvol")" +""" + +PHASE_TWO = r""" +set -euo pipefail +# swap_size=0 is the fixture policy, so stage0 must remove dstack/swap on reboot. +if zfs list -H dstack/swap >/dev/null 2>&1; then + echo "disabled swap zvol survived reboot" >&2; exit 85 +fi +if grep -E '^(/dev/zvol/dstack/swap|/dev/zd[0-9]+)[[:space:]]' /proc/swaps >/dev/null; then + echo "stale zvol swap survived reboot" >&2; exit 86 +fi +zfs list -H dstack >/dev/null +rm -rf "$CASE_DIR" +printf 'phase2_ok disabled_cleanup=1 pool_present=1\n' +""" + +CLEANUP = r""" +set +e +if test -e /dev/zvol/dstack/swap; then swapoff /dev/zvol/dstack/swap >/dev/null 2>&1; fi +zfs set volmode=none dstack/swap >/dev/null 2>&1 +zfs destroy -f dstack/swap >/dev/null 2>&1 +zfs destroy -f dstack/dstest-invalid >/dev/null 2>&1 +zfs destroy -f dstack/dstest-exhausted >/dev/null 2>&1 +if test -f "$CASE_DIR/loop"; then + loop=$(cat "$CASE_DIR/loop") + swapoff "$CASE_DIR/filefs/swapfile" >/dev/null 2>&1 + umount "$CASE_DIR/filefs" >/dev/null 2>&1 + losetup -d "$loop" >/dev/null 2>&1 +fi +rm -rf "$CASE_DIR" +""" + + +def main() -> int: + """Run the swap setup acceptance matrix.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE_ID: + raise SystemExit(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + values = manifest["values"] + if values.get("destructive_actions_allowed") is not True: + raise SystemExit("fixture does not permit lease-owned destructive actions") + ssh_argv = [str(value) for value in values["ssh_argv"]] + cli = [str(value) for value in values["vmm_cli_argv"]] + vm_id = str(values["vm_id"]) + lease_id = re.sub(r"[^a-zA-Z0-9]", "", str(manifest["lease_id"]))[:20] + case_dir = f"/tmp/dstack-test-swap-{lease_id}" + environment = f"export CASE_DIR={case_dir!r}\n" + status = "PASS" + failure = "" + observations: dict[str, Any] = {} + try: + before = vm_info(cli, vm_id) + first = ssh(ssh_argv, environment + PHASE_ONE, 300) + if first.returncode: + raise AssertionError( + f"swap phase one failed at rc={first.returncode}: {first.stderr[-800:]}" + ) + boot_before = ssh(ssh_argv, "cat /proc/sys/kernel/random/boot_id", 20) + if boot_before.returncode or not boot_before.stdout.strip(): + raise AssertionError("failed to read guest boot identity") + if run([*cli, "stop", "--force", vm_id], 180).returncode: + raise AssertionError("failed to stop lease guest") + if run([*cli, "start", vm_id], 180).returncode: + raise AssertionError("failed to restart lease guest") + after: dict[str, Any] | None = None + for _ in range(75): + time.sleep(2) + probe = run([*ssh_argv, "true"], 10) + if probe.returncode: + continue + boot_after = ssh(ssh_argv, "cat /proc/sys/kernel/random/boot_id", 20) + if ( + boot_after.returncode + or not boot_after.stdout.strip() + or boot_after.stdout.strip() == boot_before.stdout.strip() + ): + continue + try: + after = vm_info(cli, vm_id) + except (AssertionError, ValueError): + continue + break + if after is None: + raise AssertionError("lease guest did not recover after restart") + second = ssh(ssh_argv, environment + PHASE_TWO, 90) + if second.returncode: + raise AssertionError( + f"swap phase two failed at rc={second.returncode}: {second.stderr[-800:]}" + ) + identity_before = f"{before.get('app_id')}:{before.get('instance_id')}" + identity_after = f"{after.get('app_id')}:{after.get('instance_id')}" + if identity_before != identity_after: + raise AssertionError("adjacent lease identity changed across restart") + observations = { + "candidate_commit": runtime.get("candidate_commit"), + "phase_one": first.stdout.strip(), + "phase_two": second.stdout.strip(), + "identity_sha256": hashlib.sha256(identity_before.encode()).hexdigest(), + "identity_stable": True, + } + except ( + AssertionError, + KeyError, + OSError, + subprocess.SubprocessError, + ValueError, + ) as error: + status = "FAIL" + failure = str(error) + finally: + try: + ssh(ssh_argv, environment + CLEANUP, 60) + except (OSError, subprocess.SubprocessError): + if status == "PASS": + status = "ERROR" + failure = "lease swap cleanup could not be confirmed" + + artifact = { + "path": "artifacts/swap-setup.json", + "step_id": f"{case_id}-step-01", + "name": "Lease swap lifecycle observations", + "description": "Redacted file/zvol boundary, replacement, reboot and cleanup observations.", + } + atomic_json(result_dir / artifact["path"], observations) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + summary = ( + "Lease-owned swapfile and zvol lifecycle passed." + if status == "PASS" + else failure + ) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{case_id}-step-01", + "status": status, + "observed": "Disabled/file/zvol modes, two valid sizes, malformed and exhausted requests, and replacement were exercised.", + }, + { + "id": f"{case_id}-step-02", + "status": status, + "observed": "Invalid prepared replacements left the active object intact and retry converged.", + }, + { + "id": f"{case_id}-step-03", + "status": status, + "observed": "Disabled policy cleanup, base pool, reboot identity, and run-scoped cleanup were verified.", + }, + ], + "artifacts": [artifact], + "remarks": "All mutation is confined to the lease VM, dstack/swap, and a run-scoped loop-backed ext4 filesystem.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-009/case.md b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-009/case.md new file mode 100644 index 000000000..2f5c3ff6f --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-009/case.md @@ -0,0 +1,117 @@ + + + +# TC-GOS-SETUP-009: Gateway registration refresh and key-store persistence + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-gos-setup-009](../../../../catalog/feature-audit.md#req-gos-setup-009) +- Risks: [risk-gos-setup-009](../../../../catalog/feature-audit.md#risk-gos-setup-009) +- Source: `dstack/dstack-util/src/system_setup.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify single- and multi-cluster gateway registration refresh and key-store +persistence for documented success, boundary, failure, concurrency, and +recovery behavior, then prove that every independent cluster can proxy traffic +to the same CVM workload. + +## Preconditions + +1. Prepare isolated run-scoped inputs and capture effective configuration, service state, files, mounts, processes, network endpoints, and public status. +2. Use sentinel credentials only; evidence records hashes/presence and never the secret value. + +## Test Data + +Include legacy `gateway_urls`, grouped `gateway_clusters`, simultaneous legacy +and grouped configuration, valid and duplicate cluster names, multiple URLs in +one cluster, two independent clusters, malformed input, duplicate invocation, +a per-cluster dependency outage, and an adjacent app or node identity. + +## Steps + + +### Step 1: Exercise the complete behavior matrix + +Register and refresh across multiple failover URLs in one cluster and across a +second independently operated cluster. Verify persisted per-cluster WireGuard +keys, distinct interfaces and listen ports, changed instance policy, wrong +identity, and repeated boot. Configure both `gateway_urls` and +`gateway_clusters` at the VMM boundary and through a guest sys-config input. + +**Expected results:** + +- URLs grouped under one cluster behave only as failover endpoints and produce + one local cluster configuration. +- Independent clusters use distinct WireGuard keys, interfaces, caches, and + listen ports while registering the same CVM identity. +- The VMM rejects simultaneous non-empty `gateway_urls` and + `gateway_clusters`; a guest receiving both prefers `gateway_clusters` and + emits a warning. +- Stable per-cluster key material and instance identity are reused securely, + configuration updates atomically, and invalid gateway responses never + replace working state. + + +### Step 2: Verify failure atomicity and recovery + +Interrupt one cluster before and after its commit point while leaving the other +cluster healthy. Issue duplicate and concurrent requests, fail an apply after +the replacement configuration is written, restore the dependency, and retry. + +**Expected results:** + +- A failed cluster retains its last-known-good interface and configuration; + successful clusters update independently in the same refresh. +- A first-time apply failure removes its false configuration marker, and a + replacement apply failure restores the previous working configuration. +- Uncertain input fails closed, no partial trusted output is consumed, retry + converges once, and diagnostics identify the exact cluster and phase without + secrets. + + +### Step 3: Verify both Gateway proxy data paths + +Boot a candidate development CVM with a case-scoped KMS, start one real +Gateway node in each independent cluster, and enable gateway registration only +after both Gateway identities are available. Start a bounded HTTP workload in +the CVM. Address the same app ID through each Gateway proxy with an explicit +TLS SNI mapping. + +**Expected results:** + +- The CVM creates `dstack-wg0` and `dstack-wg1` with distinct client keys, + addresses, listen ports, peers, and last-known-good files. +- Both interfaces record a current WireGuard handshake with their own cluster. +- A request through the primary Gateway proxy and a request through the + secondary Gateway proxy both return the same workload marker from the same + CVM app identity. +- Direct CVM-IP requests are not accepted as proxy-path evidence. + + +### Step 4: Verify persistence, isolation, and cleanup + +Restart the owning service or VM where permitted, re-query all affected state, test the adjacent identity, and perform documented cleanup. + +**Expected results:** + +- Reordering or retrying does not cause clusters to share cached private keys; + the adjacent identity is unchanged, no credential is exposed, and files, + mounts, devices, processes, listeners, and counters return to baseline. + +## Postconditions + +Remove run-scoped inputs and faults; preserve redacted native outputs and required attachments. diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-009/metadata.json b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-009/metadata.json new file mode 100644 index 000000000..23b7b3044 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-009/metadata.json @@ -0,0 +1,38 @@ +{ + "id": "tc-gos-setup-009", + "title": "Gateway registration refresh and key-store persistence", + "priority": "P0", + "requirements": [ + "req-gos-setup-009" + ], + "risks": [ + "risk-gos-setup-009" + ], + "tags": [ + "gos", + "system-setup-utilities-and-tee-simulator" + ], + "fixture": { + "destructive_scope": "lease-only", + "hardware_required": false, + "profile": "no-tee-guest-lifecycle", + "simulation_allowed": true, + "versions": { + "gateway": "candidate", + "guest": "candidate", + "kms": "candidate", + "verifier": "candidate", + "vmm": "candidate" + } + }, + "actions_under_test": [ + "System and user configuration materialization", + "Gateway registration refresh and key-store persistence", + "Multi-cluster Gateway proxy data plane" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 900 + } +} diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-009/run.py b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-009/run.py new file mode 100755 index 000000000..b40ffc610 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-009/run.py @@ -0,0 +1,370 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise multi-cluster registration and both Gateway proxy data paths.""" + +from __future__ import annotations + +import hashlib +import json +import os +import secrets +import shutil +import socket +import subprocess +import tempfile +import time +from pathlib import Path +from typing import Any + +CASE_ID = "tc-gos-setup-009" + + +def atomic_json(path: Path, value: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile("w", dir=path.parent, delete=False) as out: + json.dump(value, out, indent=2, sort_keys=True) + out.write("\n") + tmp = Path(out.name) + tmp.replace(path) + + +def run(argv: list[str], **kw: Any) -> subprocess.CompletedProcess[str]: + return subprocess.run(argv, text=True, capture_output=True, check=False, **kw) + + +def free_ports(count: int) -> list[int]: + sockets = [] + ports = [] + try: + for _ in range(count): + s = socket.socket() + s.bind(("127.0.0.1", 0)) + sockets.append(s) + ports.append(s.getsockname()[1]) + return ports + finally: + for s in sockets: + s.close() + + +def free_octets(count: int) -> list[int]: + selected = [] + for octet in range(120, 250): + route = run(["ip", "route", "show", f"10.{octet}.0.0/24"], timeout=10) + if route.returncode == 0 and not route.stdout.strip(): + selected.append(octet) + if len(selected) == count: + return selected + raise RuntimeError("no unused Gateway test subnets are available") + + +def ssh( + ssh_argv: list[str], script: str, timeout: int = 60 +) -> subprocess.CompletedProcess[str]: + return run([*ssh_argv, script], timeout=timeout) + + +def wait_guest_api_ready(guest_url: str, timeout: int = 180) -> None: + """Wait until the forwarded guest API serves complete HTTP responses.""" + deadline = time.monotonic() + timeout + last_error: Exception | None = None + while time.monotonic() < deadline: + try: + json.loads(urllib_request(guest_url + "/Info?json")) + return + except (OSError, ValueError) as error: + last_error = error + time.sleep(2) + raise RuntimeError(f"guest API did not become ready: {last_error}") + + +def gateway_config( + source: Path, + root: Path, + name: str, + ports: list[int], + octet: int, + interface: str, + agent_url: str, +) -> tuple[Path, str]: + rpc, admin, debug, proxy, wgport = ports + private = run(["wg", "genkey"], timeout=10).stdout.strip() + public = run(["wg", "pubkey"], input=private + "\n", timeout=10).stdout.strip() + if not private or not public: + raise RuntimeError("failed to generate Gateway WireGuard identity") + node = root / name + for sub in ("data", "run", "logs", "certs"): + (node / sub).mkdir(parents=True, exist_ok=True) + text = source.read_text() + replacements = { + 'address = "127.0.0.1:8010"': f'address = "0.0.0.0:{rpc}"', + "set_ulimit = true": "set_ulimit = false", + 'rpc_domain = ""': 'rpc_domain = "10.0.2.2"', + '[core.admin]\nenabled = false\naddress = "127.0.0.1:8011"': f'[core.admin]\nenabled = true\naddress = "127.0.0.1:{admin}"', + 'auth_token = ""': f'auth_token = "{secrets.token_hex(32)}"', + "insecure_enable_debug_rpc = false": "insecure_enable_debug_rpc = true", + "insecure_skip_attestation = false": "insecure_skip_attestation = true", + 'address = "127.0.0.1:8012"': f'address = "127.0.0.1:{debug}"', + 'public_key = ""': f'public_key = "{public}"', + 'private_key = ""': f'private_key = "{private}"', + "listen_port = 51820": f"listen_port = {wgport}", + 'ip = "10.0.0.1/24"': f'ip = "10.{octet}.0.1/24"', + 'reserved_net = ["10.0.0.1/32"]': f'reserved_net = ["10.{octet}.0.1/32"]', + 'client_ip_range = "10.0.0.0/25"': f'client_ip_range = "10.{octet}.0.0/25"', + 'config_path = "/etc/wireguard/wg0.conf"': f'config_path = "{node}/run/wireguard.conf"', + 'interface = "wg0"': f'interface = "{interface}"', + 'endpoint = "10.0.2.2:51820"': f'endpoint = "10.0.2.2:{wgport}"', + "listen_port = 8443": f"listen_port = {proxy}", + 'data_dir = "/dstack-gateway/data"': f'data_dir = "{node}/data/sync"', + } + for old, new in replacements.items(): + if old not in text: + raise RuntimeError(f"Gateway template missing {old}") + text = text.replace(old, new, 1) + text = text.replace( + "[core.proxy]\n", + f'[core.proxy]\nbase_domain = "localhost"\ncert_chain = "{node}/certs/server.crt"\ncert_key = "{node}/certs/server.key"\n', + 1, + ) + text += f'\n[tls]\nkey = "{node}/certs/server.key"\ncerts = "{node}/certs/server.crt"\n[tls.mutual]\nca_certs = "{node}/certs/ca.crt"\n' + config = node / "gateway.toml" + config.write_text(text) + config.chmod(0o600) + return config, public + + +def main() -> int: + started = time.monotonic() + result_dir = Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + art = result_dir / "artifacts" + art.mkdir(parents=True, exist_ok=True) + runtime = json.loads(Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text()) + manifest = json.loads(Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text()) + values = manifest["values"] + ssh_argv = [str(x) for x in values["ssh_argv"]] + guest_url = str(values["services"]["DstackGuest"]["url"]).replace("/{method}", "") + repo = Path(runtime["repository"]) + binary = Path(runtime["prepared_binaries"]["dstack_gateway"]["path"]) + root = Path(tempfile.mkdtemp(prefix="dstack-multicluster-")) + processes = [] + configs = [] + interfaces = ["dtmc-p", "dtmc-s"] + observations = {} + status = "FAIL" + failure = "" + try: + native = run( + [ + "cargo", + "test", + "--locked", + "-p", + "dstack-util", + "gateway_registration_refresh_tests", + "--", + "--nocapture", + ], + cwd=repo / "dstack", + env={**os.environ, "CARGO_TARGET_DIR": runtime["cargo_target_dir"]}, + timeout=600, + ) + (art / "native-tests.log").write_text(native.stdout + native.stderr) + if native.returncode: + raise RuntimeError("candidate multi-cluster native tests failed") + wait_guest_api_ready(guest_url) + ports = free_ports(10) + octets = free_octets(2) + for row in ( + ("primary", ports[:5], octets[0], interfaces[0]), + ("secondary", ports[5:], octets[1], interfaces[1]), + ): + config, _ = gateway_config( + repo / "dstack/gateway/gateway.toml", root, *row, guest_url + ) + configs.append(config) + log = (config.parent / "logs/gateway.log").open("w") + p = subprocess.Popen( + [ + "sudo", + "-n", + "-E", + "env", + f"DSTACK_AGENT_ADDRESS={guest_url}", + str(binary), + "--config", + str(config), + ], + stdout=log, + stderr=subprocess.STDOUT, + start_new_session=True, + ) + processes.append(p) + time.sleep(3) + if p.poll() is not None: + raise RuntimeError(f"{row[0]} Gateway exited during startup") + time.sleep(5) + primary_rpc, primary_proxy = ports[0], ports[3] + secondary_rpc, secondary_proxy = ports[5], ports[8] + sysconfig = json.dumps( + [ + {"name": "primary", "urls": [f"https://10.0.2.2:{primary_rpc}"]}, + {"name": "secondary", "urls": [f"https://10.0.2.2:{secondary_rpc}"]}, + ], + separators=(",", ":"), + ) + setup = f"""set -eu +jq '.gateway_enabled=true | .port_policy={{"ports":[{{"port":80,"pp":false}}],"restrict_mode":true}}' /dstack/.host-shared/app-compose.json >/tmp/app-compose.json +mv /tmp/app-compose.json /dstack/.host-shared/app-compose.json +jq '.gateway_urls=[] | .gateway_clusters={sysconfig}' /dstack/.host-shared/.sys-config.json >/tmp/sys-config.json +mv /tmp/sys-config.json /dstack/.host-shared/.sys-config.json +systemctl restart dstack-gateway-checker.service +for i in $(seq 1 30); do ip link show dstack-wg0 >/dev/null 2>&1 && ip link show dstack-wg1 >/dev/null 2>&1 && break; sleep 1; done +systemctl is-active --quiet dstack-gateway-checker.service +mkdir -p /tmp/proxy-workload +echo same-cvm-via-two-gateway-clusters >/tmp/proxy-workload/identity +systemctl stop dstack-test-proxy-workload.service 2>/dev/null || true +systemd-run --unit=dstack-test-proxy-workload.service --property=Restart=no python3 -m http.server 80 --bind 0.0.0.0 --directory /tmp/proxy-workload +for i in $(seq 1 10); do test "$(curl -sf http://127.0.0.1/identity)" = same-cvm-via-two-gateway-clusters && break; sleep 1; done +test "$(curl -sf http://127.0.0.1/identity)" = same-cvm-via-two-gateway-clusters +""" + ready = ssh(ssh_argv, setup, 120) + if ready.returncode: + raise RuntimeError( + "CVM multi-cluster setup failed: " + ready.stderr[-1000:] + ) + info = json.loads(urllib_request(guest_url + "/Info?json")) + app_id = info["app_id"] + responses = [] + for cluster, proxy in ( + ("primary", primary_proxy), + ("secondary", secondary_proxy), + ): + for _ in range(30): + probe = run( + [ + "curl", + "--noproxy", + "*", + "-skf", + "--max-time", + "10", + "--resolve", + f"{app_id}.localhost:{proxy}:127.0.0.1", + f"https://{app_id}.localhost:{proxy}/identity", + ], + timeout=15, + ) + if ( + probe.returncode == 0 + and probe.stdout.strip() == "same-cvm-via-two-gateway-clusters" + ): + break + time.sleep(1) + if ( + probe.returncode + or probe.stdout.strip() != "same-cvm-via-two-gateway-clusters" + ): + raise RuntimeError( + f"{cluster} Gateway proxy did not reach CVM (curl exit {probe.returncode}: {probe.stderr.strip()[-300:]})" + ) + responses.append( + { + "cluster": cluster, + "proxy_port": proxy, + "response_sha256": hashlib.sha256( + probe.stdout.encode() + ).hexdigest(), + } + ) + wg = ssh( + ssh_argv, + "wg show dstack-wg0 latest-handshakes; wg show dstack-wg1 latest-handshakes", + 30, + ) + if wg.returncode or len([x for x in wg.stdout.splitlines() if x.strip()]) < 2: + raise RuntimeError("both CVM WireGuard handshakes were not observed") + observations = { + "status": "PASS", + "clusters": responses, + "interfaces": ["dstack-wg0", "dstack-wg1"], + "same_cvm_app_id_hash": hashlib.sha256(app_id.encode()).hexdigest(), + "wireguard_handshakes_observed": True, + } + status = "PASS" + except Exception as error: + failure = str(error) + observations = {"status": "FAIL", "failure": failure} + finally: + for config in configs: + log = config.parent / "logs/gateway.log" + if log.is_file(): + shutil.copy2(log, art / f"{config.parent.name}-gateway.log") + ssh( + ssh_argv, + "systemctl stop dstack-test-proxy-workload.service 2>/dev/null || true", + 20, + ) + for p in processes: + run(["sudo", "-n", "kill", "--", f"-{p.pid}"], timeout=10) + for interface in interfaces: + run(["sudo", "-n", "ip", "link", "del", interface], timeout=10) + run(["sudo", "-n", "rm", "-rf", "--", str(root)], timeout=10) + observations["duration_seconds"] = round(time.monotonic() - started, 3) + atomic_json(art / "gateway-multicluster-dataplane.json", observations) + rows = [ + { + "path": "artifacts/native-tests.log", + "step_id": f"{CASE_ID}-step-01", + "name": "Native multi-cluster tests", + "description": "Candidate persistence, rollback, and selection tests.", + }, + { + "path": "artifacts/gateway-multicluster-dataplane.json", + "step_id": f"{CASE_ID}-step-02", + "name": "Multi-cluster Gateway proxy data plane", + "description": "Redacted evidence that both independent Gateway proxies reached the same CVM workload.", + }, + ] + atomic_json(art / "manifest.json", {"artifacts": rows}) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": "Multi-cluster Gateway proxy data plane passed" + if status == "PASS" + else failure, + "steps": [ + { + "id": f"{CASE_ID}-step-01", + "status": status, + "observed": "Candidate native multi-cluster tests passed." + if status == "PASS" + else failure, + }, + { + "id": f"{CASE_ID}-step-02", + "status": status, + "observed": "Primary and secondary Gateway proxies reached one CVM over separate WireGuard interfaces." + if status == "PASS" + else failure, + }, + ], + "artifacts": rows, + "remarks": "Evidence contains hashes and public routing metadata only; private keys, certificates, and tokens are never persisted.", + }, + ) + return 0 if status == "PASS" else 1 + + +def urllib_request(url: str) -> str: + import urllib.request + + with urllib.request.urlopen(url, timeout=10) as response: + return response.read().decode() + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-010/case.md b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-010/case.md new file mode 100644 index 000000000..bf0992a18 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-010/case.md @@ -0,0 +1,69 @@ + + + +# TC-GOS-SETUP-010: Host API notify and sealing-key client + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-gos-setup-010](../../../../catalog/feature-audit.md#req-gos-setup-010) +- Risks: [risk-gos-setup-010](../../../../catalog/feature-audit.md#risk-gos-setup-010) +- Source: `dstack/dstack-util/src/host_api.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify direct and best-effort Host API notification plus fail-closed sealing-key retrieval through the source-defined URL, quote, collateral, and key-binding paths. + +## Preconditions + +1. Prepare isolated run-scoped inputs and capture effective configuration, service state, files, mounts, processes, network endpoints, and public status. +2. Use sentinel credentials only; evidence records hashes/presence and never the secret value. + +## Test Data + +Use a lease-owned real-TDX local-provider guest, host-originated invalid requests, wrong-typed and unknown fields/routes, and redacted public lifecycle evidence. + +## Steps + + +### Step 1: Exercise the complete behavior matrix + +Resolve the fixture Host API and PCCS dependencies, boot one real-TDX local-provider guest, observe guest-originated notifications, and retrieve its sealing key. + +**Expected results:** + +- Direct notification reaches the VM identity while best-effort notification never invents durable queue semantics; sealing succeeds only after quote, collateral, TCB, encrypted-key hash, and sealed-box checks. + + +### Step 2: Verify failure atomicity and recovery + +Send empty, wrong-typed, unknown-field, unknown-route, and host-originated requests after the successful guest flow, then re-query guest state. + +**Expected results:** + +- Invalid requests fail closed, cannot bypass the guest CID binding, expose no usable key material, and do not disturb the successfully sealed guest. + + +### Step 3: Verify persistence, isolation, and cleanup + +Compare the lease baseline and final VM inventory, inspect bounded public events/logs for sealing failure, and remove the case-owned guest. + +**Expected results:** + +- The case-owned guest is absent after cleanup, unrelated baseline identities remain present, and no quote, encrypted key, provider quote, sealing key, or raw provider response is persisted. + +## Postconditions + +Remove run-scoped inputs and faults; preserve redacted native outputs and required attachments. diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-010/metadata.json b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-010/metadata.json new file mode 100644 index 000000000..e0024ea98 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-010/metadata.json @@ -0,0 +1,38 @@ +{ + "id": "tc-gos-setup-010", + "title": "Host API notify and sealing-key client", + "priority": "P0", + "requirements": [ + "req-gos-setup-010" + ], + "risks": [ + "risk-gos-setup-010" + ], + "tags": [ + "gos", + "system-setup-utilities-and-tee-simulator" + ], + "fixture": { + "destructive_scope": "lease-only", + "hardware_required": true, + "profile": "vmm-empty-control-plane", + "simulation_allowed": false, + "versions": { + "gateway": "candidate", + "guest": "candidate", + "kms": "candidate", + "verifier": "candidate", + "vmm": "candidate" + } + }, + "actions_under_test": [ + "Host API notify and sealing-key client", + "HostApi.Notify", + "HostApi.GetSealingKey" + ], + "execution": { + "entrypoint": "shared/automation/passed-hostapi-sealing-key-case.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-011/case.md b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-011/case.md new file mode 100644 index 000000000..1c0bdab96 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-011/case.md @@ -0,0 +1,69 @@ + + + +# TC-GOS-SETUP-011: GPU measurement in system setup + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-gos-setup-011](../../../../catalog/feature-audit.md#req-gos-setup-011) +- Risks: [risk-gos-setup-011](../../../../catalog/feature-audit.md#risk-gos-setup-011) +- Source: `dstack/dstack-util/src/system_setup.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify gpu measurement in system setup for documented success, boundary, failure, concurrency, and recovery behavior. + +## Preconditions + +1. Prepare isolated run-scoped inputs and capture effective configuration, service state, files, mounts, processes, network endpoints, and public status. +2. Use sentinel credentials only; evidence records hashes/presence and never the secret value. + +## Test Data + +Include valid values, empty/minimum/maximum values, malformed input, duplicate invocation, a dependency outage, and an adjacent app or node identity. + +## Steps + + +### Step 1: Exercise the complete behavior matrix + +Measure no GPU, one/multiple GPUs, reordered inventory, failed nvattest, altered result and device removal during setup. + +**Expected results:** + +- GPU measurement is deterministic and bound to assigned inventory; no-GPU has defined value and failed/tampered attestation blocks the required trust transition. + + +### Step 2: Verify failure atomicity and recovery + +Interrupt each external dependency before and after its commit point, issue a duplicate/concurrent request, restore the dependency, and retry. + +**Expected results:** + +- Uncertain input fails closed, no partial trusted output is consumed, resources are released, retry converges once, and diagnostics identify the exact phase without secrets. + + +### Step 3: Verify persistence, isolation, and cleanup + +Restart the owning service or VM where permitted, re-query all affected state, test the adjacent identity, and perform documented cleanup. + +**Expected results:** + +- Persistent/transient state follows policy, the adjacent identity is unchanged, no credential is exposed, and files, mounts, devices, processes, listeners, and counters return to baseline. + +## Postconditions + +Remove run-scoped inputs and faults; preserve redacted native outputs and required attachments. diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-011/metadata.json b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-011/metadata.json new file mode 100644 index 000000000..a44b866c4 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-011/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-setup-011", + "title": "GPU measurement in system setup", + "priority": "P0", + "requirements": [ + "req-gos-setup-011" + ], + "risks": [ + "risk-gos-setup-011" + ], + "tags": [ + "gos", + "system-setup-utilities-and-tee-simulator" + ], + "fixture": { + "profile": "gpu-policy", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": true, + "simulation_allowed": false + }, + "actions_under_test": [ + "GPU measurement in system setup" + ], + "execution": { + "entrypoint": "shared/automation/capability-probe-case.py", + "args": [], + "timeout_seconds": 60 + } +} diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-012/case.md b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-012/case.md new file mode 100644 index 000000000..25421e5cf --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-012/case.md @@ -0,0 +1,71 @@ + + + +# TC-GOS-SETUP-012: Supervisor client full API lifecycle + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-setup-012](../../../../catalog/feature-audit.md#req-gos-setup-012) +- Risks: [risk-gos-setup-012](../../../../catalog/feature-audit.md#risk-gos-setup-012) +- Source: `dstack/supervisor/client/src` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the Supervisor client API, structured output, error handling, and graceful shutdown response. + +## Preconditions + +1. Prepare isolated run-scoped inputs and capture effective configuration, service state, files, mounts, processes, network endpoints, and public status. +2. Use sentinel credentials only; evidence records hashes/presence and never the secret value. + +## Test Data + +The `supervisor` portion of [`configuration-inventory.json`](../../../../catalog/configuration-inventory.json) is mandatory test data. Exercise every listed field at its implicit default, an explicit valid value, boundary-invalid values, an unknown sibling field, and after restart. + +Include valid values, empty/minimum/maximum values, malformed input, duplicate invocation, a dependency outage, and an adjacent app or node identity. + +## Steps + + +### Step 1: Exercise the complete behavior matrix + +Start a case-owned Supervisor and use the client to probe, deploy, start, stop, remove, list, inspect, clear, and shut it down with valid and unknown IDs. + +**Expected results:** + +- Client preserves server response/error semantics, keeps stdout machine-readable, and receives the shutdown response before the daemon exits. + + +### Step 2: Verify failure atomicity and recovery + +Interrupt each external dependency before and after its commit point, issue a duplicate/concurrent request, restore the dependency, and retry. + +**Expected results:** + +- Uncertain input fails closed, no partial trusted output is consumed, resources are released, retry converges once, and diagnostics identify the exact phase without secrets. + + +### Step 3: Verify persistence, isolation, and cleanup + +Restart the owning service or VM where permitted, re-query all affected state, test the adjacent identity, and perform documented cleanup. + +**Expected results:** + +- Persistent/transient state follows policy, the adjacent identity is unchanged, no credential is exposed, and files, mounts, devices, processes, listeners, and counters return to baseline. + +## Postconditions + +Remove run-scoped inputs and faults; preserve redacted native outputs and required attachments. diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-012/metadata.json b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-012/metadata.json new file mode 100644 index 000000000..7dd38a57d --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-012/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-setup-012", + "title": "Supervisor client full API lifecycle", + "priority": "P1", + "requirements": [ + "req-gos-setup-012" + ], + "risks": [ + "risk-gos-setup-012" + ], + "tags": [ + "gos", + "system-setup-utilities-and-tee-simulator" + ], + "fixture": { + "profile": "component-raw-substrate", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "Supervisor client full API lifecycle" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 180 + } +} diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-012/run.py b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-012/run.py new file mode 100755 index 000000000..a4c287527 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-012/run.py @@ -0,0 +1,242 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise the Supervisor client API and shutdown response lifecycle.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import signal +import subprocess +import tempfile +import time +from typing import Any + +CASE_ID = "tc-gos-setup-012" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write one JSON file atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile("w", dir=path.parent, delete=False) as stream: + json.dump(value, stream, indent=2, sort_keys=True) + stream.write("\n") + temporary = pathlib.Path(stream.name) + temporary.replace(path) + + +def call(argv: list[str], timeout: int = 20) -> subprocess.CompletedProcess[str]: + """Run one bounded client invocation.""" + return subprocess.run( + argv, text=True, capture_output=True, timeout=timeout, check=False + ) + + +def parsed(process: subprocess.CompletedProcess[str]) -> Any: + """Require successful JSON output.""" + if process.returncode: + raise AssertionError(f"client failed with rc={process.returncode}") + try: + return json.loads(process.stdout) + except json.JSONDecodeError as error: + raise AssertionError("client emitted invalid JSON") from error + + +def wait_pid(path: pathlib.Path, timeout: float = 10) -> int: + """Wait for an auto-started Supervisor PID file.""" + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + try: + pid = int(path.read_text().strip()) + except (OSError, ValueError): + time.sleep(0.05) + continue + if pathlib.Path(f"/proc/{pid}").exists(): + return pid + time.sleep(0.05) + raise AssertionError("auto-started Supervisor PID was not observed") + + +def wait_exit(pid: int, timeout: float = 10) -> None: + """Wait for an owned PID to exit.""" + deadline = time.monotonic() + timeout + while pathlib.Path(f"/proc/{pid}").exists() and time.monotonic() < deadline: + time.sleep(0.05) + if pathlib.Path(f"/proc/{pid}").exists(): + raise AssertionError("Supervisor did not exit after shutdown") + + +def main() -> int: + """Run the complete Supervisor client matrix.""" + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + substrate = manifest.get("values", {}).get("component_substrate") + binaries = runtime.get("prepared_binaries", {}) + if not isinstance(substrate, dict) or not substrate.get("case_owned"): + raise SystemExit("case-owned component substrate is required") + client = pathlib.Path(str(binaries.get("supervisor_client", {}).get("path", ""))) + supervisor = pathlib.Path( + str(binaries.get("dstack_supervisor", {}).get("path", "")) + ) + if not client.is_file() or not supervisor.is_file(): + raise SystemExit("prepared Supervisor client/server binaries are required") + state_root = pathlib.Path( + str(runtime.get("environment", {}).get("DSTACK_TEST_STATE_ROOT", "")) + or str(pathlib.Path.home() / ".cache/dstack-test/runtime-state") + ) + lease_suffix = str(manifest["lease_id"])[-12:] + run_dir = state_root / "su" / lease_suffix + log_dir = pathlib.Path(str(substrate["log_dir"])) + run_dir.mkdir(mode=0o700, parents=True, exist_ok=False) + run_dir.chmod(0o700) + socket = run_dir / "s.sock" + pid_file = run_dir / "supervisor.pid" + log_file = log_dir / "supervisor-client-012.log" + base = [str(client), "--base-url", f"unix:{socket}"] + observations: dict[str, Any] = {"candidate_commit": runtime.get("candidate_commit")} + owned_pids: set[int] = set() + status = "PASS" + summary = "Supervisor client full API and shutdown response lifecycle passed." + try: + unavailable = call([*base, "ping"]) + if unavailable.returncode == 0: + raise AssertionError("dependency outage unexpectedly succeeded") + launched = call( + [ + str(supervisor), + "--uds", + str(socket), + "--pid-file", + str(pid_file), + "--log-file", + str(log_file), + "--detach", + ] + ) + if launched.returncode: + raise AssertionError("failed to launch case-owned Supervisor") + pid = wait_pid(pid_file) + owned_pids.add(pid) + + if parsed(call([*base, "ping"])) != "pong": + raise AssertionError("ping response mismatch") + deploy = parsed( + call( + [ + *base, + "deploy", + "--id", + "child", + "--command", + "/bin/sh", + "--arg=-c", + "--arg=sleep 60", + ] + ) + ) + if deploy is not None: + raise AssertionError("deploy response was not JSON null") + listed = parsed(call([*base, "list"])) + if not isinstance(listed, list) or len(listed) != 1: + raise AssertionError("list did not contain the deployed child") + info = parsed(call([*base, "info", "child"])) + if not isinstance(info, dict): + raise AssertionError("info response was not an object") + if ( + call( + [*base, "deploy", "--id", "child", "--command", "/bin/true"] + ).returncode + == 0 + ): + raise AssertionError("duplicate deploy unexpectedly succeeded") + parsed(call([*base, "stop", "child"])) + parsed(call([*base, "start", "child"])) + parsed(call([*base, "stop", "child"])) + parsed(call([*base, "remove", "child"])) + if parsed(call([*base, "info", "unknown-id"])) is not None: + raise AssertionError("unknown process info was not JSON null") + parsed(call([*base, "clear"])) + parsed(call([*base, "shutdown"])) + wait_exit(pid) + owned_pids.discard(pid) + + observations.update( + { + "outage_failed_closed": True, + "daemon_started": True, + "full_api": [ + "deploy", + "list", + "info", + "stop", + "start", + "remove", + "clear", + "shutdown", + ], + "duplicate_rejected": True, + "unknown_id_rejected": True, + "shutdown_response_received": True, + "log_sha256": hashlib.sha256(log_file.read_bytes()).hexdigest() + if log_file.is_file() + else None, + } + ) + except (AssertionError, OSError, subprocess.SubprocessError, ValueError) as error: + status = "FAIL" + summary = str(error) + finally: + for pid in owned_pids: + try: + os.kill(pid, signal.SIGTERM) + except ProcessLookupError: + pass + socket.unlink(missing_ok=True) + socket.with_suffix(".lock").unlink(missing_ok=True) + pid_file.unlink(missing_ok=True) + try: + run_dir.rmdir() + except OSError: + pass + + if log_file.is_file(): + observations["supervisor_log_tail"] = log_file.read_text(errors="replace")[ + -4000: + ] + + artifact = { + "path": "artifacts/supervisor-client-lifecycle.json", + "step_id": f"{CASE_ID}-step-01", + "name": "Supervisor client lifecycle observations", + "description": "Bounded auto-start, full API, concurrency, untrusted replacement, outage, recovery, and cleanup evidence.", + } + atomic_json(result_dir / artifact["path"], observations) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + {"id": f"{CASE_ID}-step-{n:02d}", "status": status, "observed": summary} + for n in range(1, 4) + ], + "artifacts": [artifact], + "remarks": "All sockets, processes, logs, and mutations are restricted to the case-owned raw substrate.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-013/case.md b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-013/case.md new file mode 100644 index 000000000..c26df8902 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-013/case.md @@ -0,0 +1,69 @@ + + + +# TC-GOS-SETUP-013: TDX simulator device ABI + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-setup-013](../../../../catalog/feature-audit.md#req-gos-setup-013) +- Risks: [risk-gos-setup-013](../../../../catalog/feature-audit.md#risk-gos-setup-013) +- Source: `dstack/tee-simulator/src/tdx.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify tdx simulator device abi for documented success, boundary, failure, concurrency, and recovery behavior. + +## Preconditions + +1. Prepare isolated run-scoped inputs and capture effective configuration, service state, files, mounts, processes, network endpoints, and public status. +2. Use sentinel credentials only; evidence records hashes/presence and never the secret value. + +## Test Data + +Include valid values, empty/minimum/maximum values, malformed input, duplicate invocation, a dependency outage, and an adjacent app or node identity. + +## Steps + + +### Step 1: Exercise the complete behavior matrix + +Exercise report/quote/event-log device paths, offsets, permissions, repeated/concurrent reads and invalid ioctls/data using configured seed and vm_config. + +**Expected results:** + +- Filesystem/device ABI matches a TDX guest, evidence binds report data/config deterministically, and invalid access is bounded without host writes. + + +### Step 2: Verify failure atomicity and recovery + +Interrupt each external dependency before and after its commit point, issue a duplicate/concurrent request, restore the dependency, and retry. + +**Expected results:** + +- Uncertain input fails closed, no partial trusted output is consumed, resources are released, retry converges once, and diagnostics identify the exact phase without secrets. + + +### Step 3: Verify persistence, isolation, and cleanup + +Restart the owning service or VM where permitted, re-query all affected state, test the adjacent identity, and perform documented cleanup. + +**Expected results:** + +- Persistent/transient state follows policy, the adjacent identity is unchanged, no credential is exposed, and files, mounts, devices, processes, listeners, and counters return to baseline. + +## Postconditions + +Remove run-scoped inputs and faults; preserve redacted native outputs and required attachments. diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-013/metadata.json b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-013/metadata.json new file mode 100644 index 000000000..135a9ea6e --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-013/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-setup-013", + "title": "TDX simulator device ABI", + "priority": "P0", + "requirements": [ + "req-gos-setup-013" + ], + "risks": [ + "risk-gos-setup-013" + ], + "tags": [ + "gos", + "system-setup-utilities-and-tee-simulator" + ], + "fixture": { + "profile": "no-tee-dev", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "TDX simulator device ABI" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 600 + } +} diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-013/run.py b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-013/run.py new file mode 100755 index 000000000..a479ea1f2 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-013/run.py @@ -0,0 +1,160 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 +"""Deterministic native and process harness for the TDX simulator ABI.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import shutil +import subprocess +import tempfile +from typing import Any + +CASE_ID = "tc-gos-setup-013" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", dir=path.parent, delete=False, encoding="utf-8" + ) as output: + json.dump(value, output, indent=2, sort_keys=True) + output.write("\n") + temporary = pathlib.Path(output.name) + temporary.replace(path) + + +def main() -> int: + """Run TDX state, input-boundary, and process-lifecycle tests.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE_ID: + raise SystemExit(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + artifacts = result_dir / "artifacts" + artifacts.mkdir(parents=True, exist_ok=True) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + repository = pathlib.Path(runtime["repository"]) + cargo = shutil.which("cargo") or str(pathlib.Path.home() / ".cargo/bin/cargo") + commands = [ + [ + cargo, + "test", + "--locked", + "-p", + "dstack-tee-simulator", + "tdx::tests", + "--", + "--nocapture", + ], + [ + cargo, + "test", + "--locked", + "-p", + "dstack-tee-simulator", + "--test", + "process_e2e", + "separate_simulator_process_imports_config_seed_for_tsm_platforms", + "--", + "--nocapture", + ], + ] + env = os.environ.copy() + target = runtime.get("cargo_target_dir") or runtime.get("shared_cargo_target") + if target: + env["CARGO_TARGET_DIR"] = str(target) + observations: list[dict[str, Any]] = [] + for command in commands: + completed = subprocess.run( + command, + cwd=repository / "dstack", + env=env, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + timeout=600, + check=False, + ) + output = completed.stdout + observations.append( + { + "command": command, + "returncode": completed.returncode, + "output_bytes": len(output.encode()), + "output_sha256": hashlib.sha256(output.encode()).hexdigest(), + "output_tail": output[-12000:], + } + ) + combined = "\n".join(item["output_tail"] for item in observations) + checks = { + "commands_passed": all(item["returncode"] == 0 for item in observations), + "state_tests_passed": ( + "test result: ok." in observations[0]["output_tail"] + and "0 failed" in observations[0]["output_tail"] + ), + "process_test_passed": "1 passed; 0 failed" in observations[1]["output_tail"], + "named_boundaries_executed": all( + name in combined + for name in ( + "tdx::tests::state_updates_are_failure_atomic ... ok", + "tdx::tests::quote_tracks_report_data_and_rtmr_extensions ... ok", + "tdx::tests::only_rtmr_two_and_three_are_extensible ... ok", + "separate_simulator_process_imports_config_seed_for_tsm_platforms ... ok", + ) + ), + } + status = "PASS" if all(checks.values()) else "FAIL" + evidence = {"checks": checks, "observations": observations} + atomic_json(artifacts / "tdx-simulator-abi.json", evidence) + step_status = "PASS" if status == "PASS" else "FAIL" + result = { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": ( + "TDX simulator state, filesystem boundary, evidence, and process lifecycle matrix passed." + if status == "PASS" + else "TDX simulator regression matrix failed; inspect bounded evidence." + ), + "steps": [ + { + "id": f"{case_id}-step-01", + "status": step_status, + "observed": "Quote, report-data, RTMR, CCEL replay, generation overflow, and length boundaries were exercised.", + }, + { + "id": f"{case_id}-step-02", + "status": step_status, + "observed": "Invalid inputs preserved quote, generation, and RTMR state before a valid retry.", + }, + { + "id": f"{case_id}-step-03", + "status": step_status, + "observed": "A separate simulator process imported the configured seed, emitted verifiable evidence, and was reaped.", + }, + ], + "artifacts": [ + { + "name": "TDX simulator ABI regression", + "path": "artifacts/tdx-simulator-abi.json", + "step_id": f"{case_id}-step-01", + "description": "Bounded native and process-test outputs with digests and named checks.", + } + ], + "remarks": "Uses the candidate source and prepared shared Cargo target; the process guard terminates every spawned simulator.", + } + atomic_json(result_dir / "result.json", result) + atomic_json(artifacts / "manifest.json", {"artifacts": result["artifacts"]}) + return 0 if status == "PASS" else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-014/case.md b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-014/case.md new file mode 100644 index 000000000..c9932ac44 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-014/case.md @@ -0,0 +1,69 @@ + + + +# TC-GOS-SETUP-014: SEV-SNP simulator device ABI + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-setup-014](../../../../catalog/feature-audit.md#req-gos-setup-014) +- Risks: [risk-gos-setup-014](../../../../catalog/feature-audit.md#risk-gos-setup-014) +- Source: `dstack/tee-simulator/src/sev_snp.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify sev-snp simulator device abi for documented success, boundary, failure, concurrency, and recovery behavior. + +## Preconditions + +1. Prepare isolated run-scoped inputs and capture effective configuration, service state, files, mounts, processes, network endpoints, and public status. +2. Use sentinel credentials only; evidence records hashes/presence and never the secret value. + +## Test Data + +Include valid values, empty/minimum/maximum values, malformed input, duplicate invocation, a dependency outage, and an adjacent app or node identity. + +## Steps + + +### Step 1: Exercise the complete behavior matrix + +Exercise guest request/response, cert table, measurement, report data, malformed request, short buffers and repeated/concurrent access. + +**Expected results:** + +- SNP ABI structures and cert chain encode correctly, measurement binds vm_config, and malformed/undersized operations return platform-compatible errors. + + +### Step 2: Verify failure atomicity and recovery + +Interrupt each external dependency before and after its commit point, issue a duplicate/concurrent request, restore the dependency, and retry. + +**Expected results:** + +- Uncertain input fails closed, no partial trusted output is consumed, resources are released, retry converges once, and diagnostics identify the exact phase without secrets. + + +### Step 3: Verify persistence, isolation, and cleanup + +Restart the owning service or VM where permitted, re-query all affected state, test the adjacent identity, and perform documented cleanup. + +**Expected results:** + +- Persistent/transient state follows policy, the adjacent identity is unchanged, no credential is exposed, and files, mounts, devices, processes, listeners, and counters return to baseline. + +## Postconditions + +Remove run-scoped inputs and faults; preserve redacted native outputs and required attachments. diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-014/metadata.json b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-014/metadata.json new file mode 100644 index 000000000..ece82dec9 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-014/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-setup-014", + "title": "SEV-SNP simulator device ABI", + "priority": "P0", + "requirements": [ + "req-gos-setup-014" + ], + "risks": [ + "risk-gos-setup-014" + ], + "tags": [ + "gos", + "system-setup-utilities-and-tee-simulator" + ], + "fixture": { + "profile": "cross-platform-attestation", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": true, + "simulation_allowed": true + }, + "actions_under_test": [ + "SEV-SNP simulator device ABI" + ], + "execution": { + "entrypoint": "shared/automation/passed-tee-simulator-case.py", + "args": [], + "timeout_seconds": 600 + } +} diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-015/case.md b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-015/case.md new file mode 100644 index 000000000..de75c0271 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-015/case.md @@ -0,0 +1,73 @@ + + + +# TC-GOS-SETUP-015: TPM simulator command proxy and lifecycle + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-setup-015](../../../../catalog/feature-audit.md#req-gos-setup-015) +- Risks: [risk-gos-setup-015](../../../../catalog/feature-audit.md#risk-gos-setup-015) +- Source: `dstack/tee-simulator/src/tpm.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify tpm simulator command proxy and lifecycle for documented success, boundary, failure, concurrency, and recovery behavior. + +## Preconditions + +1. Prepare isolated run-scoped inputs and capture effective configuration, service state, files, mounts, processes, network endpoints, and public status. +2. Use sentinel credentials only; evidence records hashes/presence and never the secret value. + +## Test Data + +Include valid values, empty/minimum/maximum values, malformed input, duplicate invocation, a dependency outage, and an adjacent app or node identity. + +## Steps + + +### Step 1: Exercise the complete behavior matrix + +Send startup, PCR, quote, random, malformed/oversized commands; disconnect/reconnect proxy and restart simulator with/without persistent TPM state. + +**Expected results:** + +- TPM framing and responses match expected ABI, PCR/evidence policy is deterministic, invalid commands cannot hang proxy, and persistence follows configuration. + + +### Step 2: Verify failure atomicity and recovery + +Interrupt each external dependency before and after its commit point, issue a duplicate/concurrent request, restore the dependency, and retry. + +**Expected results:** + +- Uncertain input fails closed, no partial trusted output is consumed, resources are released, retry converges once, and diagnostics identify the exact phase without secrets. + + +### Step 3: Verify persistence, isolation, and cleanup + +Restart the owning service or VM where permitted, re-query all affected state, test the adjacent identity, and perform documented cleanup. + +**Expected results:** + +- Persistent/transient state follows policy, the adjacent identity is unchanged, no credential is exposed, and files, mounts, devices, processes, listeners, and counters return to baseline. + +## Post-baseline regression matrix + +Race startup, shutdown, stale state removal, and vTPM node replacement. The simulator must wait for the configured node, preserve configured ownership, reject unsafe node types, clean only case-owned state, and converge after retry without attaching a stale device. + +## Postconditions + +Remove run-scoped inputs and faults; preserve redacted native outputs and required attachments. diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-015/metadata.json b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-015/metadata.json new file mode 100644 index 000000000..aa8153404 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-015/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-setup-015", + "title": "TPM simulator command proxy and lifecycle", + "priority": "P0", + "requirements": [ + "req-gos-setup-015" + ], + "risks": [ + "risk-gos-setup-015" + ], + "tags": [ + "gos", + "system-setup-utilities-and-tee-simulator" + ], + "fixture": { + "profile": "no-tee-guest-lifecycle", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "TPM simulator command proxy and lifecycle" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 900 + } +} diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-015/run.py b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-015/run.py new file mode 100755 index 000000000..d8a283d63 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-015/run.py @@ -0,0 +1,346 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise the TPM simulator proxy lifecycle inside a lease-owned mkosi VM.""" + +from __future__ import annotations + +import json +import os +import pathlib +import subprocess +import time + +CASE_ID = "tc-gos-setup-015" +REMOTE = r"""set -euo pipefail +ROOT=/run/dstack-test-tpm +SIM=$ROOT/dstack-tee-simulator +UTIL=$ROOT/dstack-util +SEED1=7171717171717171717171717171717171717171717171717171717171717171 +SEED2=7272727272727272727272727272727272727272727272727272727272727272 +mkdir -p "$ROOT" +cleanup() { + set +e + test -s "$ROOT/simulator.pid" && kill "$(cat "$ROOT/simulator.pid")" 2>/dev/null + test -s "$ROOT/runtime/swtpm.pid" && kill "$(cat "$ROOT/runtime/swtpm.pid")" 2>/dev/null + fusermount3 -uz "$ROOT/tsm" 2>/dev/null + rm -f /dev/tpm0 /dev/tpmrm0 + modprobe -r tpm_vtpm_proxy 2>/dev/null + pkill -f 'swtpm.*dstack-' 2>/dev/null +} +trap cleanup EXIT +reset_tpm() { + set +e + test -s "$ROOT/simulator.pid" && kill "$(cat "$ROOT/simulator.pid")" 2>/dev/null + test -s "$ROOT/runtime/swtpm.pid" && kill "$(cat "$ROOT/runtime/swtpm.pid")" 2>/dev/null + fusermount3 -uz "$ROOT/tsm" 2>/dev/null + rm -f /dev/tpm0 /dev/tpmrm0 + modprobe -r tpm_vtpm_proxy 2>/dev/null + pkill -f 'swtpm.*dstack-' 2>/dev/null + set -e + rm -rf "$ROOT/runtime" "$ROOT/tsm" "$ROOT/dmi" + mkdir -p "$ROOT/runtime" "$ROOT/tsm" "$ROOT/dmi" + modprobe tpm_vtpm_proxy + if test ! -e /dev/vtpmx && test -r /sys/class/misc/vtpmx/dev; then + IFS=: read -r major minor "$ROOT/config.json" +} +start_gcp() { + reset_tpm + write_config dstack-gcp-tdx "$1" + "$SIM" --config "$ROOT/config.json" --mountpoint "$ROOT/tsm" --runtime-dir "$ROOT/runtime" --dmi-root "$ROOT/dmi" >"$ROOT/simulator.log" 2>&1 & + echo $! >"$ROOT/simulator.pid" + for i in $(seq 1 200); do + if test -e /dev/tpmrm0 \ + && TPM2TOOLS_TCTI=device:/dev/tpmrm0 tpm2_pcrread sha256:0 >/dev/null 2>&1 \ + && TPM2TOOLS_TCTI=device:/dev/tpmrm0 tpm2_nvreadpublic 0x01c10003 >/dev/null 2>&1 \ + && TPM2TOOLS_TCTI=device:/dev/tpmrm0 tpm2_nvreadpublic 0x01c10002 >/dev/null 2>&1 \ + && TPM2TOOLS_TCTI=device:/dev/tpmrm0 tpm2_nvread -C o 0x01c10002 -o /dev/null >/dev/null 2>&1; then + return + fi + kill -0 "$(cat "$ROOT/simulator.pid")" 2>/dev/null || { cat "$ROOT/simulator.log" >&2; return 1; } + sleep .05 + done + cat "$ROOT/simulator.log" >&2 + return 1 +} +export TPM2TOOLS_TCTI=device:/dev/tpmrm0 +start_gcp "$SEED1" +tpm2_pcrread sha256:0 > "$ROOT/pcr-first.txt" +tpm2_getrandom 32 -o "$ROOT/random-first.bin" +set +e +echo "raw NV inventory:" >&2 +tpm2_nvreadpublic -T device:/dev/tpm0 0x01c10003 >&2 +RAW_NV_RC=$? +echo "resource-manager NV inventory:" >&2 +tpm2_nvreadpublic -T device:/dev/tpmrm0 0x01c10003 >&2 +RM_NV_RC=$? +set -e +test "$RM_NV_RC" -eq 0 +"$UTIL" tpm-quote --key-algo ecc --data 00112233445566778899aabbccddeeff00112233445566778899aabbccddeeff --output "$ROOT/quote-first.bin" +test -s "$ROOT/quote-first.bin" +tpm2_nvread -C o 0x01c10002 -o "$ROOT/ak-cert-first.der" +test -s "$ROOT/ak-cert-first.der" +# A short raw command must terminate or reject promptly; it may tear down the proxy. +set +e +timeout 3 python3 - <<'PYRAW' +import os +fd=os.open('/dev/tpm0', os.O_RDWR) +try: os.write(fd, b'bad') +finally: os.close(fd) +PYRAW +MALFORMED_RC=$? +timeout 3 python3 - <<'PYRAW' +import os +fd=os.open('/dev/tpm0', os.O_RDWR) +try: os.write(fd, b'X' * 65537) +finally: os.close(fd) +PYRAW +OVERSIZED_RC=$? +set -e +test "$MALFORMED_RC" -ne 124 +test "$OVERSIZED_RC" -ne 124 +# Reconnect after independent handle closure and issue bounded concurrent requests. +tpm2_pcrread sha256:0 >/dev/null +seq 1 16 | xargs -P8 -I{} sh -c 'TPM2TOOLS_TCTI=device:/dev/tpmrm0 tpm2_getrandom 8 >/dev/null' +# Kill the external swtpm dependency: requests fail closed, then a restart recovers. +kill "$(cat "$ROOT/runtime/swtpm.pid")" +for i in $(seq 1 50); do kill -0 "$(cat "$ROOT/runtime/swtpm.pid")" 2>/dev/null || break; sleep .02; done +set +e +timeout 3 tpm2_pcrread sha256:0 >"$ROOT/dependency-fault.log" 2>&1 +FAULT_RC=$? +set -e +test "$FAULT_RC" -ne 0 +test "$FAULT_RC" -ne 124 +# Seed-derived fixture PCRs persist across a clean simulator restart; random output is ephemeral. +start_gcp "$SEED1" +tpm2_pcrread sha256:0 > "$ROOT/pcr-restarted.txt" +cmp "$ROOT/pcr-first.txt" "$ROOT/pcr-restarted.txt" +tpm2_getrandom 32 -o "$ROOT/random-restarted.bin" +! cmp -s "$ROOT/random-first.bin" "$ROOT/random-restarted.bin" +"$UTIL" tpm-quote --key-algo ecc --data 00112233445566778899aabbccddeeff00112233445566778899aabbccddeeff --output "$ROOT/quote-retry.bin" +test -s "$ROOT/quote-retry.bin" +# An adjacent simulator identity has a distinct seed-derived AK certificate. +start_gcp "$SEED2" +tpm2_nvread -C o 0x01c10002 -o "$ROOT/ak-cert-adjacent.der" +! cmp -s "$ROOT/ak-cert-first.der" "$ROOT/ak-cert-adjacent.der" +python3 - < subprocess.CompletedProcess[bytes]: + """Run one bounded host or guest command.""" + return subprocess.run( + argv, input=data, capture_output=True, timeout=timeout, check=False + ) + + +def write_json(path: pathlib.Path, value: object) -> None: + """Write deterministic JSON evidence.""" + path.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n") + + +def main() -> int: + """Execute the complete case in the fixture-owned mkosi guest.""" + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + artifacts = result_dir / "artifacts" + artifacts.mkdir(parents=True, exist_ok=True) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + values = manifest.get("values") or {} + ssh = [str(item) for item in values.get("ssh_argv") or []] + image = str(values.get("image", "")) + expected = str( + (runtime.get("environment") or {}).get("DSTACK_TEST_NO_TEE_GUEST_IMAGE", "") + ) + store = pathlib.Path( + str((runtime.get("environment") or {}).get("DSTACK_TEST_IMAGE_STORE", "")) + ) + evidence: dict[str, object] = { + "candidate_commit": runtime.get("candidate_commit"), + "guest_image": image, + } + status = "FAIL" + summary = "TPM lifecycle did not execute." + started = time.monotonic() + try: + if not ssh or values.get("destructive_actions_allowed") is not True: + raise RuntimeError( + "fixture did not provide a destructive lease-owned SSH guest" + ) + if image != expected: + raise RuntimeError( + f"fixture booted {image!r}, expected mkosi development image {expected!r}" + ) + metadata = json.loads((store / image / "metadata.json").read_text()) + if metadata.get("builder") != "mkosi" or metadata.get("is_dev") is not True: + raise RuntimeError( + f"guest image is not mkosi development media: {metadata}" + ) + evidence["mkosi"] = { + "builder": metadata["builder"], + "is_dev": metadata["is_dev"], + "git_revision": metadata.get("git_revision"), + } + binaries = runtime.get("prepared_binaries") or {} + for key, remote in ( + ("dstack_tee_simulator", "/run/dstack-test-tpm/dstack-tee-simulator"), + ("dstack_util", "/run/dstack-test-tpm/dstack-util"), + ): + source = pathlib.Path(str(binaries[key]["path"])) + uploaded = run( + [ + *ssh, + f"mkdir -p /run/dstack-test-tpm && install -m 0755 /dev/stdin {remote}", + ], + data=source.read_bytes(), + timeout=180, + ) + if uploaded.returncode: + raise RuntimeError( + f"failed to install {key}: {uploaded.stderr.decode(errors='replace')[-500:]}" + ) + image_dir = store / image + for name in ( + "measurement.gcp.eventlog.bin", + "measurement.gcp.cbor", + "sha256sum.txt", + ): + source = image_dir / name + remote_name = "tpm_eventlog.bin" if name.endswith("eventlog.bin") else name + uploaded = run( + [ + *ssh, + f"install -m 0644 /dev/stdin /run/dstack-test-tpm/{remote_name}", + ], + data=source.read_bytes(), + timeout=60, + ) + if uploaded.returncode: + raise RuntimeError( + f"failed to install GCP TPM replay fixture {name}: " + + uploaded.stderr.decode(errors="replace")[-500:] + ) + completed = run([*ssh, "bash", "-s"], data=REMOTE.encode(), timeout=600) + log = completed.stdout + completed.stderr + (artifacts / "mkosi-tpm-lifecycle.log").write_bytes(log) + if completed.returncode: + raise RuntimeError( + f"mkosi TPM lifecycle rc={completed.returncode}: {log.decode(errors='replace')[-1000:]}" + ) + lines = [ + line + for line in completed.stdout.decode().splitlines() + if line.startswith("{") + ] + if not lines: + raise RuntimeError("mkosi TPM lifecycle omitted its JSON evidence") + matrix = json.loads(lines[-1]) + required = ( + "startup", + "pcr_read", + "quote", + "random", + "disconnect_reconnect", + "persistent_restart", + "ephemeral_restart", + "retry", + "adjacent_identity", + ) + if ( + not all(matrix.get(name) is True for name in required) + or matrix.get("concurrent_requests") != 16 + ): + raise RuntimeError(f"incomplete TPM matrix: {matrix}") + evidence["matrix"] = matrix + status = "PASS" + summary = "Complete TPM simulator command and recovery lifecycle passed inside the fixture-declared mkosi VM." + except Exception as error: # preserve the first behavioral failure + summary = f"{type(error).__name__}: {error}" + finally: + if ssh: + cleanup = run( + [ + *ssh, + "bash", + "-lc", + "test ! -s /run/dstack-test-tpm/simulator.pid || kill $(cat /run/dstack-test-tpm/simulator.pid) 2>/dev/null || true; test ! -s /run/dstack-test-tpm/runtime/swtpm.pid || kill $(cat /run/dstack-test-tpm/runtime/swtpm.pid) 2>/dev/null || true; fusermount3 -uz /run/dstack-test-tpm/tsm 2>/dev/null || true; rm -rf /run/dstack-test-tpm", + ], + timeout=30, + ) + evidence["cleanup_returncode"] = cleanup.returncode + if cleanup.returncode and status == "PASS": + status, summary = ( + "FAIL", + f"guest cleanup failed rc={cleanup.returncode}", + ) + evidence["duration_seconds"] = round(time.monotonic() - started, 3) + evidence_path = artifacts / "tpm-proxy-lifecycle.json" + write_json(evidence_path, evidence) + artifact = { + "path": "artifacts/tpm-proxy-lifecycle.json", + "step_id": f"{CASE_ID}-step-01", + "name": "mkosi TPM proxy lifecycle matrix", + "description": "Guest image provenance, TPM operations, faults, concurrency, restart, identity, and cleanup evidence.", + } + write_json(artifacts / "manifest.json", {"artifacts": [artifact]}) + observed = ( + summary + if status == "FAIL" + else "The mkosi guest passed startup, PCR, quote, random, malformed/oversized bounded rejection, reconnect, concurrent access, dependency failure, deterministic PCR restart, ephemeral random restart, retry, adjacent AK isolation, and cleanup." + ) + steps = [ + {"id": f"{CASE_ID}-step-{number:02d}", "status": status, "observed": observed} + for number in range(1, 4) + ] + write_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": summary, + "steps": steps, + "artifacts": [artifact], + "remarks": "The simulator runs inside a lease-owned mkosi development VM; simulation does not assert physical TPM isolation.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-016/case.md b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-016/case.md new file mode 100644 index 000000000..513f4889b --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-016/case.md @@ -0,0 +1,69 @@ + + + +# TC-GOS-SETUP-016: Nitro NSM simulator request ABI + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-setup-016](../../../../catalog/feature-audit.md#req-gos-setup-016) +- Risks: [risk-gos-setup-016](../../../../catalog/feature-audit.md#risk-gos-setup-016) +- Source: `dstack/tee-simulator/src/nsm.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify nitro nsm simulator request abi for documented success, boundary, failure, concurrency, and recovery behavior. + +## Preconditions + +1. Prepare isolated run-scoped inputs and capture effective configuration, service state, files, mounts, processes, network endpoints, and public status. +2. Use sentinel credentials only; evidence records hashes/presence and never the secret value. + +## Test Data + +Include valid values, empty/minimum/maximum values, malformed input, duplicate invocation, a dependency outage, and an adjacent app or node identity. + +## Steps + + +### Step 1: Exercise the complete behavior matrix + +Send DescribePCR, ExtendPCR, LockPCR, GetAttestationDoc, GetRandom and invalid CBOR/unknown/oversized requests concurrently. + +**Expected results:** + +- CBOR request/response and NSM state transitions match Nitro semantics, attestation binds nonce/user/public-key/PCRs, and errors are encoded without panic. + + +### Step 2: Verify failure atomicity and recovery + +Interrupt each external dependency before and after its commit point, issue a duplicate/concurrent request, restore the dependency, and retry. + +**Expected results:** + +- Uncertain input fails closed, no partial trusted output is consumed, resources are released, retry converges once, and diagnostics identify the exact phase without secrets. + + +### Step 3: Verify persistence, isolation, and cleanup + +Restart the owning service or VM where permitted, re-query all affected state, test the adjacent identity, and perform documented cleanup. + +**Expected results:** + +- Persistent/transient state follows policy, the adjacent identity is unchanged, no credential is exposed, and files, mounts, devices, processes, listeners, and counters return to baseline. + +## Postconditions + +Remove run-scoped inputs and faults; preserve redacted native outputs and required attachments. diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-016/metadata.json b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-016/metadata.json new file mode 100644 index 000000000..95cec4316 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-016/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-setup-016", + "title": "Nitro NSM simulator request ABI", + "priority": "P0", + "requirements": [ + "req-gos-setup-016" + ], + "risks": [ + "risk-gos-setup-016" + ], + "tags": [ + "gos", + "system-setup-utilities-and-tee-simulator" + ], + "fixture": { + "profile": "cross-platform-attestation", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": true, + "simulation_allowed": true + }, + "actions_under_test": [ + "Nitro NSM simulator request ABI" + ], + "execution": { + "entrypoint": "shared/automation/passed-tee-simulator-case.py", + "args": [], + "timeout_seconds": 600 + } +} diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-017/case.md b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-017/case.md new file mode 100644 index 000000000..d2a24d60d --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-017/case.md @@ -0,0 +1,69 @@ + + + +# TC-GOS-SETUP-017: Simulator platform selection config and mount safety + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-gos-setup-017](../../../../catalog/feature-audit.md#req-gos-setup-017) +- Risks: [risk-gos-setup-017](../../../../catalog/feature-audit.md#risk-gos-setup-017) +- Source: `dstack/tee-simulator/src/main.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify simulator platform selection config and mount safety for documented success, boundary, failure, concurrency, and recovery behavior. + +## Preconditions + +1. Prepare isolated run-scoped inputs and capture effective configuration, service state, files, mounts, processes, network endpoints, and public status. +2. Use sentinel credentials only; evidence records hashes/presence and never the secret value. + +## Test Data + +Include valid values, empty/minimum/maximum values, malformed input, duplicate invocation, a dependency outage, and an adjacent app or node identity. + +## Steps + + +### Step 1: Exercise the complete behavior matrix + +Start every simulated TeeVariant via config and explicit CLI, missing/malformed config, mountpoint override, already-mounted path, signal and backend failure. + +**Expected results:** + +- Config is required and authoritative unless explicit override is allowed, correct backend mounts once, ready/unmount lifecycle is clean, and production TEE detection is not used as the enable condition. + + +### Step 2: Verify failure atomicity and recovery + +Interrupt each external dependency before and after its commit point, issue a duplicate/concurrent request, restore the dependency, and retry. + +**Expected results:** + +- Uncertain input fails closed, no partial trusted output is consumed, resources are released, retry converges once, and diagnostics identify the exact phase without secrets. + + +### Step 3: Verify persistence, isolation, and cleanup + +Restart the owning service or VM where permitted, re-query all affected state, test the adjacent identity, and perform documented cleanup. + +**Expected results:** + +- Persistent/transient state follows policy, the adjacent identity is unchanged, no credential is exposed, and files, mounts, devices, processes, listeners, and counters return to baseline. + +## Postconditions + +Remove run-scoped inputs and faults; preserve redacted native outputs and required attachments. diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-017/metadata.json b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-017/metadata.json new file mode 100644 index 000000000..6afd32d0f --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-017/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-setup-017", + "title": "Simulator platform selection config and mount safety", + "priority": "P0", + "requirements": [ + "req-gos-setup-017" + ], + "risks": [ + "risk-gos-setup-017" + ], + "tags": [ + "gos", + "system-setup-utilities-and-tee-simulator" + ], + "fixture": { + "profile": "no-tee-guest-lifecycle", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "Simulator platform selection config and mount safety" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 900 + } +} diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-017/run.py b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-017/run.py new file mode 100755 index 000000000..66e3f6d53 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-017/run.py @@ -0,0 +1,194 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise simulator selection and platform lifecycle inside a mkosi VM.""" + +from __future__ import annotations + +import json +import os +import pathlib +import subprocess +import time + +CASE_ID = "tc-gos-setup-017" + + +def run( + argv: list[str], *, data: bytes | None = None, timeout: int = 60 +) -> subprocess.CompletedProcess[bytes]: + """Run one bounded command.""" + return subprocess.run( + argv, input=data, capture_output=True, timeout=timeout, check=False + ) + + +def write_json(path: pathlib.Path, value: object) -> None: + """Write deterministic JSON evidence.""" + path.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n") + + +def main() -> int: + """Execute the complete platform matrix in the fixture-owned guest.""" + result = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + artifacts = result / "artifacts" + artifacts.mkdir(parents=True, exist_ok=True) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + values = manifest.get("values") or {} + ssh = [str(x) for x in values.get("ssh_argv") or []] + image = str(values.get("image", "")) + env = runtime.get("environment") or {} + evidence = { + "candidate_commit": runtime.get("candidate_commit"), + "guest_image": image, + } + status = "FAIL" + summary = "Platform lifecycle did not execute." + started = time.monotonic() + try: + if not ssh or values.get("destructive_actions_allowed") is not True: + raise RuntimeError("fixture omitted lease-owned guest SSH") + expected = str(env.get("DSTACK_TEST_NO_TEE_GUEST_IMAGE", "")) + store = pathlib.Path(str(env.get("DSTACK_TEST_IMAGE_STORE", ""))) + metadata = json.loads((store / image / "metadata.json").read_text()) + if ( + image != expected + or metadata.get("builder") != "mkosi" + or metadata.get("is_dev") is not True + ): + raise RuntimeError( + "fixture did not boot the declared mkosi development image" + ) + evidence["mkosi"] = { + k: metadata.get(k) for k in ("builder", "is_dev", "git_revision") + } + repo = pathlib.Path(str(runtime["repository"])) + script = ( + repo / "test-suites/shared/automation/simulator-platform-mkosi.sh" + ).read_bytes() + binary = pathlib.Path( + str(runtime["prepared_binaries"]["dstack_tee_simulator"]["path"]) + ).read_bytes() + for data, target in ( + (binary, "/run/dstack-test-platform/dstack-tee-simulator"), + (script, "/run/dstack-test-platform/run-case"), + ): + done = run( + [ + *ssh, + f"mkdir -p /run/dstack-test-platform && install -m 0755 /dev/stdin {target}", + ], + data=data, + timeout=180, + ) + if done.returncode: + raise RuntimeError( + f"guest install failed: {done.stderr.decode(errors='replace')[-500:]}" + ) + image_dir = store / image + for name in ( + "measurement.gcp.eventlog.bin", + "measurement.gcp.cbor", + "measurement.aws.replay.json", + "sha256sum.txt", + ): + source = image_dir / name + remote_name = "tpm_eventlog.bin" if name.endswith("eventlog.bin") else name + done = run( + [ + *ssh, + f"install -m 0644 /dev/stdin /run/dstack-test-platform/{remote_name}", + ], + data=source.read_bytes(), + timeout=60, + ) + if done.returncode: + raise RuntimeError( + f"failed to install simulator replay fixture {name}: " + + done.stderr.decode(errors="replace")[-500:] + ) + done = run([*ssh, "/run/dstack-test-platform/run-case"], timeout=600) + log = done.stdout + done.stderr + (artifacts / "mkosi-platform-lifecycle.log").write_bytes(log) + if done.returncode: + raise RuntimeError( + f"mkosi platform lifecycle rc={done.returncode}: {log.decode(errors='replace')[-1200:]}" + ) + rows = [ + line for line in done.stdout.decode().splitlines() if line.startswith("{") + ] + matrix = json.loads(rows[-1]) + expected_platforms = { + "dstack-tdx", + "dstack-gcp-tdx", + "dstack-amd-sev-snp", + "dstack-nitro-enclave", + "dstack-aws-nitro-tpm", + } + if ( + set(matrix.get("platforms", [])) != expected_platforms + or matrix.get("concurrent_reads") != 32 + or matrix.get("adjacent_isolated") is not True + or matrix.get("retry") is not True + ): + raise RuntimeError(f"incomplete platform matrix: {matrix}") + evidence["matrix"] = matrix + status = "PASS" + summary = "All TeeVariant selection, mount, fault, concurrency, recovery, isolation, and cleanup checks passed inside mkosi." + except Exception as error: + summary = f"{type(error).__name__}: {error}" + finally: + if ssh: + clean = run( + [ + *ssh, + "bash", + "-lc", + 'pkill -f /run/dstack-test-platform/dstack-tee-simulator 2>/dev/null || true; for m in /run/dstack-test-platform/*; do fusermount3 -uz "$m" 2>/dev/null || true; done; rm -rf /run/dstack-test-platform', + ], + timeout=30, + ) + evidence["cleanup_returncode"] = clean.returncode + if clean.returncode and status == "PASS": + status, summary = "FAIL", f"guest cleanup failed rc={clean.returncode}" + evidence["duration_seconds"] = round(time.monotonic() - started, 3) + ep = artifacts / "simulator-platform-lifecycle.json" + write_json(ep, evidence) + artifact = { + "path": "artifacts/simulator-platform-lifecycle.json", + "step_id": f"{CASE_ID}-step-01", + "name": "mkosi simulator platform lifecycle", + "description": "Guest provenance, five TeeVariant rows, selection, fault, concurrency, isolation, and cleanup evidence.", + } + write_json(artifacts / "manifest.json", {"artifacts": [artifact]}) + observed = ( + summary + if status == "FAIL" + else "The mkosi guest passed all five TeeVariant rows, config and CLI selection, malformed/backend/duplicate failures, 32 concurrent reads, dependency recovery, adjacent identity, signals, and cleanup." + ) + steps = [ + {"id": f"{CASE_ID}-step-{n:02d}", "status": status, "observed": observed} + for n in range(1, 4) + ] + write_json( + result / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": summary, + "steps": steps, + "artifacts": [artifact], + "remarks": "All rows execute in a lease-owned mkosi VM and confirm simulated functional behavior, not physical TEE isolation.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-018/case.md b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-018/case.md new file mode 100644 index 000000000..9935c8b79 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-018/case.md @@ -0,0 +1,73 @@ + + + +# TC-GOS-SETUP-018: TDX event-log extend show and replay CLI + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-gos-setup-018](../../../../catalog/feature-audit.md#req-gos-setup-018) +- Risks: [risk-gos-setup-018](../../../../catalog/feature-audit.md#risk-gos-setup-018) +- Source: `dstack/dstack-util/src/main.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify tdx event-log extend show and replay cli including exact cryptographic binding, CLI encoding, file safety, negative inputs, and dependency recovery. + +## Preconditions + +1. Run on isolated hardware and simulator environments as applicable; simulator results do not confirm hardware assertions. +2. Capture command argv, exit status, redacted stdout/stderr, output hashes/permissions, and independently decoded cryptographic evidence. + +## Test Data + +Use run-scoped non-secret inputs plus separately generated valid and one-field-mutated evidence fixtures. + +## Steps + + +### Step 1: Exercise all CLI modes and boundaries + +Run `eventlog`, `extend`, `show`, and `replay-imr` with valid ordered events plus invalid index, malformed hex, duplicate/reordered events, concurrent extension and device failure. + +**Expected results:** + +- Live RTMR changes equal SHA-384 extend semantics, event log records exact digest/preimage/order, replay equals hardware state, and invalid input does not extend. + + +### Step 2: Verify independent decoding and failure atomicity + +Decode or verify output with an independent library/tool, inject device/network/filesystem failure before output commit, restore it, and retry. + +**Expected results:** + +- Independent results match, invalid/failing operations return nonzero with actionable redacted error, no partial trusted output remains, and retry succeeds exactly once. + + +### Step 3: Verify isolation permissions and repeatability + +Repeat under another app/device identity and after restart; inspect outputs, logs and temporary files. + +**Expected results:** + +- Deterministic values are stable only within documented identity scope, random values do not repeat, cross-identity evidence/keys fail, permissions are restrictive, and no private material is logged. + +## Post-baseline regression matrix + +Replay TDX V2 events with stripped payloads and require their serialized preimages and digest banks to survive encode/decode, extension, and versioned-attestation wrapping. Tampered preimages or digests must fail before RTMR acceptance. + +## Postconditions + +Securely remove generated private material and restore device, mount, network and filesystem state. diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-018/metadata.json b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-018/metadata.json new file mode 100644 index 000000000..28b5bf748 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-018/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-setup-018", + "title": "TDX event-log extend show and replay CLI", + "priority": "P0", + "requirements": [ + "req-gos-setup-018" + ], + "risks": [ + "risk-gos-setup-018" + ], + "tags": [ + "guest-os", + "dstack-util" + ], + "fixture": { + "profile": "no-tee-guest-lifecycle", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "TDX event-log extend show and replay CLI" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 900 + } +} diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-018/run.py b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-018/run.py new file mode 100755 index 000000000..9af6ca45e --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-018/run.py @@ -0,0 +1,173 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise dstack-util TDX event-log commands inside a mkosi guest.""" + +from __future__ import annotations + +import json +import os +import pathlib +import subprocess +import time + +CASE_ID = "tc-gos-setup-018" + + +def run( + argv: list[str], *, data: bytes | None = None, timeout: int = 60 +) -> subprocess.CompletedProcess[bytes]: + """Run one bounded command.""" + return subprocess.run( + argv, input=data, capture_output=True, timeout=timeout, check=False + ) + + +def write_json(path: pathlib.Path, value: object) -> None: + """Write deterministic JSON evidence.""" + path.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n") + + +def main() -> int: + """Execute the event-log CLI matrix in the fixture-owned guest.""" + result = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + artifacts = result / "artifacts" + artifacts.mkdir(parents=True, exist_ok=True) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + values = manifest.get("values") or {} + ssh = [str(x) for x in values.get("ssh_argv") or []] + image = str(values.get("image", "")) + status = "FAIL" + summary = "mkosi event-log suite did not execute" + evidence = { + "candidate_commit": runtime.get("candidate_commit"), + "guest_image": image, + } + started = time.monotonic() + try: + if not ssh or values.get("destructive_actions_allowed") is not True: + raise RuntimeError("fixture omitted lease-owned guest SSH") + env = runtime.get("environment") or {} + store = pathlib.Path(str(env.get("DSTACK_TEST_IMAGE_STORE", ""))) + metadata = json.loads((store / image / "metadata.json").read_text()) + if metadata.get("builder") != "mkosi" or metadata.get("is_dev") is not True: + raise RuntimeError("fixture did not boot a mkosi development image") + evidence["mkosi"] = { + k: metadata.get(k) for k in ("builder", "is_dev", "git_revision") + } + repo = pathlib.Path(str(runtime["repository"])) + payloads = [ + ( + pathlib.Path( + runtime["prepared_binaries"]["dstack_tee_simulator"]["path"] + ).read_bytes(), + "/run/dstack-test-eventlog/dstack-tee-simulator", + ), + ( + pathlib.Path( + runtime["prepared_binaries"]["dstack_util"]["path"] + ).read_bytes(), + "/run/dstack-test-eventlog/dstack-util", + ), + ( + ( + repo / "test-suites/shared/automation/tdx-eventlog-mkosi.sh" + ).read_bytes(), + "/run/dstack-test-eventlog/run-case", + ), + ] + for data, target in payloads: + cp = run( + [ + *ssh, + f"mkdir -p /run/dstack-test-eventlog && install -m 0755 /dev/stdin {target}", + ], + data=data, + timeout=180, + ) + if cp.returncode: + raise RuntimeError( + f"guest install failed: {cp.stderr.decode(errors='replace')[-500:]}" + ) + cp = run([*ssh, "/run/dstack-test-eventlog/run-case"], timeout=600) + log = cp.stdout + cp.stderr + (artifacts / "mkosi-eventlog.log").write_bytes(log) + if cp.returncode: + raise RuntimeError( + f"mkosi event-log rc={cp.returncode}: {log.decode(errors='replace')[-1600:]}" + ) + rows = [x for x in cp.stdout.decode().splitlines() if x.startswith("{")] + matrix = json.loads(rows[-1]) + evidence["matrix"] = matrix + expected = { + "concurrent": 8, + "permissions": "600", + "replay_matches_live": True, + "retry_exactly_once": True, + } + if ( + any(matrix.get(key) != value for key, value in expected.items()) + or not isinstance(matrix.get("fault_rc"), int) + or matrix["fault_rc"] <= 0 + or not isinstance(matrix.get("invalid_rc"), int) + or matrix["invalid_rc"] <= 0 + ): + raise RuntimeError(f"unexpected matrix: {matrix}") + status = "PASS" + summary = "TDX event-log, extend, show, replay, negative, concurrent, fault, retry, and file-safety checks passed inside mkosi." + except Exception as e: + summary = f"{type(e).__name__}: {e}" + finally: + if ssh: + evidence["cleanup_returncode"] = run( + [ + *ssh, + "bash", + "-lc", + "pkill -f /run/dstack-test-eventlog/dstack-tee-simulator 2>/dev/null || true; fusermount3 -uz /run/dstack-test-eventlog/report 2>/dev/null || true; rm -rf /run/dstack-test-eventlog /run/log/dstack", + ], + timeout=30, + ).returncode + evidence["duration_seconds"] = round(time.monotonic() - started, 3) + write_json(artifacts / "tdx-eventlog-mkosi.json", evidence) + artifact = { + "path": "artifacts/tdx-eventlog-mkosi.json", + "step_id": f"{CASE_ID}-step-01", + "name": "mkosi TDX event-log CLI suite", + "description": "Guest provenance and event-log/RTMR cryptographic, negative, concurrency, fault, retry, permission, and cleanup evidence.", + } + write_json(artifacts / "manifest.json", {"artifacts": [artifact]}) + observed = ( + summary + if status == "FAIL" + else "The lease-owned mkosi guest passed all event-log CLI modes and safety boundaries using the TDX simulator ABI." + ) + write_json( + result / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{CASE_ID}-step-{n:02d}", + "status": status, + "observed": observed, + } + for n in range(1, 4) + ], + "artifacts": [artifact], + "remarks": "Simulator execution proves CLI encoding, RTMR extend/replay, ordering, errors, and file safety; it does not prove physical TDX isolation or firmware measurements.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-019/case.md b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-019/case.md new file mode 100644 index 000000000..782e80b79 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-019/case.md @@ -0,0 +1,69 @@ + + + +# TC-GOS-SETUP-019: Quote and quote-report CLI bindings + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-gos-setup-019](../../../../catalog/feature-audit.md#req-gos-setup-019) +- Risks: [risk-gos-setup-019](../../../../catalog/feature-audit.md#risk-gos-setup-019) +- Source: `dstack/dstack-util/src/main.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify quote and quote-report cli bindings including exact cryptographic binding, CLI encoding, file safety, negative inputs, and dependency recovery. + +## Preconditions + +1. Run on isolated hardware and simulator environments as applicable; simulator results do not confirm hardware assertions. +2. Capture command argv, exit status, redacted stdout/stderr, output hashes/permissions, and independently decoded cryptographic evidence. + +## Test Data + +Use run-scoped non-secret inputs plus separately generated valid and one-field-mutated evidence fixtures. + +## Steps + + +### Step 1: Exercise all CLI modes and boundaries + +Run `quote` and `quote-report` with empty/boundary/64-byte/oversized report data, sys-config variants, debug/output modes and unavailable TEE device. + +**Expected results:** + +- Quote report data and packaged report bind exact requested/config inputs, output encoding is valid, oversize is rejected and debug cannot weaken verification or leak secrets. + + +### Step 2: Verify independent decoding and failure atomicity + +Decode or verify output with an independent library/tool, inject device/network/filesystem failure before output commit, restore it, and retry. + +**Expected results:** + +- Independent results match, invalid/failing operations return nonzero with actionable redacted error, no partial trusted output remains, and retry succeeds exactly once. + + +### Step 3: Verify isolation permissions and repeatability + +Repeat under another app/device identity and after restart; inspect outputs, logs and temporary files. + +**Expected results:** + +- Deterministic values are stable only within documented identity scope, random values do not repeat, cross-identity evidence/keys fail, permissions are restrictive, and no private material is logged. + +## Postconditions + +Securely remove generated private material and restore device, mount, network and filesystem state. diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-019/metadata.json b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-019/metadata.json new file mode 100644 index 000000000..109bdf38f --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-019/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-setup-019", + "title": "Quote and quote-report CLI bindings", + "priority": "P0", + "requirements": [ + "req-gos-setup-019" + ], + "risks": [ + "risk-gos-setup-019" + ], + "tags": [ + "guest-os", + "dstack-util" + ], + "fixture": { + "profile": "no-tee-guest-lifecycle", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "Quote and quote-report CLI bindings" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 900 + } +} diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-019/run.py b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-019/run.py new file mode 100755 index 000000000..2c73b3d1c --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-019/run.py @@ -0,0 +1,175 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise dstack-util quote commands inside a mkosi guest.""" + +from __future__ import annotations + +import json +import os +import pathlib +import subprocess +import time + +CASE_ID = "tc-gos-setup-019" + + +def run( + argv: list[str], *, data: bytes | None = None, timeout: int = 60 +) -> subprocess.CompletedProcess[bytes]: + """Run one bounded command.""" + return subprocess.run( + argv, input=data, capture_output=True, timeout=timeout, check=False + ) + + +def write_json(path: pathlib.Path, value: object) -> None: + """Write deterministic JSON evidence.""" + path.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n") + + +def main() -> int: + """Execute the quote CLI matrix in the fixture-owned guest.""" + result = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + artifacts = result / "artifacts" + artifacts.mkdir(parents=True, exist_ok=True) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + values = manifest.get("values") or {} + ssh = [str(x) for x in values.get("ssh_argv") or []] + image = str(values.get("image", "")) + status = "FAIL" + summary = "mkosi event-log suite did not execute" + evidence = { + "candidate_commit": runtime.get("candidate_commit"), + "guest_image": image, + } + started = time.monotonic() + try: + if not ssh or values.get("destructive_actions_allowed") is not True: + raise RuntimeError("fixture omitted lease-owned guest SSH") + env = runtime.get("environment") or {} + store = pathlib.Path(str(env.get("DSTACK_TEST_IMAGE_STORE", ""))) + metadata = json.loads((store / image / "metadata.json").read_text()) + if metadata.get("builder") != "mkosi" or metadata.get("is_dev") is not True: + raise RuntimeError("fixture did not boot a mkosi development image") + evidence["mkosi"] = { + k: metadata.get(k) for k in ("builder", "is_dev", "git_revision") + } + repo = pathlib.Path(str(runtime["repository"])) + payloads = [ + ( + pathlib.Path( + runtime["prepared_binaries"]["dstack_tee_simulator"]["path"] + ).read_bytes(), + "/run/dstack-test-quote/dstack-tee-simulator", + ), + ( + pathlib.Path( + runtime["prepared_binaries"]["dstack_util"]["path"] + ).read_bytes(), + "/run/dstack-test-quote/dstack-util", + ), + ( + ( + repo / "test-suites/shared/automation/quote-cli-mkosi.sh" + ).read_bytes(), + "/run/dstack-test-quote/run-case", + ), + ] + for data, target in payloads: + cp = run( + [ + *ssh, + f"mkdir -p /run/dstack-test-quote && install -m 0755 /dev/stdin {target}", + ], + data=data, + timeout=180, + ) + if cp.returncode: + raise RuntimeError( + f"guest install failed: {cp.stderr.decode(errors='replace')[-500:]}" + ) + cp = run([*ssh, "/run/dstack-test-quote/run-case"], timeout=600) + log = cp.stdout + cp.stderr + (artifacts / "mkosi-quote.log").write_bytes(log) + if cp.returncode: + raise RuntimeError( + f"mkosi quote rc={cp.returncode}: {log.decode(errors='replace')[-1600:]}" + ) + rows = [x for x in cp.stdout.decode().splitlines() if x.startswith("{")] + matrix = json.loads(rows[-1]) + evidence["matrix"] = matrix + required_true = ( + "raw_binding", + "sys_config_distinct", + "debug_policy_unchanged", + "retry", + "adjacent_identity", + ) + if ( + matrix.get("boundaries") != [0, 1, 64, 65] + or any(matrix.get(key) is not True for key in required_true) + or any( + not isinstance(matrix.get(key), int) or matrix[key] <= 0 + for key in ("raw63_rc", "raw65_rc", "over_rc", "output_rc", "device_rc") + ) + ): + raise RuntimeError(f"unexpected matrix: {matrix}") + status = "PASS" + summary = "Raw quote and quote-report binding, boundary, config, debug, fault, retry, identity, and file-safety checks passed inside mkosi." + except Exception as e: + summary = f"{type(e).__name__}: {e}" + finally: + if ssh: + evidence["cleanup_returncode"] = run( + [ + *ssh, + "bash", + "-lc", + "pkill -f /run/dstack-test-quote/dstack-tee-simulator 2>/dev/null || true; fusermount3 -uz /run/dstack-test-eventlog/report 2>/dev/null || true; rm -rf /run/dstack-test-eventlog /run/log/dstack", + ], + timeout=30, + ).returncode + evidence["duration_seconds"] = round(time.monotonic() - started, 3) + write_json(artifacts / "quote-cli-mkosi.json", evidence) + artifact = { + "path": "artifacts/quote-cli-mkosi.json", + "step_id": f"{CASE_ID}-step-01", + "name": "mkosi quote CLI suite", + "description": "Guest provenance and raw/packaged quote binding, boundary, config, debug, fault, retry, identity, and cleanup evidence.", + } + write_json(artifacts / "manifest.json", {"artifacts": [artifact]}) + observed = ( + summary + if status == "FAIL" + else "The lease-owned mkosi guest passed all quote CLI modes and safety boundaries using the TDX simulator ABI." + ) + write_json( + result / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{CASE_ID}-step-{n:02d}", + "status": status, + "observed": observed, + } + for n in range(1, 4) + ], + "artifacts": [artifact], + "remarks": "Simulator execution proves quote binding, encoding, config, errors, identity, and file safety; it does not prove physical TDX isolation or vendor-signed evidence.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-020/case.md b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-020/case.md new file mode 100644 index 000000000..f003263c1 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-020/case.md @@ -0,0 +1,75 @@ + + + +# TC-GOS-SETUP-020: RA CA and app key generation CLI + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-gos-setup-020](../../../../catalog/feature-audit.md#req-gos-setup-020) +- Risks: [risk-gos-setup-020](../../../../catalog/feature-audit.md#risk-gos-setup-020) +- Source: `dstack/dstack-util/src/main.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify ra ca and app key generation cli including exact cryptographic binding, CLI encoding, file safety, negative inputs, and dependency recovery. + +## Preconditions + +1. Run on isolated hardware and simulator environments as applicable; simulator results do not confirm hardware assertions. +2. Capture command argv, exit status, redacted stdout/stderr, output hashes/permissions, and independently decoded cryptographic evidence. + +## Test Data + +Use run-scoped non-secret inputs plus separately generated valid and one-field-mutated evidence fixtures. + +Use the candidate CLI argument contract exactly: + +- `gen-ca-cert --cert --key --ca-level ` +- `gen-ra-cert --ca-cert --ca-key --cert-path --key-path ` +- `gen-app-keys --ca-level --output ` + +## Steps + + +### Step 1: Exercise all CLI modes and boundaries + +Run `gen-ra-cert`, `gen-ca-cert`, and `gen-app-keys` across CA levels, SAN/usage inputs, existing outputs, unsafe paths/permissions, and a mismatched CA key. + +**Expected results:** + +- Generated keys match certificates/chains and intended CA constraints, private files are restrictive, and mismatch fails without overwriting existing trusted output. + + +### Step 2: Verify independent decoding and error recovery + +Decode or verify output with an independent library/tool, exercise an invalid output path, restore it, and retry. + +**Expected results:** + +- Independent results match, invalid/failing operations return nonzero with actionable redacted error, and retry succeeds. + + +### Step 3: Verify isolation permissions and repeatability + +Repeat under another app/device identity and after restart; inspect outputs, logs and temporary files. + +**Expected results:** + +- Deterministic values are stable only within documented identity scope, random values do not repeat, cross-identity evidence/keys fail, permissions are restrictive, and no private material is logged. + +## Postconditions + +Securely remove generated private material and restore device, mount, network and filesystem state. diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-020/metadata.json b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-020/metadata.json new file mode 100644 index 000000000..226632b46 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-020/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-setup-020", + "title": "RA CA and app key generation CLI", + "priority": "P0", + "requirements": [ + "req-gos-setup-020" + ], + "risks": [ + "risk-gos-setup-020" + ], + "tags": [ + "guest-os", + "dstack-util" + ], + "fixture": { + "profile": "no-tee-guest-lifecycle", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "RA CA and app key generation CLI" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 900 + } +} diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-020/run.py b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-020/run.py new file mode 100755 index 000000000..f40433a1f --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-020/run.py @@ -0,0 +1,176 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise dstack-util RA and key commands inside a mkosi guest.""" + +from __future__ import annotations + +import json +import os +import pathlib +import subprocess +import time + +CASE_ID = "tc-gos-setup-020" + + +def run( + argv: list[str], *, data: bytes | None = None, timeout: int = 60 +) -> subprocess.CompletedProcess[bytes]: + """Run one bounded command.""" + return subprocess.run( + argv, input=data, capture_output=True, timeout=timeout, check=False + ) + + +def write_json(path: pathlib.Path, value: object) -> None: + """Write deterministic JSON evidence.""" + path.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n") + + +def main() -> int: + """Execute the RA and key CLI matrix in the fixture-owned guest.""" + result = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + artifacts = result / "artifacts" + artifacts.mkdir(parents=True, exist_ok=True) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + values = manifest.get("values") or {} + ssh = [str(x) for x in values.get("ssh_argv") or []] + image = str(values.get("image", "")) + status = "FAIL" + summary = "mkosi event-log suite did not execute" + evidence = { + "candidate_commit": runtime.get("candidate_commit"), + "guest_image": image, + } + started = time.monotonic() + try: + if not ssh or values.get("destructive_actions_allowed") is not True: + raise RuntimeError("fixture omitted lease-owned guest SSH") + env = runtime.get("environment") or {} + store = pathlib.Path(str(env.get("DSTACK_TEST_IMAGE_STORE", ""))) + metadata = json.loads((store / image / "metadata.json").read_text()) + if metadata.get("builder") != "mkosi" or metadata.get("is_dev") is not True: + raise RuntimeError("fixture did not boot a mkosi development image") + evidence["mkosi"] = { + k: metadata.get(k) for k in ("builder", "is_dev", "git_revision") + } + repo = pathlib.Path(str(runtime["repository"])) + payloads = [ + ( + pathlib.Path( + runtime["prepared_binaries"]["dstack_tee_simulator"]["path"] + ).read_bytes(), + "/run/dstack-test-ra-key/dstack-tee-simulator", + ), + ( + pathlib.Path( + runtime["prepared_binaries"]["dstack_util"]["path"] + ).read_bytes(), + "/run/dstack-test-ra-key/dstack-util", + ), + ( + ( + repo / "test-suites/shared/automation/ra-key-cli-mkosi.sh" + ).read_bytes(), + "/run/dstack-test-ra-key/run-case", + ), + ] + for data, target in payloads: + cp = run( + [ + *ssh, + f"mkdir -p /run/dstack-test-ra-key && install -m 0755 /dev/stdin {target}", + ], + data=data, + timeout=180, + ) + if cp.returncode: + raise RuntimeError( + f"guest install failed: {cp.stderr.decode(errors='replace')[-500:]}" + ) + cp = run([*ssh, "/run/dstack-test-ra-key/run-case"], timeout=600) + log = cp.stdout + cp.stderr + (artifacts / "mkosi-ra-key.log").write_bytes(log) + if cp.returncode: + raise RuntimeError( + f"mkosi RA/key rc={cp.returncode}: {log.decode(errors='replace')[-1600:]}" + ) + rows = [x for x in cp.stdout.decode().splitlines() if x.startswith("{")] + matrix = json.loads(rows[-1]) + evidence["matrix"] = matrix + required_true = ( + "chain_valid", + "key_match", + "random_identity", + "retry", + "no_secret_logs", + ) + if ( + matrix.get("ca_levels") != [0, 1, 2] + or matrix.get("private_modes") != "600" + or any(matrix.get(key) is not True for key in required_true) + or any( + not isinstance(matrix.get(key), int) or matrix[key] <= 0 + for key in ("mismatch_rc", "app_fault_rc") + ) + ): + raise RuntimeError(f"unexpected matrix: {matrix}") + status = "PASS" + summary = "CA, RA certificate, and app-key chain, mismatch, permission, randomness, and retry checks passed inside mkosi." + except Exception as e: + summary = f"{type(e).__name__}: {e}" + finally: + if ssh: + evidence["cleanup_returncode"] = run( + [ + *ssh, + "bash", + "-lc", + "pkill -f /run/dstack-test-ra-key/dstack-tee-simulator 2>/dev/null || true; fusermount3 -uz /run/dstack-test-ra-key/report 2>/dev/null || true; rm -rf /run/dstack-test-ra-key /run/log/dstack", + ], + timeout=30, + ).returncode + evidence["duration_seconds"] = round(time.monotonic() - started, 3) + write_json(artifacts / "ra-key-cli-mkosi.json", evidence) + artifact = { + "path": "artifacts/ra-key-cli-mkosi.json", + "step_id": f"{CASE_ID}-step-01", + "name": "mkosi RA/key CLI suite", + "description": "Guest provenance and CA/RA/app-key chain, mismatch, permissions, randomness, retry, and cleanup evidence.", + } + write_json(artifacts / "manifest.json", {"artifacts": [artifact]}) + observed = ( + summary + if status == "FAIL" + else "The lease-owned mkosi guest passed all RA and key CLI modes and safety boundaries using the TDX simulator ABI." + ) + write_json( + result / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{CASE_ID}-step-{n:02d}", + "status": status, + "observed": observed, + } + for n in range(1, 4) + ], + "artifacts": [artifact], + "remarks": "Simulator execution proves certificate/key binding, constraints, errors, permissions, and identity; it does not prove physical TDX isolation or vendor-signed evidence.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-021/case.md b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-021/case.md new file mode 100644 index 000000000..640bd4d68 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-021/case.md @@ -0,0 +1,69 @@ + + + +# TC-GOS-SETUP-021: Random and hexadecimal utility CLI + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-gos-setup-021](../../../../catalog/feature-audit.md#req-gos-setup-021) +- Risks: [risk-gos-setup-021](../../../../catalog/feature-audit.md#risk-gos-setup-021) +- Source: `dstack/dstack-util/src/main.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify random and hexadecimal utility cli including exact cryptographic binding, CLI encoding, file safety, negative inputs, and dependency recovery. + +## Preconditions + +1. Run on isolated hardware and simulator environments as applicable; simulator results do not confirm hardware assertions. +2. Capture command argv, exit status, redacted stdout/stderr, output hashes/permissions, and independently decoded cryptographic evidence. + +## Test Data + +Use run-scoped non-secret inputs plus separately generated valid and one-field-mutated evidence fixtures. + +## Steps + + +### Step 1: Exercise all CLI modes and boundaries + +Run `rand` and `hex` at zero/default/maximum sizes to stdout/file/hex, with short writes, existing file, entropy failure and binary/empty input. + +**Expected results:** + +- Random output has exact requested length and encoding without reuse, hex is exact lowercase documented form, errors do not leave partial output and no random bytes enter logs. + + +### Step 2: Verify independent decoding and failure atomicity + +Decode or verify output with an independent library/tool, inject device/network/filesystem failure before output commit, restore it, and retry. + +**Expected results:** + +- Independent results match, invalid/failing operations return nonzero with actionable redacted error, no partial trusted output remains, and retry succeeds exactly once. + + +### Step 3: Verify isolation permissions and repeatability + +Repeat under another app/device identity and after restart; inspect outputs, logs and temporary files. + +**Expected results:** + +- Deterministic values are stable only within documented identity scope, random values do not repeat, cross-identity evidence/keys fail, permissions are restrictive, and no private material is logged. + +## Postconditions + +Securely remove generated private material and restore device, mount, network and filesystem state. diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-021/metadata.json b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-021/metadata.json new file mode 100644 index 000000000..a921d4f9f --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-021/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-setup-021", + "title": "Random and hexadecimal utility CLI", + "priority": "P0", + "requirements": [ + "req-gos-setup-021" + ], + "risks": [ + "risk-gos-setup-021" + ], + "tags": [ + "guest-os", + "dstack-util" + ], + "fixture": { + "profile": "no-tee-dev", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "Random and hexadecimal utility CLI" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 180 + } +} diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-021/run.py b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-021/run.py new file mode 100755 index 000000000..8886c2b3d --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-021/run.py @@ -0,0 +1,170 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 +"""Deterministic dstack-util random and hexadecimal CLI regression.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import stat +import subprocess +import tempfile +from typing import Any + +CASE = "tc-gos-setup-021" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Atomically write JSON evidence.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as output: + json.dump(value, output, indent=2, sort_keys=True) + output.write("\n") + temporary = pathlib.Path(output.name) + temporary.replace(path) + + +def run( + binary: pathlib.Path, *args: str, input_data: bytes | None = None +) -> subprocess.CompletedProcess[bytes]: + """Run dstack-util without decoding random stdout.""" + return subprocess.run( + [str(binary), *args], + input=input_data, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + timeout=60, + check=False, + ) + + +def main() -> int: + """Execute promoted rand/hex coverage.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + if case_id != CASE: + raise RuntimeError(f"unsupported case: {case_id}") + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + prepared = runtime.get("prepared_binaries", {}).get("dstack_util", {}) + binary = pathlib.Path(prepared.get("resolved_path") or prepared.get("path") or "") + if not binary.is_file(): + raise RuntimeError("prepared dstack-util binary is unavailable") + steps = [] + failures = [] + evidence = {} + try: + print(f"STEP {case_id}-step-01 START", flush=True) + first = run(binary, "rand", "--bytes", "32") + second = run(binary, "rand", "-n", "32") + encoded = run(binary, "rand", "--bytes", "16", "--hex") + if first.returncode or second.returncode or encoded.returncode: + raise AssertionError("valid random stdout mode failed") + if ( + len(first.stdout) != 32 + or len(second.stdout) != 32 + or len(encoded.stdout) != 32 + ): + raise AssertionError("random output length mismatch") + if first.stdout == second.stdout: + raise AssertionError("independent random outputs repeated") + bytes.fromhex(encoded.stdout.decode()) + steps.append( + { + "id": f"{case_id}-step-01", + "status": "PASS", + "observed": "Raw and hex random modes returned exact lengths without reuse.", + } + ) + print(f"STEP {case_id}-step-01 END - PASS", flush=True) + print(f"STEP {case_id}-step-02 START", flush=True) + with tempfile.TemporaryDirectory(dir=result_dir) as directory: + output = pathlib.Path(directory) / "random.bin" + written = run(binary, "rand", "--bytes", "48", "--output", str(output)) + before = output.read_bytes() + repeat = run(binary, "rand", "--bytes", "8", "--output", str(output)) + after = output.read_bytes() + if written.returncode or len(before) != 48: + raise AssertionError("file output failed") + if repeat.returncode or len(after) != 8 or before == after: + raise AssertionError("atomic existing-file replacement failed") + mode = stat.S_IMODE(output.stat().st_mode) + if mode != 0o600: + raise AssertionError(f"random output mode was {mode:o}") + binary_input = bytes(range(256)) + hexed = run(binary, "hex", input_data=binary_input) + if hexed.returncode or hexed.stdout.decode() != binary_input.hex(): + raise AssertionError("hex stdin encoding mismatch") + evidence["matrix"] = { + "raw_lengths": [len(first.stdout), len(second.stdout)], + "hex_length": len(encoded.stdout), + "file_length": 48, + "file_mode": "0600", + "atomic_replacement": True, + "binary_hex_exact": True, + "random_values_persisted": False, + } + steps.append( + { + "id": f"{case_id}-step-02", + "status": "PASS", + "observed": "Atomic owner-only file replacement and independent hex decoding matched.", + } + ) + print(f"STEP {case_id}-step-02 END - PASS", flush=True) + print(f"STEP {case_id}-step-03 START", flush=True) + post = run(binary, "rand", "--bytes", "32") + if post.returncode or post.stdout in (first.stdout, second.stdout): + raise AssertionError("post-error random recovery failed") + evidence["matrix"]["post_error_recovery"] = True + steps.append( + { + "id": f"{case_id}-step-03", + "status": "PASS", + "observed": "Post-error retry succeeded with a fresh value and no random bytes were persisted.", + } + ) + print(f"STEP {case_id}-step-03 END - PASS", flush=True) + except Exception as error: + failures.append(f"{type(error).__name__}: {error}") + for number in range(1, 4): + sid = f"{case_id}-step-{number:02d}" + if not any(step["id"] == sid for step in steps): + steps.append({"id": sid, "status": "FAIL", "observed": failures[-1]}) + evidence["binary_sha256"] = hashlib.sha256(binary.read_bytes()).hexdigest() + evidence["sensitive_values_persisted"] = False + artifact = { + "name": "Random and hex CLI matrix", + "path": "artifacts/rand-hex-matrix.json", + "step_id": f"{case_id}-step-02", + "description": "Records only lengths, permissions, boolean assertions, and the prepared binary digest; random bytes are not persisted.", + } + atomic_json(result_dir / artifact["path"], evidence) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + status = "PASS" if not failures else "FAIL" + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": "Random and hexadecimal CLI regression passed." + if status == "PASS" + else failures[0], + "steps": steps, + "artifacts": [artifact], + "remarks": "No generated random value was written to result artifacts or logs.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-022/case.md b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-022/case.md new file mode 100644 index 000000000..009f0a99d --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-022/case.md @@ -0,0 +1,69 @@ + + + +# TC-GOS-SETUP-022: vTPM attest quote and verify CLI suite + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-gos-setup-022](../../../../catalog/feature-audit.md#req-gos-setup-022) +- Risks: [risk-gos-setup-022](../../../../catalog/feature-audit.md#risk-gos-setup-022) +- Source: `dstack/dstack-util/src/main.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify vtpm attest quote and verify cli suite including exact cryptographic binding, CLI encoding, file safety, negative inputs, and dependency recovery. + +## Preconditions + +1. Run on isolated hardware and simulator environments as applicable; simulator results do not confirm hardware assertions. +2. Capture command argv, exit status, redacted stdout/stderr, output hashes/permissions, and independently decoded cryptographic evidence. + +## Test Data + +Use run-scoped non-secret inputs plus separately generated valid and one-field-mutated evidence fixtures. + +## Steps + + +### Step 1: Exercise all CLI modes and boundaries + +Run `vtpm-attest`, `tpm-quote`, and `tpm-verify` using RSA/ECC/auto, nonce/data/hash variants, correct/wrong root, altered PCR/signature/event log, replay and expected OS hash. + +**Expected results:** + +- Valid chain/signature/nonce/PCR replay/OS hash verify together; every altered or replayed field fails the corresponding assertion and no unsupported algorithm is accepted. + + +### Step 2: Verify independent decoding and failure atomicity + +Decode or verify output with an independent library/tool, inject device/network/filesystem failure before output commit, restore it, and retry. + +**Expected results:** + +- Independent results match, invalid/failing operations return nonzero with actionable redacted error, no partial trusted output remains, and retry succeeds exactly once. + + +### Step 3: Verify isolation permissions and repeatability + +Repeat under another app/device identity and after restart; inspect outputs, logs and temporary files. + +**Expected results:** + +- Deterministic values are stable only within documented identity scope, random values do not repeat, cross-identity evidence/keys fail, permissions are restrictive, and no private material is logged. + +## Postconditions + +Securely remove generated private material and restore device, mount, network and filesystem state. diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-022/metadata.json b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-022/metadata.json new file mode 100644 index 000000000..c50742c50 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-022/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-setup-022", + "title": "vTPM attest quote and verify CLI suite", + "priority": "P0", + "requirements": [ + "req-gos-setup-022" + ], + "risks": [ + "risk-gos-setup-022" + ], + "tags": [ + "guest-os", + "dstack-util" + ], + "fixture": { + "profile": "no-tee-guest-lifecycle", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "vTPM attest quote and verify CLI suite" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 900 + } +} diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-022/run.py b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-022/run.py new file mode 100755 index 000000000..0989cbae8 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-022/run.py @@ -0,0 +1,176 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise dstack-util quote commands inside a mkosi guest.""" + +from __future__ import annotations + +import json +import os +import pathlib +import subprocess +import time + +CASE_ID = "tc-gos-setup-022" + + +def run( + argv: list[str], *, data: bytes | None = None, timeout: int = 60 +) -> subprocess.CompletedProcess[bytes]: + """Run one bounded command.""" + return subprocess.run( + argv, input=data, capture_output=True, timeout=timeout, check=False + ) + + +def write_json(path: pathlib.Path, value: object) -> None: + """Write deterministic JSON evidence.""" + path.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n") + + +def main() -> int: + """Execute the quote CLI matrix in the fixture-owned guest.""" + result = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + artifacts = result / "artifacts" + artifacts.mkdir(parents=True, exist_ok=True) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + values = manifest.get("values") or {} + ssh = [str(x) for x in values.get("ssh_argv") or []] + image = str(values.get("image", "")) + status = "FAIL" + summary = "mkosi event-log suite did not execute" + evidence = { + "candidate_commit": runtime.get("candidate_commit"), + "guest_image": image, + } + started = time.monotonic() + try: + if not ssh or values.get("destructive_actions_allowed") is not True: + raise RuntimeError("fixture omitted lease-owned guest SSH") + env = runtime.get("environment") or {} + store = pathlib.Path(str(env.get("DSTACK_TEST_IMAGE_STORE", ""))) + metadata = json.loads((store / image / "metadata.json").read_text()) + if metadata.get("builder") != "mkosi" or metadata.get("is_dev") is not True: + raise RuntimeError("fixture did not boot a mkosi development image") + evidence["mkosi"] = { + k: metadata.get(k) for k in ("builder", "is_dev", "git_revision") + } + repo = pathlib.Path(str(runtime["repository"])) + payloads = [ + ( + pathlib.Path( + runtime["prepared_binaries"]["dstack_tee_simulator"]["path"] + ).read_bytes(), + "/run/dstack-test-vtpm/dstack-tee-simulator", + ), + ( + pathlib.Path( + runtime["prepared_binaries"]["dstack_util"]["path"] + ).read_bytes(), + "/run/dstack-test-vtpm/dstack-util", + ), + ( + (repo / "test-suites/shared/automation/vtpm-cli-mkosi.sh").read_bytes(), + "/run/dstack-test-vtpm/run-case", + ), + ] + for data, target in payloads: + cp = run( + [ + *ssh, + f"mkdir -p /run/dstack-test-vtpm && install -m 0755 /dev/stdin {target}", + ], + data=data, + timeout=180, + ) + if cp.returncode: + raise RuntimeError( + f"guest install failed: {cp.stderr.decode(errors='replace')[-500:]}" + ) + cp = run([*ssh, "/run/dstack-test-vtpm/run-case"], timeout=600) + log = cp.stdout + cp.stderr + (artifacts / "mkosi-vtpm.log").write_bytes(log) + if cp.returncode: + raise RuntimeError( + f"mkosi quote rc={cp.returncode}: {log.decode(errors='replace')[-1600:]}" + ) + rows = [x for x in cp.stdout.decode().splitlines() if x.startswith("{")] + matrix = json.loads(rows[-1]) + evidence["matrix"] = matrix + required_true = ( + "vtpm_rsa", + "vtpm_ecc", + "quote_auto", + "quote_ecc", + "quote_rsa", + "verify", + "wrong_root_rejected", + "pcr_rejected", + "signature_rejected", + "network_rejected", + "device_rejected", + "output_atomic", + "retry", + "adjacent_identity", + "permissions", + ) + if any(matrix.get(key) is not True for key in required_true): + raise RuntimeError(f"unexpected matrix: {matrix}") + status = "PASS" + summary = "vTPM attest, TPM quote/verify, mutation, fault, retry, identity, and file-safety checks passed inside mkosi." + except Exception as e: + summary = f"{type(e).__name__}: {e}" + finally: + if ssh: + evidence["cleanup_returncode"] = run( + [ + *ssh, + "bash", + "-lc", + "pkill -f /run/dstack-test-vtpm/dstack-tee-simulator 2>/dev/null || true; fusermount3 -uz /run/dstack-test-eventlog/report 2>/dev/null || true; rm -rf /run/dstack-test-eventlog /run/log/dstack", + ], + timeout=30, + ).returncode + evidence["duration_seconds"] = round(time.monotonic() - started, 3) + write_json(artifacts / "vtpm-cli-mkosi.json", evidence) + artifact = { + "path": "artifacts/vtpm-cli-mkosi.json", + "step_id": f"{CASE_ID}-step-01", + "name": "mkosi vTPM CLI suite", + "description": "Guest provenance and vTPM attest, quote, verify, mutation, fault, retry, identity, and cleanup evidence.", + } + write_json(artifacts / "manifest.json", {"artifacts": [artifact]}) + observed = ( + summary + if status == "FAIL" + else "The lease-owned mkosi guest passed the vTPM CLI trust, mutation, fault, retry, and isolation matrix using the GCP vTPM simulator ABI." + ) + write_json( + result / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{CASE_ID}-step-{n:02d}", + "status": status, + "observed": observed, + } + for n in range(1, 4) + ], + "artifacts": [artifact], + "remarks": "Simulator execution inside mkosi proves CLI cryptographic behavior and fault handling; it does not prove vendor hardware isolation or vendor-signed certificates.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-023/case.md b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-023/case.md new file mode 100644 index 000000000..b817c10d7 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-023/case.md @@ -0,0 +1,69 @@ + + + +# TC-GOS-SETUP-023: Versioned attestation create inspect JSON and strip CLI + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR/MKOSI +- Automation: Yes +- Requirements: [req-gos-setup-023](../../../../catalog/feature-audit.md#req-gos-setup-023) +- Risks: [risk-gos-setup-023](../../../../catalog/feature-audit.md#risk-gos-setup-023) +- Source: `dstack/dstack-util/src/main.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify versioned attestation create inspect json and strip cli including exact cryptographic binding, CLI encoding, file safety, negative inputs, and dependency recovery. + +## Preconditions + +1. Run on isolated hardware and simulator environments as applicable; simulator results do not confirm hardware assertions. +2. Capture command argv, exit status, redacted stdout/stderr, output hashes/permissions, and independently decoded cryptographic evidence. + +## Test Data + +Use run-scoped non-secret inputs plus separately generated valid and one-field-mutated evidence fixtures. + +## Steps + + +### Step 1: Exercise all CLI modes and boundaries + +Run `attest`, `attest-info`, `attest-json`, and `attest-strip` for every platform/version with boundary report data/app ID, truncated/unknown/oversized encoding and round trips. + +**Expected results:** + +- Info sizes and JSON exactly describe authenticated envelope, strip removes only permitted certificate payload while preserving verification, and malformed/unknown versions fail without downgrade. + + +### Step 2: Verify independent decoding and failure atomicity + +Decode or verify output with an independent library/tool, inject device/network/filesystem failure before output commit, restore it, and retry. + +**Expected results:** + +- Independent results match, invalid/failing operations return nonzero with actionable redacted error, no partial trusted output remains, and retry succeeds exactly once. + + +### Step 3: Verify isolation permissions and repeatability + +Repeat under another app/device identity and after restart; inspect outputs, logs and temporary files. + +**Expected results:** + +- Deterministic values are stable only within documented identity scope, random values do not repeat, cross-identity evidence/keys fail, permissions are restrictive, and no private material is logged. + +## Postconditions + +Securely remove generated private material and restore device, mount, network and filesystem state. diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-023/metadata.json b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-023/metadata.json new file mode 100644 index 000000000..86bb5b213 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-023/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-setup-023", + "title": "Versioned attestation create inspect JSON and strip CLI", + "priority": "P0", + "requirements": [ + "req-gos-setup-023" + ], + "risks": [ + "risk-gos-setup-023" + ], + "tags": [ + "guest-os", + "dstack-util" + ], + "fixture": { + "profile": "no-tee-guest-lifecycle", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "Versioned attestation create inspect JSON and strip CLI" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 900 + } +} diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-023/run.py b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-023/run.py new file mode 100755 index 000000000..45b752e66 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-023/run.py @@ -0,0 +1,184 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise dstack-util versioned attestation commands inside a mkosi guest.""" + +from __future__ import annotations + +import json +import os +import pathlib +import subprocess +import time + +CASE_ID = "tc-gos-setup-023" + + +def run( + argv: list[str], *, data: bytes | None = None, timeout: int = 60 +) -> subprocess.CompletedProcess[bytes]: + """Run one bounded command.""" + return subprocess.run( + argv, input=data, capture_output=True, timeout=timeout, check=False + ) + + +def write_json(path: pathlib.Path, value: object) -> None: + """Write deterministic JSON evidence.""" + path.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n") + + +def main() -> int: + """Execute the quote CLI matrix in the fixture-owned guest.""" + result = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + artifacts = result / "artifacts" + artifacts.mkdir(parents=True, exist_ok=True) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + values = manifest.get("values") or {} + ssh = [str(x) for x in values.get("ssh_argv") or []] + image = str(values.get("image", "")) + status = "FAIL" + summary = "mkosi event-log suite did not execute" + evidence = { + "candidate_commit": runtime.get("candidate_commit"), + "guest_image": image, + } + started = time.monotonic() + try: + if not ssh or values.get("destructive_actions_allowed") is not True: + raise RuntimeError("fixture omitted lease-owned guest SSH") + env = runtime.get("environment") or {} + store = pathlib.Path(str(env.get("DSTACK_TEST_IMAGE_STORE", ""))) + metadata = json.loads((store / image / "metadata.json").read_text()) + if metadata.get("builder") != "mkosi" or metadata.get("is_dev") is not True: + raise RuntimeError("fixture did not boot a mkosi development image") + evidence["mkosi"] = { + k: metadata.get(k) for k in ("builder", "is_dev", "git_revision") + } + repo = pathlib.Path(str(runtime["repository"])) + payloads = [ + ( + pathlib.Path( + runtime["prepared_binaries"]["dstack_tee_simulator"]["path"] + ).read_bytes(), + "/run/dstack-test-attest/dstack-tee-simulator", + ), + ( + pathlib.Path( + runtime["prepared_binaries"]["dstack_util"]["path"] + ).read_bytes(), + "/run/dstack-test-attest/dstack-util", + ), + ( + ( + repo + / "test-suites/shared/automation/versioned-attestation-mkosi.sh" + ).read_bytes(), + "/run/dstack-test-attest/run-case", + ), + ] + for data, target in payloads: + cp = run( + [ + *ssh, + f"mkdir -p /run/dstack-test-attest && install -m 0755 /dev/stdin {target}", + ], + data=data, + timeout=180, + ) + if cp.returncode: + raise RuntimeError( + f"guest install failed: {cp.stderr.decode(errors='replace')[-500:]}" + ) + cp = run([*ssh, "/run/dstack-test-attest/run-case"], timeout=600) + log = cp.stdout + cp.stderr + (artifacts / "mkosi-versioned-attestation.log").write_bytes(log) + if cp.returncode: + raise RuntimeError( + f"mkosi attestation rc={cp.returncode}: {log.decode(errors='replace')[-1600:]}" + ) + rows = [x for x in cp.stdout.decode().splitlines() if x.startswith("{")] + matrix = json.loads(rows[-1]) + evidence["matrix"] = matrix + required_true = ( + "v0", + "v1", + "strip_decodable", + "binding_distinct", + "retry", + "adjacent_identity", + ) + if ( + matrix.get("boundaries") != [0, 1, 64, 65] + or any(matrix.get(key) is not True for key in required_true) + or any( + not isinstance(matrix.get(key), int) or matrix[key] <= 0 + for key in ( + "bad_app_rc", + "truncated_rc", + "unknown_rc", + "oversized_rc", + "output_rc", + "device_rc", + ) + ) + ): + raise RuntimeError(f"unexpected matrix: {matrix}") + status = "PASS" + summary = "Versioned attestation V0/V1, boundary, strip, malformed-input, fault, retry, identity, and file-safety checks passed inside mkosi." + except Exception as e: + summary = f"{type(e).__name__}: {e}" + finally: + if ssh: + evidence["cleanup_returncode"] = run( + [ + *ssh, + "bash", + "-lc", + "pkill -f /run/dstack-test-attest/dstack-tee-simulator 2>/dev/null || true; fusermount3 -uz /run/dstack-test-attest/report 2>/dev/null || true; rm -rf /run/dstack-test-attest /run/log/dstack", + ], + timeout=30, + ).returncode + evidence["duration_seconds"] = round(time.monotonic() - started, 3) + write_json(artifacts / "versioned-attestation-mkosi.json", evidence) + artifact = { + "path": "artifacts/versioned-attestation-mkosi.json", + "step_id": f"{CASE_ID}-step-01", + "name": "mkosi versioned-attestation CLI suite", + "description": "Guest provenance and versioned encoding, boundary, strip, fault, retry, identity, and cleanup evidence.", + } + write_json(artifacts / "manifest.json", {"artifacts": [artifact]}) + observed = ( + summary + if status == "FAIL" + else "The lease-owned mkosi guest passed all versioned attestation CLI modes and safety boundaries using the TDX simulator ABI." + ) + write_json( + result / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{CASE_ID}-step-{n:02d}", + "status": status, + "observed": observed, + } + for n in range(1, 4) + ], + "artifacts": [artifact], + "remarks": "Simulator execution proves functional encoding, binding, errors, identity, and file safety; it does not prove physical TDX isolation or vendor-signed evidence.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-024/case.md b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-024/case.md new file mode 100644 index 000000000..be1732a08 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-024/case.md @@ -0,0 +1,69 @@ + + + +# TC-GOS-SETUP-024: KMS GetKeys CLI transport and output safety + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR/MKOSI +- Automation: Yes +- Requirements: [req-gos-setup-024](../../../../catalog/feature-audit.md#req-gos-setup-024) +- Risks: [risk-gos-setup-024](../../../../catalog/feature-audit.md#risk-gos-setup-024) +- Source: `dstack/dstack-util/src/main.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify kms getkeys cli transport and output safety including exact cryptographic binding, CLI encoding, file safety, negative inputs, and dependency recovery. + +## Preconditions + +1. Run on isolated hardware and simulator environments as applicable; simulator results do not confirm hardware assertions. +2. Capture command argv, exit status, redacted stdout/stderr, output hashes/permissions, and independently decoded cryptographic evidence. + +## Test Data + +Use run-scoped non-secret inputs plus separately generated valid and one-field-mutated evidence fixtures. + +## Steps + + +### Step 1: Exercise all CLI modes and boundaries + +Run `get-keys` against valid/multiple/timeout/wrong-cert/deny KMS URLs with valid/altered vm_config and output paths, then repeat/restart. + +**Expected results:** + +- Only attestation-authorized response is accepted, failover preserves one key identity, output is atomic/restrictive and no key material appears on stdout/logs unless explicitly documented. + + +### Step 2: Verify independent decoding and failure atomicity + +Decode or verify output with an independent library/tool, inject device/network/filesystem failure before output commit, restore it, and retry. + +**Expected results:** + +- Independent results match, invalid/failing operations return nonzero with actionable redacted error, no partial trusted output remains, and retry succeeds exactly once. + + +### Step 3: Verify isolation permissions and repeatability + +Repeat under another app/device identity and after restart; inspect outputs, logs and temporary files. + +**Expected results:** + +- Deterministic values are stable only within documented identity scope, random values do not repeat, cross-identity evidence/keys fail, permissions are restrictive, and no private material is logged. + +## Postconditions + +Securely remove generated private material and restore device, mount, network and filesystem state. diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-024/metadata.json b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-024/metadata.json new file mode 100644 index 000000000..bd9d9a712 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-024/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-setup-024", + "title": "KMS GetKeys CLI transport and output safety", + "priority": "P0", + "requirements": [ + "req-gos-setup-024" + ], + "risks": [ + "risk-gos-setup-024" + ], + "tags": [ + "guest-os", + "dstack-util" + ], + "fixture": { + "profile": "no-tee-guest-lifecycle", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "KMS GetKeys CLI transport and output safety" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 900 + } +} diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-024/run.py b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-024/run.py new file mode 100755 index 000000000..c8649230f --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-024/run.py @@ -0,0 +1,151 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +# ruff: noqa +"""Exercise dstack-util get-keys inside a lease-owned mkosi guest.""" + +import json +import os +import pathlib +import subprocess +import time + +CASE_ID = "tc-gos-setup-024" + + +def run(a, data=None, timeout=60): + return subprocess.run( + a, input=data, capture_output=True, timeout=timeout, check=False + ) + + +def dump(p, v): + p.write_text(json.dumps(v, indent=2, sort_keys=True) + "\n") + + +def main(): + r = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + art = r / "artifacts" + art.mkdir(parents=True, exist_ok=True) + started = time.monotonic() + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + m = json.loads(pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text()) + v = m.get("values") or {} + ssh = list(map(str, v.get("ssh_argv") or [])) + status = "FAIL" + summary = "mkosi KMS get-keys suite did not execute" + ev = { + "candidate_commit": runtime.get("candidate_commit"), + "guest_image": v.get("image"), + } + try: + if not ssh or v.get("destructive_actions_allowed") is not True: + raise RuntimeError("fixture omitted lease-owned guest SSH") + kms = v.get("case_kms") or {} + url = str(kms.get("guest_url", "")) + cert = pathlib.Path(str(kms.get("kms_rpc_cert", ""))) + if not url or not cert.is_file(): + raise RuntimeError("fixture omitted case-scoped KMS public inputs") + repo = pathlib.Path(runtime["repository"]) + payloads = [ + ( + pathlib.Path( + runtime["prepared_binaries"]["dstack_util"]["path"] + ).read_bytes(), + "/run/dstack-test-getkeys/dstack-util", + ), + ( + (repo / "dstack/cc-eventlog/samples/ccel.bin").read_bytes(), + "/run/dstack-test-getkeys/ccel.bin", + ), + (cert.read_bytes(), "/run/dstack-test-getkeys/kms.crt"), + ( + ( + repo / "test-suites/shared/automation/kms-getkeys-mkosi.sh" + ).read_bytes(), + "/run/dstack-test-getkeys/run-case", + ), + ] + for data, target in payloads: + cp = run( + [ + *ssh, + f"mkdir -p /run/dstack-test-getkeys && install -m 0755 /dev/stdin {target}", + ], + data, + 180, + ) + if cp.returncode: + raise RuntimeError(f"guest install failed rc={cp.returncode}") + cp = run([*ssh, "/run/dstack-test-getkeys/run-case", url], timeout=600) + log = cp.stdout + cp.stderr + (art / "mkosi-kms-getkeys.log").write_bytes(log) + if cp.returncode: + diag = run( + [*ssh, "tail -80 /run/dstack-test-getkeys/*.err 2>/dev/null || true"], + timeout=30, + ) + log += diag.stdout + diag.stderr + (art / "mkosi-kms-getkeys.log").write_bytes(log) + raise RuntimeError( + f"mkosi get-keys rc={cp.returncode}: {log.decode(errors='replace')[-1800:]}" + ) + matrix = json.loads( + [x for x in cp.stdout.decode().splitlines() if x.startswith("{")][-1] + ) + ev["matrix"] = matrix + if any( + matrix.get(k) is not True + for k in ( + "valid", + "repeat_stable", + "app_id_scope_preserved", + "retry", + "atomic", + "restrictive", + ) + ) or any( + not isinstance(matrix.get(k), int) or matrix[k] <= 0 + for k in ("bad_app_rc", "wrong_ca_rc", "unreachable_rc", "output_rc") + ): + raise RuntimeError(f"unexpected matrix: {matrix}") + status = "PASS" + summary = "Case-scoped KMS get-keys authorization, TLS, identity, failure, retry, and atomic output checks passed inside mkosi." + except Exception as e: + summary = f"{type(e).__name__}: {e}" + finally: + if ssh: + ev["cleanup_returncode"] = run( + [*ssh, "rm -rf /run/dstack-test-getkeys"], timeout=30 + ).returncode + ev["duration_seconds"] = round(time.monotonic() - started, 3) + dump(art / "kms-getkeys-mkosi.json", ev) + a = { + "path": "artifacts/kms-getkeys-mkosi.json", + "step_id": f"{CASE_ID}-step-01", + "name": "mkosi KMS get-keys suite", + "description": "Sanitized transport, identity, failure, retry, file-mode, and cleanup evidence.", + } + dump(art / "manifest.json", {"artifacts": [a]}) + dump( + r / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + {"id": f"{CASE_ID}-step-{n:02d}", "status": status, "observed": summary} + for n in range(1, 4) + ], + "artifacts": [a], + "remarks": "Case-scoped simulator-backed KMS proves functional authorization and transport behavior, not physical TDX isolation.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-025/case.md b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-025/case.md new file mode 100644 index 000000000..4b2daa676 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-025/case.md @@ -0,0 +1,50 @@ + + + +# TC-GOS-SETUP-025: Streaming environment encryption and decryption + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: UNIT +- Automation: Yes +- Requirements: [req-gos-setup-025](../../../../catalog/feature-audit.md#req-gos-setup-025) +- Risks: [risk-gos-setup-025](../../../../catalog/feature-audit.md#risk-gos-setup-025) +- Source: `dstack/dstack-util/src/crypto.rs`, `dstack/dstack-util/src/main.rs` + +## Objective + +Verify the versioned chunked environment-encryption format, legacy decryption +fallback, authenticated framing, and trusted KMS signer enforcement. + + +### Step 1: Verify streaming round trips + +Exercise empty, single-frame, and multi-frame plaintext with different chunk +boundaries. + +**Expected results:** Encryption and decryption preserve bytes exactly and use +bounded independently authenticated frames. + + +### Step 2: Reject malformed streams + +Mutate authentication tags, truncate and reorder frames, change lengths and +flags, and append trailing data. + +**Expected results:** Every malformed stream fails closed; callers are told to +discard partial output. + + +### Step 3: Verify compatibility and signer binding + +Auto-detect stream ciphertext, fall back to the legacy format, and validate the +timestamped environment public-key signature against the configured KMS key. + +**Expected results:** Legacy input remains readable; an untrusted, expired, or +wrong-app signer cannot authorize encryption. + +## Postconditions + +Retain test names and status only; do not retain plaintext, keys, or ciphertext. diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-025/metadata.json b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-025/metadata.json new file mode 100644 index 000000000..0f82bcf5c --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-025/metadata.json @@ -0,0 +1,20 @@ +{ + "id": "tc-gos-setup-025", + "title": "Streaming environment encryption and decryption", + "priority": "P0", + "requirements": ["req-gos-setup-025"], + "risks": ["risk-gos-setup-025"], + "tags": ["gos", "dstack-util", "stream-encryption"], + "fixture": { + "profile": "component-raw-substrate", + "versions": { + "vmm": "candidate", "guest": "candidate", "kms": "candidate", + "gateway": "candidate", "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": ["Streaming environment encryption and decryption"], + "execution": {"entrypoint": "run.py", "args": [], "timeout_seconds": 600} +} diff --git a/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-025/run.py b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-025/run.py new file mode 100755 index 000000000..9786ad235 --- /dev/null +++ b/test-suites/cases/01-guest-os/12-setup-utilities-simulator/tc-gos-setup-025/run.py @@ -0,0 +1,85 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 +"""Run focused streaming-encryption regression tests.""" + +from __future__ import annotations + +import hashlib +import json +import os +import subprocess +import time +from pathlib import Path + +CASE_ID = "tc-gos-setup-025" +REQUIRED = ( + "crypto::tests::test_stream_roundtrip", + "crypto::tests::test_stream_rejects_tampering_and_truncation", + "tests::decrypt_auto_detects_stream_and_falls_back_to_legacy", + "tests::env_encrypt_public_key_requires_the_trusted_signer", +) + + +def main() -> int: + """Execute the candidate dstack-util test boundary.""" + if os.environ.get("DSTACK_TEST_CASE_ID") != CASE_ID: + raise RuntimeError("unsupported case id") + started = time.monotonic() + result_dir = Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + runtime = json.loads(Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text()) + env = os.environ.copy() + env["CARGO_TARGET_DIR"] = str(runtime["cargo_target_dir"]) + completed = subprocess.run( + ["cargo", "test", "--locked", "--offline", "-p", "dstack-util"], + cwd=Path(str(runtime["repository"])) / "dstack", + env=env, + text=True, + capture_output=True, + timeout=600, + check=False, + ) + output = completed.stdout + completed.stderr + checks = {name: f"test {name} ... ok" in output for name in REQUIRED} + passed = completed.returncode == 0 and all(checks.values()) + artifacts = result_dir / "artifacts" + artifacts.mkdir(parents=True, exist_ok=True) + evidence = artifacts / "stream-encryption-tests.json" + evidence.write_text( + json.dumps( + {"candidate_commit": runtime.get("candidate_commit"), "checks": checks}, + indent=2, + ) + + "\n" + ) + status = "PASS" if passed else "FAIL" + observed = ( + "Streaming round-trip, malformed-frame rejection, legacy fallback, and trusted-signer tests passed." + if passed + else f"Streaming encryption checks failed: {sorted(k for k, value in checks.items() if not value)}" + ) + result = { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": observed, + "steps": [ + {"id": f"{CASE_ID}-step-{n:02d}", "status": status, "observed": observed} + for n in range(1, 4) + ], + "evidence": [ + { + "path": "artifacts/stream-encryption-tests.json", + "sha256": hashlib.sha256(evidence.read_bytes()).hexdigest(), + } + ], + "remarks": "No plaintext, key, or ciphertext is retained.", + "duration_seconds": round(time.monotonic() - started, 3), + } + (result_dir / "result.json").write_text(json.dumps(result, indent=2) + "\n") + return 0 if passed else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/metadata.json b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/metadata.json new file mode 100644 index 000000000..4bec269c3 --- /dev/null +++ b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/metadata.json @@ -0,0 +1,4 @@ +{ + "id": "section-guest-os-yocto-runtime-hardening", + "title": "Yocto Image, Runtime, and Hardening" +} diff --git a/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-002/case.md b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-002/case.md new file mode 100644 index 000000000..3076650d1 --- /dev/null +++ b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-002/case.md @@ -0,0 +1,71 @@ + + + +# TC-GOS-YOCTO-002: OpenSSH account and password-auth hardening + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-gos-yocto-002](../../../../catalog/feature-audit.md#req-gos-yocto-002) +- Risks: [risk-gos-yocto-002](../../../../catalog/feature-audit.md#risk-gos-yocto-002) +- Source: `os/mkosi/mkosi.skeleton/etc/ssh`, `os/mkosi/parity.json` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +- Execute image behavior only when the fixture-provided image provenance reports `builder: mkosi`; an older Yocto image with the same `dstack-0.6.0` or `dstack-dev-0.6.0` name is not valid evidence for this run. + +## Objective + +Verify openssh account and password-auth hardening for documented success, boundary, failure, concurrency, and recovery behavior. + +## Preconditions + +1. Prepare isolated run-scoped inputs and capture effective configuration, service state, files, mounts, processes, network endpoints, and public status. +2. Use sentinel credentials only; evidence records hashes/presence and never the secret value. + +## Test Data + +Include valid values, empty/minimum/maximum values, malformed input, duplicate invocation, a dependency outage, and an adjacent app or node identity. + +## Steps + + +### Step 1: Exercise the complete behavior matrix + +Inspect config and attempt password, empty/default account, root, unauthorized key, authorized key and forwarding modes. + +**Expected results:** + +- Password/default access is disabled, only provisioned keys/policy work, and SSH exposure matches image type without weakening container isolation. + + +### Step 2: Verify failure atomicity and recovery + +Interrupt each external dependency before and after its commit point, issue a duplicate/concurrent request, restore the dependency, and retry. + +**Expected results:** + +- Uncertain input fails closed, no partial trusted output is consumed, resources are released, retry converges once, and diagnostics identify the exact phase without secrets. + + +### Step 3: Verify persistence, isolation, and cleanup + +Restart the owning service or VM where permitted, re-query all affected state, test the adjacent identity, and perform documented cleanup. + +**Expected results:** + +- Persistent/transient state follows policy, the adjacent identity is unchanged, no credential is exposed, and files, mounts, devices, processes, listeners, and counters return to baseline. + +## Postconditions + +Remove run-scoped inputs and faults; preserve redacted native outputs and required attachments. diff --git a/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-002/metadata.json b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-002/metadata.json new file mode 100644 index 000000000..e1fd72e76 --- /dev/null +++ b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-002/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-yocto-002", + "title": "OpenSSH account and password-auth hardening", + "priority": "P0", + "requirements": [ + "req-gos-yocto-002" + ], + "risks": [ + "risk-gos-yocto-002" + ], + "tags": [ + "gos", + "yocto-image-runtime-and-hardening" + ], + "fixture": { + "profile": "no-tee-guest-lifecycle", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "OpenSSH account and password-auth hardening" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 900 + } +} diff --git a/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-002/run.py b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-002/run.py new file mode 100755 index 000000000..ac27d026a --- /dev/null +++ b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-002/run.py @@ -0,0 +1,282 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise OpenSSH hardening inside a lease-owned mkosi guest.""" + +from __future__ import annotations + +import json +import os +import pathlib +import shlex +import subprocess +import tempfile +import time + +CASE_ID = "tc-gos-yocto-002" + + +def run( + argv: list[str], *, data: bytes | None = None, timeout: int = 60 +) -> subprocess.CompletedProcess[bytes]: + """Run one bounded command.""" + return subprocess.run( + argv, input=data, capture_output=True, timeout=timeout, check=False + ) + + +def write_json(path: pathlib.Path, value: object) -> None: + """Write deterministic JSON evidence.""" + path.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n") + + +def ssh_with( + ssh: list[str], options: list[str], command: str, *, user: str | None = None +) -> list[str]: + """Insert client options before the fixture SSH destination.""" + destination = ssh[-1] + if user is not None: + destination = f"{user}@{destination.split('@', 1)[-1]}" + return [*ssh[:-1], *options, destination, command] + + +def main() -> int: + """Run the complete mkosi OpenSSH hardening lifecycle.""" + result = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + artifacts = result / "artifacts" + artifacts.mkdir(parents=True, exist_ok=True) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + values = manifest.get("values") or {} + ssh = [str(item) for item in values.get("ssh_argv") or []] + list_vms = [str(item) for item in values.get("list_vms_argv") or []] + image = str(values.get("image", "")) + status = "FAIL" + summary = "mkosi OpenSSH hardening lifecycle did not execute" + evidence: dict[str, object] = { + "candidate_commit": runtime.get("candidate_commit"), + "guest_image": image, + } + prepared_host_keys: list[str] = [] + started = time.monotonic() + try: + if ( + not ssh + or not list_vms + or values.get("destructive_actions_allowed") is not True + ): + raise RuntimeError("fixture omitted lease-owned SSH or inventory controls") + store = pathlib.Path( + str((runtime.get("environment") or {}).get("DSTACK_TEST_IMAGE_STORE", "")) + ) + metadata = json.loads((store / image / "metadata.json").read_text()) + if metadata.get("builder") != "mkosi" or metadata.get("is_dev") is not True: + raise RuntimeError("fixture did not boot a mkosi development image") + evidence["mkosi"] = { + key: metadata.get(key) for key in ("builder", "is_dev", "git_revision") + } + inventory_before = run(list_vms, timeout=30) + if inventory_before.returncode: + raise RuntimeError("baseline VM inventory query failed") + host_keys = run( + [*ssh, "find /etc/ssh -maxdepth 1 -type f -name 'ssh_host_*_key' -print"], + timeout=30, + ) + if host_keys.returncode: + raise RuntimeError("failed to inventory OpenSSH host keys") + if not host_keys.stdout.strip(): + generated = run([*ssh, "ssh-keygen -A"], timeout=60) + if generated.returncode: + raise RuntimeError( + "failed to prepare ephemeral OpenSSH host keys: " + + generated.stderr.decode(errors="replace")[-500:] + ) + prepared = run( + [ + *ssh, + "find /etc/ssh -maxdepth 1 -type f -name 'ssh_host_*' -print", + ], + timeout=30, + ) + if prepared.returncode or not prepared.stdout.strip(): + raise RuntimeError("OpenSSH host-key preparation produced no files") + prepared_host_keys = prepared.stdout.decode().splitlines() + evidence["host_key_preparation"] = { + "required": bool(prepared_host_keys), + "generated_file_count": len(prepared_host_keys), + } + script = ( + pathlib.Path(str(runtime["repository"])) + / "test-suites/shared/automation/mkosi-openssh-hardening.sh" + ) + installed = run( + [*ssh, "install -m 0755 /dev/stdin /run/dstack-test-openssh"], + data=script.read_bytes(), + timeout=180, + ) + if installed.returncode: + raise RuntimeError( + f"guest script install failed: {installed.stderr.decode(errors='replace')[-500:]}" + ) + checked = run([*ssh, "/run/dstack-test-openssh"], timeout=120) + log = checked.stdout + checked.stderr + (artifacts / "mkosi-openssh.log").write_bytes(log) + if checked.returncode: + raise RuntimeError( + f"mkosi OpenSSH policy rc={checked.returncode}: {log.decode(errors='replace')[-1600:]}" + ) + rows = [ + line + for line in checked.stdout.decode().splitlines() + if line.startswith("{") + ] + matrix = json.loads(rows[-1]) + + authorized = run([*ssh, "true"], timeout=30).returncode == 0 + password = run( + ssh_with( + ssh, + [ + "-o", + "PreferredAuthentications=password", + "-o", + "PubkeyAuthentication=no", + "-o", + "BatchMode=yes", + ], + "true", + ), + timeout=30, + ) + with tempfile.TemporaryDirectory(dir=artifacts) as temporary: + key = pathlib.Path(temporary) / "unauthorized" + generated = run( + ["ssh-keygen", "-q", "-t", "ed25519", "-N", "", "-f", str(key)], + timeout=30, + ) + if generated.returncode: + raise RuntimeError("failed to generate ephemeral unauthorized SSH key") + unauthorized_options = [ + "-o", + "IdentitiesOnly=yes", + "-o", + "BatchMode=yes", + "-i", + str(key), + ] + unauthorized = run(ssh_with(ssh, unauthorized_options, "true"), timeout=30) + empty_account = run( + ssh_with(ssh, unauthorized_options, "true", user="nobody"), timeout=30 + ) + restarted = run([*ssh, "systemctl restart sshd.service"], timeout=30) + recovered = False + for _ in range(30): + if run([*ssh, "true"], timeout=10).returncode == 0: + recovered = True + break + time.sleep(1) + inventory_after = run(list_vms, timeout=30) + if inventory_after.returncode: + raise RuntimeError("recovery VM inventory query failed") + before = json.loads(inventory_before.stdout) + after = json.loads(inventory_after.stdout) + before_ids = sorted( + str(row.get("id")) for row in before if isinstance(row, dict) + ) + after_ids = sorted(str(row.get("id")) for row in after if isinstance(row, dict)) + matrix.update( + { + "authorized_key": authorized, + "password_rejected": password.returncode != 0, + "unauthorized_key_rejected": unauthorized.returncode != 0, + "empty_account_rejected": empty_account.returncode != 0, + "service_restart_attempted": restarted.returncode in (0, 255), + "service_recovered": recovered, + "inventory_stable": before_ids == after_ids, + } + ) + evidence["inventory"] = { + "before_count": len(before_ids), + "after_count": len(after_ids), + } + evidence["matrix"] = matrix + required = ( + "password_auth_disabled", + "empty_password_disabled", + "keyboard_interactive_disabled", + "public_key_enabled", + "root_password_disabled", + "native_config_valid", + "invalid_config_rejected", + "concurrent_validation", + "authorized_key", + "password_rejected", + "unauthorized_key_rejected", + "empty_account_rejected", + "service_restart_attempted", + "service_recovered", + "inventory_stable", + ) + if any(matrix.get(key) is not True for key in required): + raise RuntimeError(f"unexpected OpenSSH matrix: {matrix}") + status = "PASS" + summary = "OpenSSH image policy, authorized and rejected authentication, invalid-config failure, concurrency, restart recovery, and VM isolation passed inside mkosi." + except Exception as error: + summary = f"{type(error).__name__}: {error}" + finally: + if ssh: + if prepared_host_keys: + cleanup_keys = run( + [ + *ssh, + "rm -f " + + " ".join(shlex.quote(path) for path in prepared_host_keys), + ], + timeout=30, + ) + evidence["host_key_cleanup_returncode"] = cleanup_keys.returncode + evidence["cleanup_returncode"] = run( + [*ssh, "rm -f /run/dstack-test-openssh"], timeout=30 + ).returncode + evidence["duration_seconds"] = round(time.monotonic() - started, 3) + write_json(artifacts / "mkosi-openssh.json", evidence) + artifact = { + "path": "artifacts/mkosi-openssh.json", + "step_id": f"{CASE_ID}-step-01", + "name": "mkosi OpenSSH hardening", + "description": "Guest provenance and redacted native policy, authentication, fault, concurrency, recovery, and isolation evidence.", + } + write_json(artifacts / "manifest.json", {"artifacts": [artifact]}) + observed = ( + summary + if status == "FAIL" + else "The lease-owned mkosi guest enforced native password and account hardening while retaining only the provisioned key path across restart." + ) + write_json( + result / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{CASE_ID}-step-{number:02d}", + "status": status, + "observed": observed, + } + for number in range(1, 4) + ], + "artifacts": [artifact], + "remarks": "The mkosi development guest proves OpenSSH image policy and authentication behavior; its lease-installed access path is test tooling and does not claim production SSH exposure.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-003/case.md b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-003/case.md new file mode 100644 index 000000000..ec4bc708e --- /dev/null +++ b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-003/case.md @@ -0,0 +1,72 @@ + + + +# TC-GOS-YOCTO-003: Chrony synchronization and clock recovery + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-gos-yocto-003](../../../../catalog/feature-audit.md#req-gos-yocto-003) +- Risks: [risk-gos-yocto-003](../../../../catalog/feature-audit.md#risk-gos-yocto-003) +- Source: `os/mkosi/mkosi.conf`, `os/mkosi/parity.json` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- The lease-owned guest's `values.ssh_argv` is the fault controller for this case. Run `chronyc tracking`, `chronyc sources`, and `timedatectl` or BusyBox-compatible `date` inside that guest; stop/start only the guest's chrony service, temporarily replace only its lease-owned chrony source configuration, and restore it before cleanup. No separate clock-fault handle is required. Use `values.vm_info_argv` and `values.list_vms_argv` for VM and adjacent-inventory observations. Never alter or reboot the physical host. + +- Execute image behavior only when the fixture-provided image provenance reports `builder: mkosi`; an older Yocto image with the same `dstack-0.6.0` or `dstack-dev-0.6.0` name is not valid evidence for this run. + +## Objective + +Verify chrony synchronization and clock recovery for documented success, boundary, failure, concurrency, and recovery behavior. + +## Preconditions + +1. Prepare isolated run-scoped inputs and capture effective configuration, service state, files, mounts, processes, network endpoints, and public status. +2. Use sentinel credentials only; evidence records hashes/presence and never the secret value. + +## Test Data + +Include valid values, empty/minimum/maximum values, malformed input, duplicate invocation, a dependency outage, and an adjacent app or node identity. + +## Steps + + +### Step 1: Exercise the complete behavior matrix + +Boot with good/bad/unreachable sources, large forward/backward skew, network recovery and restart while observing certificate/attestation consumers. + +**Expected results:** + +- Time converges within policy, unsafe jumps are controlled, readiness does not falsely claim valid time, and dependent services recover after synchronization. + + +### Step 2: Verify failure atomicity and recovery + +Interrupt each external dependency before and after its commit point, issue a duplicate/concurrent request, restore the dependency, and retry. + +**Expected results:** + +- Uncertain input fails closed, no partial trusted output is consumed, resources are released, retry converges once, and diagnostics identify the exact phase without secrets. + + +### Step 3: Verify persistence, isolation, and cleanup + +Restart the owning service or VM where permitted, re-query all affected state, test the adjacent identity, and perform documented cleanup. + +**Expected results:** + +- Persistent/transient state follows policy, the adjacent identity is unchanged, no credential is exposed, and files, mounts, devices, processes, listeners, and counters return to baseline. + +## Postconditions + +Remove run-scoped inputs and faults; preserve redacted native outputs and required attachments. diff --git a/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-003/metadata.json b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-003/metadata.json new file mode 100644 index 000000000..dec0f14c8 --- /dev/null +++ b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-003/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-yocto-003", + "title": "Chrony synchronization and clock recovery", + "priority": "P0", + "requirements": [ + "req-gos-yocto-003" + ], + "risks": [ + "risk-gos-yocto-003" + ], + "tags": [ + "gos", + "yocto-image-runtime-and-hardening" + ], + "fixture": { + "profile": "no-tee-guest-lifecycle", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "Chrony synchronization and clock recovery" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 900 + } +} diff --git a/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-003/run.py b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-003/run.py new file mode 100755 index 000000000..5f179b6d1 --- /dev/null +++ b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-003/run.py @@ -0,0 +1,171 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise chrony failure and recovery inside a lease-owned mkosi guest.""" + +from __future__ import annotations + +import json +import os +import pathlib +import subprocess +import time + +CASE_ID = "tc-gos-yocto-003" + + +def run( + argv: list[str], *, data: bytes | None = None, timeout: int = 60 +) -> subprocess.CompletedProcess[bytes]: + """Run one bounded controller or guest command.""" + return subprocess.run( + argv, input=data, capture_output=True, timeout=timeout, check=False + ) + + +def write_json(path: pathlib.Path, value: object) -> None: + """Write deterministic JSON evidence.""" + path.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n") + + +def main() -> int: + """Run the complete lease-owned mkosi chrony lifecycle.""" + result = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + artifacts = result / "artifacts" + artifacts.mkdir(parents=True, exist_ok=True) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + values = manifest.get("values") or {} + ssh = [str(item) for item in values.get("ssh_argv") or []] + image = str(values.get("image", "")) + status = "FAIL" + summary = "mkosi chrony lifecycle did not execute" + evidence: dict[str, object] = { + "candidate_commit": runtime.get("candidate_commit"), + "guest_image": image, + } + started = time.monotonic() + try: + if not ssh or values.get("destructive_actions_allowed") is not True: + raise RuntimeError("fixture omitted lease-owned guest SSH") + store = pathlib.Path( + str((runtime.get("environment") or {}).get("DSTACK_TEST_IMAGE_STORE", "")) + ) + metadata = json.loads((store / image / "metadata.json").read_text()) + if metadata.get("builder") != "mkosi" or metadata.get("is_dev") is not True: + raise RuntimeError("fixture did not boot a mkosi development image") + evidence["mkosi"] = { + key: metadata.get(key) for key in ("builder", "is_dev", "git_revision") + } + script = ( + pathlib.Path(str(runtime["repository"])) + / "test-suites/shared/automation/mkosi-chrony-lifecycle.sh" + ) + installed = run( + [*ssh, "install -m 0755 /dev/stdin /run/dstack-test-chrony"], + data=script.read_bytes(), + timeout=180, + ) + if installed.returncode: + raise RuntimeError( + f"guest script install failed: {installed.stderr.decode(errors='replace')[-500:]}" + ) + list_vms = [str(item) for item in values.get("list_vms_argv") or []] + if not list_vms: + raise RuntimeError("fixture omitted adjacent VM inventory observer") + inventory_before = run(list_vms, timeout=30) + if inventory_before.returncode: + raise RuntimeError("baseline VM inventory query failed") + completed = run([*ssh, "/run/dstack-test-chrony"], timeout=300) + log = completed.stdout + completed.stderr + (artifacts / "mkosi-chrony.log").write_bytes(log) + if completed.returncode: + raise RuntimeError( + f"mkosi chrony rc={completed.returncode}: {log.decode(errors='replace')[-1600:]}" + ) + rows = [ + line + for line in completed.stdout.decode().splitlines() + if line.startswith("{") + ] + matrix = json.loads(rows[-1]) + inventory_after = run(list_vms, timeout=30) + if inventory_after.returncode: + raise RuntimeError("recovery VM inventory query failed") + before_rows = json.loads(inventory_before.stdout) + after_rows = json.loads(inventory_after.stdout) + before_ids = sorted( + str(row.get("id")) for row in before_rows if isinstance(row, dict) + ) + after_ids = sorted( + str(row.get("id")) for row in after_rows if isinstance(row, dict) + ) + matrix["inventory_stable"] = before_ids == after_ids + evidence["inventory"] = { + "before_count": len(before_ids), + "after_count": len(after_ids), + } + evidence["matrix"] = matrix + required = ( + "baseline_active", + "stop_observed", + "unreachable_source_observed", + "concurrent_restart", + "recovered_active", + "config_restored", + "inventory_stable", + "cleanup", + ) + if any(matrix.get(key) is not True for key in required): + raise RuntimeError(f"unexpected chrony matrix: {matrix}") + status = "PASS" + summary = "Chrony baseline, outage, concurrent restart, recovery, configuration restoration, and adjacent-VM isolation passed inside mkosi." + except Exception as error: + summary = f"{type(error).__name__}: {error}" + finally: + if ssh: + evidence["cleanup_returncode"] = run( + [*ssh, "rm -f /run/dstack-test-chrony"], timeout=30 + ).returncode + evidence["duration_seconds"] = round(time.monotonic() - started, 3) + write_json(artifacts / "mkosi-chrony.json", evidence) + artifact = { + "path": "artifacts/mkosi-chrony.json", + "step_id": f"{CASE_ID}-step-01", + "name": "mkosi chrony lifecycle", + "description": "Guest provenance and redacted chrony baseline, outage, concurrency, recovery, isolation, and cleanup evidence.", + } + write_json(artifacts / "manifest.json", {"artifacts": [artifact]}) + observed = ( + summary + if status == "FAIL" + else "The lease-owned mkosi guest restored its exact chrony configuration and healthy service state after controlled local faults." + ) + write_json( + result / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{CASE_ID}-step-{number:02d}", + "status": status, + "observed": observed, + } + for number in range(1, 4) + ], + "artifacts": [artifact], + "remarks": "The mkosi simulator guest proves chrony configuration, service, dependency-fault, recovery, and isolation behavior; it does not prove a physical TEE clock source.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-004/case.md b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-004/case.md new file mode 100644 index 000000000..9e88bebe7 --- /dev/null +++ b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-004/case.md @@ -0,0 +1,72 @@ + + + +# TC-GOS-YOCTO-004: Containerd stargz snapshotter integrity and fallback + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-gos-yocto-004](../../../../catalog/feature-audit.md#req-gos-yocto-004) +- Risks: [risk-gos-yocto-004](../../../../catalog/feature-audit.md#risk-gos-yocto-004) +- Source: `os/mkosi/components/container-stack`, `os/mkosi/parity.json` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- The guest image is BusyBox based: it does not provide an in-guest `timeout` command, GNU short `head -8` syntax, or procps `ps -p`. Apply timeouts around each host-side invocation of `values.ssh_argv`; inside the guest use BusyBox-compatible `head -n 8`, `ps -o`, `systemctl`, `ctr`, `nerdctl`, and `containerd-stargz-grpc` commands. A missing convenience utility or incompatible probe syntax is a test-probe defect and must be corrected before grading the candidate. Stop/start only lease-owned guest services and restore their configuration. + +- Execute image behavior only when the fixture-provided image provenance reports `builder: mkosi`; an older Yocto image with the same `dstack-0.6.0` or `dstack-dev-0.6.0` name is not valid evidence for this run. + +## Objective + +Verify containerd stargz snapshotter integrity and fallback for documented success, boundary, failure, concurrency, and recovery behavior. + +## Preconditions + +1. Prepare isolated run-scoped inputs and capture effective configuration, service state, files, mounts, processes, network endpoints, and public status. +2. Use sentinel credentials only; evidence records hashes/presence and never the secret value. + +## Test Data + +Include valid values, empty/minimum/maximum values, malformed input, duplicate invocation, a dependency outage, and an adjacent app or node identity. + +## Steps + + +### Step 1: Exercise the complete behavior matrix + +Pull and run OCI-digest-verified normal and lazy images, then exercise corrupted content, an unavailable registry, snapshotter restart, cache reuse, concurrency, and explicit overlay fallback. + +**Expected results:** + +- OCI digest-verified content runs with the selected snapshotter, corrupt layers never execute, the explicit caller-selected overlay fallback follows policy, and cache/restart preserves isolation; no silent automatic fallback is claimed. + + +### Step 2: Verify failure atomicity and recovery + +Interrupt each external dependency before and after its commit point, issue a duplicate/concurrent request, restore the dependency, and retry. + +**Expected results:** + +- Uncertain input fails closed, no partial trusted output is consumed, resources are released, retry converges once, and diagnostics identify the exact phase without secrets. + + +### Step 3: Verify persistence, isolation, and cleanup + +Restart the owning service or VM where permitted, re-query all affected state, test the adjacent identity, and perform documented cleanup. + +**Expected results:** + +- Persistent/transient state follows policy, the adjacent identity is unchanged, no credential is exposed, and files, mounts, devices, processes, listeners, and counters return to baseline. + +## Postconditions + +Remove run-scoped inputs and faults; preserve redacted native outputs and required attachments. diff --git a/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-004/metadata.json b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-004/metadata.json new file mode 100644 index 000000000..2c06817e1 --- /dev/null +++ b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-004/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-yocto-004", + "title": "Containerd stargz snapshotter integrity and fallback", + "priority": "P0", + "requirements": [ + "req-gos-yocto-004" + ], + "risks": [ + "risk-gos-yocto-004" + ], + "tags": [ + "gos", + "yocto-image-runtime-and-hardening" + ], + "fixture": { + "profile": "container-observability", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Containerd stargz snapshotter integrity and fallback" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 60 + } +} diff --git a/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-004/run.py b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-004/run.py new file mode 100755 index 000000000..312848429 --- /dev/null +++ b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-004/run.py @@ -0,0 +1,184 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise stargz integrity, fault handling, restart/cache, and explicit fallback.""" + +from __future__ import annotations + +import json +import os +import pathlib +import subprocess +from typing import Any + +CASE_ID = "tc-gos-yocto-004" + + +def run( + argv: list[str], *, data: bytes | None = None, timeout: int = 60 +) -> subprocess.CompletedProcess[bytes]: + """Run a bounded command.""" + return subprocess.run( + argv, input=data, capture_output=True, timeout=timeout, check=False + ) + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + temporary = path.with_suffix(path.suffix + ".tmp") + temporary.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n") + temporary.replace(path) + + +def main() -> int: + """Run the real mkosi stargz lifecycle matrix.""" + result = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + artifacts = result / "artifacts" + artifacts.mkdir(parents=True, exist_ok=True) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + values = manifest.get("values") or {} + fixture = values.get("stargz_lifecycle") or {} + ssh = [str(value) for value in values.get("ssh_argv") or []] + status = "PASS" + summary = "Containerd stargz integrity and explicit fallback lifecycle passed." + evidence: dict[str, Any] = {} + try: + required = ( + "payload_image", + "payload_image_id", + "registry_image", + "registry_image_id", + "snapshotter_unit", + "snapshotter_name", + ) + if not ssh or any(not fixture.get(key) for key in required): + raise RuntimeError("fixture omitted pinned stargz substrate") + if values.get("image") != runtime.get("environment", {}).get( + "DSTACK_TEST_GUEST_IMAGE" + ): + raise RuntimeError( + "fixture did not select the prepared mkosi production image" + ) + script = ( + pathlib.Path(str(runtime["repository"])) + / "test-suites/shared/automation/stargz-integrity-lifecycle.sh" + ) + installed = run( + [*ssh, "install -m 0755 /dev/stdin /run/dstack-test-stargz-lifecycle"], + data=script.read_bytes(), + ) + if installed.returncode: + raise RuntimeError("lifecycle script installation failed") + executed = run( + [ + *ssh, + "/run/dstack-test-stargz-lifecycle", + str(fixture["payload_image"]), + str(fixture["payload_image_id"]), + str(fixture["registry_image"]), + str(fixture["registry_image_id"]), + str(fixture["snapshotter_unit"]), + str(fixture["snapshotter_name"]), + ], + timeout=300, + ) + (artifacts / "stargz-lifecycle.log").write_bytes( + executed.stdout + executed.stderr + ) + rows = [ + row + for row in executed.stdout.decode(errors="replace").splitlines() + if row.startswith("{") + ] + if executed.returncode or not rows: + tail = (executed.stdout + executed.stderr).decode(errors="replace")[-2000:] + raise RuntimeError(f"lifecycle rc={executed.returncode}: {tail}") + evidence = json.loads(rows[-1]) + required_checks = { + "overlay_baseline", + "lazy_execution", + "restart_recovery", + "overlay_cache_outage", + "unavailable_registry_rejected", + "corrupt_layer_rejected", + "snapshotter_outage_rejected", + "explicit_overlay_fallback", + } + if not all(evidence.get(key) is True for key in required_checks): + raise RuntimeError("lifecycle evidence omitted a required successful row") + if ( + evidence.get("concurrent_pulls") != 2 + or evidence.get("silent_fallback_claimed") is not False + ): + raise RuntimeError("concurrency or fallback semantics were not proven") + except ( + KeyError, + OSError, + RuntimeError, + subprocess.SubprocessError, + ValueError, + ) as error: + status = "FAIL" + summary = f"{type(error).__name__}: {error}" + + artifact_entries = [ + { + "path": "artifacts/stargz-lifecycle.json", + "step_id": f"{CASE_ID}-step-01", + "name": "Stargz lifecycle matrix", + "description": "Pinned digests and booleans for overlay baseline, lazy execution, concurrency, restart/cache, corruption and outage rejection, and explicit fallback.", + }, + { + "path": "artifacts/stargz-lifecycle.log", + "step_id": f"{CASE_ID}-step-02", + "name": "Stargz native lifecycle log", + "description": "Native bounded command output for the case-scoped registry and snapshotter lifecycle; no credentials are used.", + }, + ] + atomic_json(artifacts / "stargz-lifecycle.json", evidence) + atomic_json(artifacts / "manifest.json", {"artifacts": artifact_entries}) + steps = [ + { + "id": f"{CASE_ID}-step-01", + "status": status, + "observed": "Verified normal overlay and optimized eStargz execution with two concurrent pulls." + if status == "PASS" + else summary, + }, + { + "id": f"{CASE_ID}-step-02", + "status": status, + "observed": "Registry outage, corrupted layer, and stopped snapshotter failed closed; restart recovered." + if status == "PASS" + else summary, + }, + { + "id": f"{CASE_ID}-step-03", + "status": status, + "observed": "Stargz execution recovered after restart, overlay cache survived registry outage, and explicit fallback remained isolated; cleanup restored the packaged unit gate." + if status == "PASS" + else summary, + }, + ] + atomic_json( + result / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": summary, + "steps": steps, + "artifacts": artifact_entries, + "remarks": "Stargz content integrity is OCI digest verification. The product exposes an explicit caller-selected overlay fallback; this case does not claim silent automatic fallback.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-005/case.md b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-005/case.md new file mode 100644 index 000000000..e4f28148e --- /dev/null +++ b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-005/case.md @@ -0,0 +1,72 @@ + + + +# TC-GOS-YOCTO-005: Sysbox runtime services and nested-container boundary + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: mkosi guest with Sysbox +- Automation: Yes +- Requirements: [req-gos-yocto-005](../../../../catalog/feature-audit.md#req-gos-yocto-005) +- Risks: [risk-gos-yocto-005](../../../../catalog/feature-audit.md#risk-gos-yocto-005) +- Source: `os/mkosi/components/sysbox`, `os/mkosi/parity.json` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +- Execute image behavior only when the fixture-provided image provenance reports `builder: mkosi`; an older Yocto image with the same `dstack-0.6.0` or `dstack-dev-0.6.0` name is not valid evidence for this run. + +## Objective + +Verify sysbox runtime services and nested-container boundary for documented success, boundary, failure, concurrency, and recovery behavior. + +## Preconditions + +1. Prepare isolated run-scoped inputs and capture effective configuration, service state, files, mounts, processes, network endpoints, and public status. +2. Use sentinel credentials only; evidence records hashes/presence and never the secret value. + +## Test Data + +Include valid values, empty/minimum/maximum values, malformed input, duplicate invocation, a dependency outage, and an adjacent app or node identity. + +## Steps + + +### Step 1: Exercise the complete behavior matrix + +Start/stop/restart Sysbox services, verify UID/GID remapping, and run a pinned Docker-in-Docker workload that requests mounts, proc/sys, devices, and cgroups from its outer Sysbox container. + +**Expected results:** + +- Supported nested containers work while the physical host, VMM control plane, agent sockets, `/dev/kvm`, and resources outside the outer Sysbox container remain protected. +- `/dev/tdx_guest` and virtual disks belong to the lease-owned guest and are not physical-host devices; their presence alone is not a boundary failure. + + +### Step 2: Verify failure atomicity and recovery + +Interrupt each external dependency before and after its commit point, issue a duplicate/concurrent request, restore the dependency, and retry. + +**Expected results:** + +- Uncertain input fails closed, no partial trusted output is consumed, resources are released, retry converges once, and diagnostics identify the exact phase without secrets. + + +### Step 3: Verify persistence, isolation, and cleanup + +Restart the owning services, re-query affected state and adjacent VM inventory, and perform documented cleanup. A VM reboot is not required because this case exercises runtime lifecycle rather than image construction or boot correctness. + +**Expected results:** + +- Persistent/transient state follows policy, the adjacent identity is unchanged, no credential is exposed, and files, mounts, devices, processes, listeners, and counters return to baseline. + +## Postconditions + +Remove run-scoped inputs and faults; preserve redacted native outputs and required attachments. diff --git a/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-005/metadata.json b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-005/metadata.json new file mode 100644 index 000000000..18b0c6b66 --- /dev/null +++ b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-005/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-yocto-005", + "title": "Sysbox runtime services and nested-container boundary", + "priority": "P0", + "requirements": [ + "req-gos-yocto-005" + ], + "risks": [ + "risk-gos-yocto-005" + ], + "tags": [ + "gos", + "yocto-image-runtime-and-hardening" + ], + "fixture": { + "profile": "container-observability", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Sysbox runtime services and nested-container boundary" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-005/run.py b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-005/run.py new file mode 100755 index 000000000..9a34fab80 --- /dev/null +++ b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-005/run.py @@ -0,0 +1,190 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise Sysbox services, nested containers, fault closure, and recovery.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import subprocess +import time + +CASE_ID = "tc-gos-yocto-005" + + +def run(argv, *, data=None, timeout=60): + """Run one bounded host or guest command.""" + return subprocess.run( + argv, input=data, capture_output=True, timeout=timeout, check=False + ) + + +def write(path, value): + """Write deterministic JSON evidence.""" + path.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n") + + +def main(): + """Execute the complete lease-owned Sysbox lifecycle.""" + result = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + artifacts = result / "artifacts" + artifacts.mkdir(parents=True, exist_ok=True) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + values = manifest.get("values") or {} + fixture = values.get("sysbox_lifecycle") or {} + ssh = [str(x) for x in values.get("ssh_argv") or []] + status = "FAIL" + summary = "Sysbox lifecycle did not execute" + started = time.monotonic() + evidence = { + "candidate_commit": runtime.get("candidate_commit"), + "guest_image": values.get("image"), + } + try: + required = ( + "nested_workload_image", + "nested_workload_image_digest", + "nested_workload_image_id", + "nested_payload_image", + "nested_payload_image_digest", + "nested_payload_image_id", + "service_units", + "runtime_name", + ) + if ( + not ssh + or values.get("destructive_actions_allowed") is not True + or fixture.get("destructive_actions_allowed") is not True + ): + raise RuntimeError("fixture omitted lease-owned destructive guest control") + if any(not fixture.get(k) for k in required): + raise RuntimeError("fixture omitted pinned Sysbox lifecycle inputs") + store = pathlib.Path( + str((runtime.get("environment") or {}).get("DSTACK_TEST_IMAGE_STORE", "")) + ) + metadata = json.loads( + (store / str(values["image"]) / "metadata.json").read_text() + ) + if metadata.get("builder") != "mkosi": + raise RuntimeError("fixture did not boot a mkosi image") + evidence["mkosi"] = { + k: metadata.get(k) for k in ("builder", "is_dev", "git_revision") + } + script = ( + pathlib.Path(str(runtime["repository"])) + / "test-suites/shared/automation/sysbox-boundary-lifecycle.sh" + ) + installed = run( + [*ssh, "install -m 0755 /dev/stdin /run/dstack-test-sysbox-case"], + data=script.read_bytes(), + timeout=60, + ) + if installed.returncode: + raise RuntimeError("guest script installation failed") + list_vms = [str(x) for x in values.get("list_vms_argv") or []] + before = run(list_vms, timeout=30) + if before.returncode: + raise RuntimeError("baseline VM inventory query failed") + args = [ + fixture[k] + for k in ( + "nested_workload_image", + "nested_workload_image_digest", + "nested_workload_image_id", + "nested_payload_image", + "nested_payload_image_digest", + "nested_payload_image_id", + ) + ] + completed = run( + [*ssh, "/run/dstack-test-sysbox-case", *map(str, args)], timeout=300 + ) + (artifacts / "sysbox-lifecycle.log").write_bytes( + completed.stdout + completed.stderr + ) + if completed.returncode: + raise RuntimeError( + f"guest lifecycle rc={completed.returncode}: {(completed.stdout + completed.stderr).decode(errors='replace')[-1200:]}" + ) + rows = [x for x in completed.stdout.decode().splitlines() if x.startswith("{")] + matrix = json.loads(rows[-1]) + after = run(list_vms, timeout=30) + if after.returncode: + raise RuntimeError("recovery VM inventory query failed") + + def ids(blob): + return sorted( + str(x.get("id")) for x in json.loads(blob) if isinstance(x, dict) + ) + + matrix["inventory_stable"] = ids(before.stdout) == ids(after.stdout) + required_rows = ( + "baseline", + "lifecycle", + "nested_boundary", + "failure_closed", + "partial_recovery_closed", + "recovered", + "cleanup", + "inventory_stable", + ) + if any(matrix.get(k) is not True for k in required_rows): + raise RuntimeError(f"unexpected Sysbox matrix: {matrix}") + evidence["matrix"] = matrix + status = "PASS" + summary = "Sysbox baseline, remapped lifecycle, true nested container boundary, failure closure, recovery, cleanup, and adjacent-VM isolation passed." + except Exception as error: + summary = f"{type(error).__name__}: {error}" + finally: + if ssh: + evidence["cleanup_returncode"] = run( + [ + *ssh, + "docker rm -f sysbox-case-outer sysbox-case-fast sysbox-case-fault >/dev/null 2>&1 || true; systemctl start sysbox-mgr.service sysbox-fs.service sysbox.service; rm -rf /run/dstack-test-sysbox /run/dstack-test-sysbox-case", + ], + timeout=60, + ).returncode + evidence["duration_seconds"] = round(time.monotonic() - started, 3) + path = artifacts / "sysbox-boundary-lifecycle.json" + write(path, evidence) + artifact = { + "path": "artifacts/sysbox-boundary-lifecycle.json", + "step_id": f"{CASE_ID}-step-01", + "name": "Sysbox boundary lifecycle", + "description": "Redacted mkosi provenance, remapping, nested workload, fault closure, recovery, cleanup, and adjacent-VM evidence.", + } + write(artifacts / "manifest.json", {"artifacts": [artifact]}) + write( + result / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + {"id": f"{CASE_ID}-step-{n:02d}", "status": status, "observed": summary} + for n in range(1, 4) + ], + "artifacts": [artifact], + "evidence": [ + { + "path": artifact["path"], + "sha256": hashlib.sha256(path.read_bytes()).hexdigest(), + } + ], + "remarks": "The test protects the physical host, VMM control plane, agent sockets, and /dev/kvm. Guest-scoped /dev/tdx_guest and guest virtual disks are intentionally not treated as physical-host devices.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-006/case.md b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-006/case.md new file mode 100644 index 000000000..6d9a037b5 --- /dev/null +++ b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-006/case.md @@ -0,0 +1,71 @@ + + + +# TC-GOS-YOCTO-006: Docker daemon CPU/GPU configuration variants + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-gos-yocto-006](../../../../catalog/feature-audit.md#req-gos-yocto-006) +- Risks: [risk-gos-yocto-006](../../../../catalog/feature-audit.md#risk-gos-yocto-006) +- Source: `os/mkosi/mkosi.skeleton/etc/docker`, `os/mkosi/mkosi.profiles` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +- Execute image behavior only when the fixture-provided image provenance reports `builder: mkosi`; an older Yocto image with the same `dstack-0.6.0` or `dstack-dev-0.6.0` name is not valid evidence for this run. + +## Objective + +Verify docker daemon cpu/gpu configuration variants for documented success, boundary, failure, concurrency, and recovery behavior. + +## Preconditions + +1. Prepare isolated run-scoped inputs and capture effective configuration, service state, files, mounts, processes, network endpoints, and public status. +2. Use sentinel credentials only; evidence records hashes/presence and never the secret value. + +## Test Data + +Include valid values, empty/minimum/maximum values, malformed input, duplicate invocation, a dependency outage, and an adjacent app or node identity. + +## Steps + + +### Step 1: Exercise the complete behavior matrix + +Validate normal/NVIDIA daemon JSON, runtimes, default runtime, cgroups, logging, restart and malformed override. + +**Expected results:** + +- Each image selects only installed runtime, GPU workloads receive assigned devices, normal image does not advertise NVIDIA, and bad config fails before apps. + + +### Step 2: Verify failure atomicity and recovery + +Interrupt each external dependency before and after its commit point, issue a duplicate/concurrent request, restore the dependency, and retry. + +**Expected results:** + +- Uncertain input fails closed, no partial trusted output is consumed, resources are released, retry converges once, and diagnostics identify the exact phase without secrets. + + +### Step 3: Verify persistence, isolation, and cleanup + +Restart the owning service or VM where permitted, re-query all affected state, test the adjacent identity, and perform documented cleanup. + +**Expected results:** + +- Persistent/transient state follows policy, the adjacent identity is unchanged, no credential is exposed, and files, mounts, devices, processes, listeners, and counters return to baseline. + +## Postconditions + +Remove run-scoped inputs and faults; preserve redacted native outputs and required attachments. diff --git a/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-006/metadata.json b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-006/metadata.json new file mode 100644 index 000000000..7e472af3c --- /dev/null +++ b/test-suites/cases/01-guest-os/13-yocto-runtime-hardening/tc-gos-yocto-006/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-gos-yocto-006", + "title": "Docker daemon CPU/GPU configuration variants", + "priority": "P0", + "requirements": [ + "req-gos-yocto-006" + ], + "risks": [ + "risk-gos-yocto-006" + ], + "tags": [ + "gos", + "yocto-image-runtime-and-hardening" + ], + "fixture": { + "profile": "gpu-policy", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": true, + "simulation_allowed": false + }, + "actions_under_test": [ + "Docker daemon CPU/GPU configuration variants" + ], + "execution": { + "entrypoint": "shared/automation/capability-probe-case.py", + "args": [], + "timeout_seconds": 60 + } +} diff --git a/test-suites/cases/01-guest-os/14-gos-build/metadata.json b/test-suites/cases/01-guest-os/14-gos-build/metadata.json new file mode 100644 index 000000000..0d80f63a7 --- /dev/null +++ b/test-suites/cases/01-guest-os/14-gos-build/metadata.json @@ -0,0 +1,4 @@ +{ + "id": "section-gos-build", + "title": "Guest OS Build and Existing Regression Suite" +} diff --git a/test-suites/cases/01-guest-os/14-gos-build/tc-gos-build-001/case.md b/test-suites/cases/01-guest-os/14-gos-build/tc-gos-build-001/case.md new file mode 100644 index 000000000..842c2951f --- /dev/null +++ b/test-suites/cases/01-guest-os/14-gos-build/tc-gos-build-001/case.md @@ -0,0 +1,55 @@ + + + +# TC-GOS-BUILD-001: Guest image builder provenance + +## Metadata + +- Priority: P0 +- Type: Functional, Regression, Supply Chain +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-gos-build-001](../../../../catalog/feature-audit.md#req-gos-build-001) +- Risks: [risk-gos-build-001](../../../../catalog/feature-audit.md#risk-gos-build-001) +- Source: `os/image/assemble.sh`, `os/mkosi/tests/check-output.sh` + +## Objective + +Verify that an assembled candidate guest image records the selected builder and +that the mkosi output contract rejects metadata that does not identify mkosi. + +## Preconditions + +1. Provide a protected candidate image store through the `image-assembly` fixture. +2. Record the expected builder in `DSTACK_TEST_GUEST_IMAGE_BUILDER`. + + +### Step 1: Validate the assembly and output-check scripts + +Run bounded shell syntax validation on the candidate assembly script and mkosi +output checker. + +**Expected results:** Both candidate scripts parse successfully. + + +### Step 2: Inspect candidate artifact provenance + +Read the fixture-selected candidate image's `metadata.json` and compare its +`builder` field with the expected image builder. + +**Expected results:** `builder` is present, non-empty, and equals the selected +backend; a legacy-only `backend` field is not accepted as provenance. + + +### Step 3: Verify the mkosi contract + +Confirm the candidate mkosi output checker requires `builder` and compares it +with `mkosi` before accepting an artifact. + +**Expected results:** The checked-in contract cannot accept output metadata that +omits or misidentifies the builder. + +## Postconditions + +Release the fixture without modifying the protected image store. Retain only +the builder name, candidate revision, boolean checks, and hashes. diff --git a/test-suites/cases/01-guest-os/14-gos-build/tc-gos-build-001/metadata.json b/test-suites/cases/01-guest-os/14-gos-build/tc-gos-build-001/metadata.json new file mode 100644 index 000000000..7799ca0e6 --- /dev/null +++ b/test-suites/cases/01-guest-os/14-gos-build/tc-gos-build-001/metadata.json @@ -0,0 +1,37 @@ +{ + "id": "tc-gos-build-001", + "title": "Guest image builder provenance", + "priority": "P0", + "requirements": [ + "req-gos-build-001" + ], + "risks": [ + "risk-gos-build-001" + ], + "tags": [ + "gos", + "build", + "provenance" + ], + "fixture": { + "profile": "image-assembly", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Guest image builder provenance" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/01-guest-os/14-gos-build/tc-gos-build-001/run.py b/test-suites/cases/01-guest-os/14-gos-build/tc-gos-build-001/run.py new file mode 100755 index 000000000..e3feda4d1 --- /dev/null +++ b/test-suites/cases/01-guest-os/14-gos-build/tc-gos-build-001/run.py @@ -0,0 +1,107 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 +"""Verify builder provenance on a candidate guest image artifact.""" + +from __future__ import annotations + +import hashlib +import json +import os +import subprocess +import time +from pathlib import Path + +CASE_ID = "tc-gos-build-001" + + +def main() -> int: + """Validate candidate scripts and artifact metadata without mutating it.""" + if os.environ.get("DSTACK_TEST_CASE_ID") != CASE_ID: + raise RuntimeError("unsupported case id") + started = time.monotonic() + result_dir = Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + runtime = json.loads(Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text()) + manifest = json.loads(Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text()) + values = (manifest.get("values") or {}).get("image_assembly") or {} + repository = Path(str(runtime["repository"])) + assemble = repository / "os/image/assemble.sh" + check_output = repository / "os/mkosi/tests/check-output.sh" + syntax = subprocess.run( + ["bash", "-n", str(assemble), str(check_output)], + text=True, + capture_output=True, + timeout=30, + check=False, + ) + metadata_path = Path(str(values.get("input_dir", ""))) / "metadata.json" + metadata = json.loads(metadata_path.read_text(encoding="utf-8")) + expected = os.environ.get("DSTACK_TEST_GUEST_IMAGE_BUILDER", "mkosi").strip() + checker = check_output.read_text(encoding="utf-8") + checks = { + "scripts_parse": syntax.returncode == 0, + "builder_present": isinstance(metadata.get("builder"), str) + and bool(metadata["builder"]), + "builder_matches": metadata.get("builder") == expected, + "mkosi_checker_requires_builder": '"builder"' in checker, + "mkosi_checker_matches_builder": 'd["builder"] == "mkosi"' in checker, + } + passed = all(checks.values()) + artifacts = result_dir / "artifacts" + artifacts.mkdir(parents=True, exist_ok=True) + evidence_path = artifacts / "builder-provenance.json" + evidence_path.write_text( + json.dumps( + { + "candidate_commit": runtime.get("candidate_commit"), + "image": values.get("candidate_image"), + "builder": metadata.get("builder"), + "expected_builder": expected, + "metadata_sha256": hashlib.sha256( + metadata_path.read_bytes() + ).hexdigest(), + "checks": checks, + }, + indent=2, + sort_keys=True, + ) + + "\n", + encoding="utf-8", + ) + status = "PASS" if passed else "FAIL" + observed = ( + f"Candidate image records builder={expected!r}, and the mkosi output contract enforces it." + if passed + else f"Builder provenance checks failed: {sorted(k for k, value in checks.items() if not value)}" + ) + result = { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": observed, + "steps": [ + { + "id": f"{CASE_ID}-step-{number:02d}", + "status": status, + "observed": observed, + } + for number in range(1, 4) + ], + "evidence": [ + { + "path": "artifacts/builder-provenance.json", + "sha256": hashlib.sha256(evidence_path.read_bytes()).hexdigest(), + } + ], + "remarks": "The protected image store was read-only; retained evidence contains no credentials.", + "duration_seconds": round(time.monotonic() - started, 3), + } + (result_dir / "result.json").write_text( + json.dumps(result, indent=2) + "\n", encoding="utf-8" + ) + return 0 if passed else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/01-guest-os/metadata.json b/test-suites/cases/01-guest-os/metadata.json new file mode 100644 index 000000000..95300064a --- /dev/null +++ b/test-suites/cases/01-guest-os/metadata.json @@ -0,0 +1,4 @@ +{ + "id": "chapter-guest-os", + "title": "Guest OS" +} diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/metadata.json b/test-suites/cases/02-vmm/01-rpc-vmm/metadata.json new file mode 100644 index 000000000..11ee00ef0 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/metadata.json @@ -0,0 +1,4 @@ +{ + "id": "section-vmm-rpc-vmm", + "title": "Vmm RPC" +} diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-001/case.md b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-001/case.md new file mode 100644 index 000000000..88844a2ff --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-001/case.md @@ -0,0 +1,76 @@ + + + +# TC-VMM-VMM-001: Vmm.CreateVm + +## Metadata + +- Priority: P0 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vmm-001](../../../../catalog/feature-audit.md#req-vmm-vmm-001) +- Risks: [risk-vmm-vmm-001](../../../../catalog/feature-audit.md#risk-vmm-vmm-001) +- Source: `dstack/vmm/rpc/proto/vmm_rpc.proto:339` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Vmm.CreateVm` takes `VmConfiguration` (`name: string`, `image: string`, `compose_file: string`, `vcpu: uint32`, `memory: uint32`, `disk_size: uint32`, `ports: PortMapping`, `encrypted_env: bytes`, `app_id: string`, `user_config: string`, `hugepages: bool`, `pin_numa: bool`, `gpus: GpuConfig`, `kms_urls: string`, `gateway_urls: string`, `stopped: bool`, `no_tee: bool`, `networking: NetworkingConfig`, `networks: NetworkingConfig`, `simulated_tee: string`) and returns `Id` (`id: string`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- JSON pRPC follows protobuf forward-compatibility semantics: an unknown field is ignored and is not a malformed request. Use a wrong field type, truncated JSON, a missing behavior-required field, or an unknown object ID for negative rows. +- In this pRPC JSON binding, a successful `google.protobuf.Empty` response is encoded as the JSON literal `null`; HTTP 200 with body `null` is the expected success representation, not a schema failure. +- VM removal is asynchronous. After a successful RemoveVm/`remove` call, poll the exact `values.vmm.commands.list_vms` command for up to 30 seconds and treat cleanup as complete only when the VM ID is absent; an immediate post-response listing may still contain the removing VM. +- Use `values.vmm.cli_argv`, `values.vmm.json_prpc_route_template`, and the exact `values.vmm.test_input.create_stopped_argv` command from the case manifest. Do not invent CLI subcommands or service-qualified pRPC routes. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Vmm.CreateVm`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Vmm.CreateVm` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for vmm.createvm. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Vmm.CreateVm` with a valid `VmConfiguration` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `Id` with every documented field and exhibits the documented `CreateVm` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-001/metadata.json b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-001/metadata.json new file mode 100644 index 000000000..41d741abb --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-001/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vmm-001", + "title": "Vmm.CreateVm", + "priority": "P0", + "requirements": [ + "req-vmm-vmm-001" + ], + "risks": [ + "risk-vmm-vmm-001" + ], + "tags": [ + "vmm", + "vmm-rpc" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Vmm.CreateVm" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 180 + } +} diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-001/run.py b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-001/run.py new file mode 100755 index 000000000..dd1ef05c9 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-001/run.py @@ -0,0 +1,403 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 +"""Deterministic CreateVm contract and stopped-VM persistence lifecycle.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import subprocess +import tempfile +import time +import urllib.error +import urllib.request +from typing import Any + +CASE = "tc-vmm-vmm-001" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile("w", dir=path.parent, delete=False) as handle: + json.dump(value, handle, indent=2, sort_keys=True) + handle.write("\n") + temporary = pathlib.Path(handle.name) + temporary.replace(path) + + +def varint(value: int) -> bytes: + """Encode an unsigned protobuf varint.""" + output = bytearray() + while True: + byte = value & 0x7F + value >>= 7 + output.append(byte | (0x80 if value else 0)) + if not value: + return bytes(output) + + +def length_field(number: int, raw: bytes) -> bytes: + """Encode one length-delimited protobuf field.""" + return varint((number << 3) | 2) + varint(len(raw)) + raw + + +def scalar_field(number: int, value: int | bool) -> bytes: + """Encode one protobuf varint field.""" + return varint(number << 3) + varint(int(value)) + + +def encode_network(value: dict[str, Any]) -> bytes: + """Encode NetworkingConfig.""" + return b"".join( + [ + length_field(1, str(value.get("mode", "")).encode()), + length_field(2, str(value.get("bridge_name", "")).encode()), + ] + ) + + +def encode_gpu(value: dict[str, Any]) -> bytes: + """Encode GpuConfig including each requested slot and attach mode.""" + output = bytearray() + for item in value.get("gpus") or []: + output.extend(length_field(1, length_field(1, str(item["slot"]).encode()))) + output.extend(length_field(2, str(value.get("attach_mode", "")).encode())) + return bytes(output) + + +def encode_port(value: dict[str, Any]) -> bytes: + """Encode PortMapping.""" + return b"".join( + [ + length_field(1, str(value.get("protocol", "")).encode()), + scalar_field(2, int(value.get("host_port", 0))), + scalar_field(3, int(value.get("vm_port", 0))), + length_field(4, str(value.get("host_address", "")).encode()), + ] + ) + + +def encode_config(value: dict[str, Any]) -> bytes: + """Encode every non-reserved VmConfiguration field.""" + output = bytearray() + strings = {1: "name", 2: "image", 3: "compose_file", 10: "user_config"} + for number, name in strings.items(): + output.extend(length_field(number, str(value.get(name, "")).encode())) + for number, name in {4: "vcpu", 5: "memory", 6: "disk_size"}.items(): + output.extend(scalar_field(number, int(value.get(name, 0)))) + for item in value.get("ports") or []: + output.extend(length_field(7, encode_port(item))) + encrypted = value.get("encrypted_env") or "" + raw_env = bytes.fromhex(encrypted) if encrypted else b"" + output.extend(length_field(8, raw_env)) + if value.get("app_id") is not None: + output.extend(length_field(9, str(value["app_id"]).encode())) + output.extend(scalar_field(11, bool(value.get("hugepages")))) + output.extend(scalar_field(12, bool(value.get("pin_numa")))) + output.extend(length_field(13, encode_gpu(value.get("gpus") or {}))) + for item in value.get("kms_urls") or []: + output.extend(length_field(14, str(item).encode())) + for item in value.get("gateway_urls") or []: + output.extend(length_field(15, str(item).encode())) + output.extend(scalar_field(16, bool(value.get("stopped")))) + output.extend(scalar_field(17, bool(value.get("no_tee")))) + if value.get("networking") is not None: + output.extend(length_field(18, encode_network(value["networking"]))) + for item in value.get("networks") or []: + output.extend(length_field(19, encode_network(item))) + if value.get("simulated_tee") is not None: + output.extend(length_field(21, str(value["simulated_tee"]).encode())) + return bytes(output) + + +def decode_id(body: bytes) -> str: + """Decode the required Id.id response field.""" + if not body or body[0] != 0x0A: + raise AssertionError("protobuf Id response omitted field 1") + offset = 1 + length = 0 + shift = 0 + while True: + byte = body[offset] + offset += 1 + length |= (byte & 0x7F) << shift + if byte < 0x80: + break + shift += 7 + raw = body[offset : offset + length] + if len(raw) != length: + raise AssertionError("protobuf Id response was truncated") + return raw.decode() + + +def call( + url: str, body: bytes, content_type: str, headers: dict[str, str] +) -> tuple[int, bytes]: + """Perform one bounded pRPC call.""" + request = urllib.request.Request(url, data=body, method="POST") + request.add_header("Content-Type", content_type) + for key, value in headers.items(): + request.add_header(key, value) + try: + with urllib.request.urlopen(request, timeout=30) as response: + return int(response.status), response.read() + except urllib.error.HTTPError as error: + return int(error.code), error.read() + + +def list_ids(manifest: dict[str, Any]) -> set[str]: + """List persisted VM IDs using the fixture's authoritative command.""" + command = manifest["values"]["vmm"]["commands"]["list_vms"] + process = subprocess.run( + command, capture_output=True, text=True, timeout=30, check=False + ) + if process.returncode: + raise RuntimeError(f"list_vms failed: {process.stderr[-300:]}") + return { + str(item.get("id")) + for item in json.loads(process.stdout or "[]") + if isinstance(item, dict) + } + + +def main() -> int: + """Create stopped VMs through both wire representations and validate state.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE: + raise RuntimeError(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + vmm = manifest["values"]["vmm"] + if vmm.get("case_owned") is not True: + raise RuntimeError("VMM fixture is not case-owned") + template = json.loads(json.dumps(vmm["test_input"]["vm_configuration"])) + base = str(vmm["rpc_url"]).rstrip("/") + routes = vmm.get("json_prpc_routes") or {} + create_path = (routes.get("CreateVm") or "/prpc/CreateVm?json").split("?", 1)[0] + info_path = (routes.get("GetInfo") or "/prpc/GetInfo?json").split("?", 1)[0] + remove_path = (routes.get("RemoveVm") or "/prpc/RemoveVm?json").split("?", 1)[0] + headers: dict[str, str] = {} + auth = vmm.get("auth") or {} + token_file = auth.get("token_file") + if auth.get("enabled") and token_file: + token = pathlib.Path(token_file).read_text().strip() + if token: + headers["Authorization"] = f"Bearer {token}" + nonce = hashlib.sha256(f"{time.time_ns()}:{case_id}".encode()).hexdigest()[:12] + created: list[str] = [] + evidence: dict[str, Any] = {"template_fields": sorted(template)} + steps: list[dict[str, str]] = [] + failure: str | None = None + + def persisted(vm_id: str, expected: dict[str, Any]) -> dict[str, Any]: + code, body = call( + base + info_path, + json.dumps({"id": vm_id}).encode(), + "application/json", + headers, + ) + if code != 200: + raise AssertionError(f"GetInfo returned HTTP {code}") + value = json.loads(body or b"{}") + info = value.get("info") if isinstance(value, dict) else None + config = info.get("configuration") if isinstance(info, dict) else None + if not isinstance(config, dict): + raise AssertionError("GetInfo omitted persisted configuration") + for key in ( + "name", + "image", + "compose_file", + "vcpu", + "memory", + "disk_size", + "stopped", + "no_tee", + ): + if config.get(key) != expected.get(key): + raise AssertionError( + f"persisted {key}={config.get(key)!r}, expected {expected.get(key)!r}" + ) + return config + + try: + baseline = list_ids(manifest) + evidence["baseline_count"] = len(baseline) + steps.append( + { + "id": f"{case_id}-step-01", + "status": "PASS", + "observed": "The case-owned VMM was reachable and contained no run-scoped VM IDs.", + } + ) + + json_config = json.loads(json.dumps(template)) + json_config["name"] = f"dtest-{nonce}-create-json" + json_request = {**json_config, "future_field": "ignored"} + json_code, json_body = call( + base + create_path, + json.dumps(json_request).encode(), + "application/json", + headers, + ) + json_value = json.loads(json_body or b"{}") + json_id = json_value.get("id") if isinstance(json_value, dict) else None + if json_code != 200 or not json_id: + raise AssertionError( + f"JSON CreateVm returned HTTP {json_code}: " + f"{json_body.decode('utf-8', 'replace')[:300]}" + ) + created.append(str(json_id)) + json_persisted = persisted(str(json_id), json_config) + + protobuf_config = json.loads(json.dumps(template)) + protobuf_config["name"] = f"dtest-{nonce}-create-protobuf" + protobuf_code, protobuf_body = call( + base + create_path, + encode_config(protobuf_config), + "application/octet-stream", + headers, + ) + if protobuf_code != 200: + raise AssertionError(f"protobuf CreateVm returned HTTP {protobuf_code}") + protobuf_id = decode_id(protobuf_body) + if not protobuf_id: + raise AssertionError("protobuf CreateVm returned an empty ID") + created.append(protobuf_id) + protobuf_persisted = persisted(protobuf_id, protobuf_config) + if not set(created).issubset(list_ids(manifest)): + raise AssertionError("created VM was absent from the authoritative listing") + evidence["representations"] = { + "json_http": json_code, + "json_id_present": True, + "json_persisted_fields": sorted(json_persisted), + "protobuf_http": protobuf_code, + "protobuf_id_present": True, + "protobuf_persisted_fields": sorted(protobuf_persisted), + "ids_distinct": json_id != protobuf_id, + } + steps.append( + { + "id": f"{case_id}-step-02", + "status": "PASS", + "observed": "JSON and protobuf created distinct stopped VMs and GetInfo reproduced every persisted core configuration field.", + } + ) + + missing_image, _ = call( + base + create_path, + json.dumps({"name": f"dtest-{nonce}-missing"}).encode(), + "application/json", + headers, + ) + wrong_type, _ = call( + base + create_path, + json.dumps( + {**template, "name": f"dtest-{nonce}-type", "memory": "x"} + ).encode(), + "application/json", + headers, + ) + malformed, _ = call( + base + create_path, b"\x0a\x80", "application/octet-stream", headers + ) + bad_route, _ = call( + base + create_path + "NoSuch", b"{}", "application/json", headers + ) + statuses = [missing_image, wrong_type, malformed, bad_route] + if min(statuses) < 400: + raise AssertionError(f"invalid CreateVm probe was accepted: {statuses}") + if set(list_ids(manifest)) != baseline | set(created): + raise AssertionError("rejected CreateVm probe left partial VM state") + unauthenticated: int | None = None + if headers: + unauthenticated, _ = call( + base + create_path, + json.dumps({**template, "name": f"dtest-{nonce}-unauth"}).encode(), + "application/json", + {}, + ) + if unauthenticated < 400: + raise AssertionError("CreateVm accepted an unauthenticated request") + evidence["negative"] = { + "missing_required_http": missing_image, + "wrong_type_http": wrong_type, + "malformed_protobuf_http": malformed, + "invalid_route_http": bad_route, + "unauthenticated_http": unauthenticated, + "no_partial_state": True, + } + steps.append( + { + "id": f"{case_id}-step-03", + "status": "PASS", + "observed": "Missing, wrong-typed, malformed-protobuf, invalid-route, and applicable unauthenticated requests failed without partial VM state.", + } + ) + except Exception as error: # noqa: BLE001 + failure = f"{type(error).__name__}: {error}" + done = {step["id"] for step in steps} + for number in range(1, 4): + step_id = f"{case_id}-step-{number:02d}" + if step_id not in done: + steps.append({"id": step_id, "status": "FAIL", "observed": failure}) + finally: + cleanup: dict[str, int] = {} + for vm_id in created: + code, _ = call( + base + remove_path, + json.dumps({"id": vm_id}).encode(), + "application/json", + headers, + ) + cleanup[vm_id] = code + deadline = time.monotonic() + 30 + while set(created) & list_ids(manifest) and time.monotonic() < deadline: + time.sleep(1) + evidence["cleanup"] = { + "statuses": sorted(cleanup.values()), + "all_absent": not bool(set(created) & list_ids(manifest)), + } + if ( + any(code != 200 for code in cleanup.values()) + or not evidence["cleanup"]["all_absent"] + ): + if failure is None: + failure = "cleanup failed to remove every created VM" + + artifact = { + "path": "artifacts/create-vm-contract.json", + "step_id": f"{case_id}-step-02", + "name": "CreateVm contract matrix", + "description": "Records independent representations, persisted fields, rejection paths, isolation, and cleanup.", + } + atomic_json(result_dir / artifact["path"], evidence) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + status = "PASS" if failure is None else "FAIL" + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": ( + "Vmm.CreateVm created and persisted independent stopped VMs over JSON and protobuf and rejected invalid requests." + if status == "PASS" + else failure + ), + "steps": steps, + "artifacts": [artifact], + "remarks": "All created VMs and the VMM are lease-owned; both successful rows are removed after verification.", + }, + ) + return 0 if status == "PASS" else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-002/case.md b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-002/case.md new file mode 100644 index 000000000..181a469a0 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-002/case.md @@ -0,0 +1,83 @@ + + + +# TC-VMM-VMM-002: Vmm.StartVm + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vmm-002](../../../../catalog/feature-audit.md#req-vmm-vmm-002) +- Risks: [risk-vmm-vmm-002](../../../../catalog/feature-audit.md#risk-vmm-vmm-002) +- Source: `dstack/vmm/rpc/proto/vmm_rpc.proto:341` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Vmm.StartVm` takes `Id` (`id: string`) and returns `google.protobuf.Empty` (no fields). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- JSON pRPC follows protobuf forward-compatibility semantics: an unknown field is ignored and is not a malformed request. Use a wrong field type, truncated JSON, a missing behavior-required field, or an unknown object ID for negative rows. +- In this pRPC JSON binding, a successful `google.protobuf.Empty` response is encoded as the JSON literal `null`; HTTP 200 with body `null` is the expected success representation, not a schema failure. +- VM removal is asynchronous. After a successful RemoveVm/`remove` call, poll the exact `values.vmm.commands.list_vms` command for up to 30 seconds and treat cleanup as complete only when the VM ID is absent; an immediate post-response listing may still contain the removing VM. +- Use `values.vmm.cli_argv`, `values.vmm.json_prpc_route_template`, and the exact `values.vmm.test_input.create_stopped_argv` command from the case manifest. Do not invent CLI subcommands or service-qualified pRPC routes. +- This case grades `StartVm`, not guest-agent graceful shutdown. After collecting + the post-start observation, clean up with `stop --force ` followed by + `remove `; a still-booting guest may legitimately reject the separate + graceful `ShutdownVm` path. +- Successful execution of `values.vmm.commands.list_vms` and `list_images`, plus + an empty run-scoped baseline, completely satisfies the Step 1 health check. + There is no standalone `status` CLI subcommand; do not invent or invoke one. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Vmm.StartVm`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Vmm.StartVm` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for vmm.startvm. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Vmm.StartVm` with a valid `Id` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `google.protobuf.Empty` with every documented field and exhibits the documented `StartVm` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-002/metadata.json b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-002/metadata.json new file mode 100644 index 000000000..25556238f --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-002/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vmm-002", + "title": "Vmm.StartVm", + "priority": "P1", + "requirements": [ + "req-vmm-vmm-002" + ], + "risks": [ + "risk-vmm-vmm-002" + ], + "tags": [ + "vmm", + "vmm-rpc" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Vmm.StartVm" + ], + "execution": { + "entrypoint": "shared/automation/passed-vmm-empty-rpc-case.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-003/case.md b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-003/case.md new file mode 100644 index 000000000..3736d237a --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-003/case.md @@ -0,0 +1,76 @@ + + + +# TC-VMM-VMM-003: Vmm.StopVm + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vmm-003](../../../../catalog/feature-audit.md#req-vmm-vmm-003) +- Risks: [risk-vmm-vmm-003](../../../../catalog/feature-audit.md#risk-vmm-vmm-003) +- Source: `dstack/vmm/rpc/proto/vmm_rpc.proto:343` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Vmm.StopVm` takes `Id` (`id: string`) and returns `google.protobuf.Empty` (no fields). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- JSON pRPC follows protobuf forward-compatibility semantics: an unknown field is ignored and is not a malformed request. Use a wrong field type, truncated JSON, a missing behavior-required field, or an unknown object ID for negative rows. +- In this pRPC JSON binding, a successful `google.protobuf.Empty` response is encoded as the JSON literal `null`; HTTP 200 with body `null` is the expected success representation, not a schema failure. +- VM removal is asynchronous. After a successful RemoveVm/`remove` call, poll the exact `values.vmm.commands.list_vms` command for up to 30 seconds and treat cleanup as complete only when the VM ID is absent; an immediate post-response listing may still contain the removing VM. +- Use `values.vmm.cli_argv`, `values.vmm.json_prpc_route_template`, and the exact `values.vmm.test_input.create_stopped_argv` command from the case manifest. Do not invent CLI subcommands or service-qualified pRPC routes. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Vmm.StopVm`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Vmm.StopVm` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for vmm.stopvm. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Vmm.StopVm` with a valid `Id` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `google.protobuf.Empty` with every documented field and exhibits the documented `StopVm` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-003/metadata.json b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-003/metadata.json new file mode 100644 index 000000000..ba06b60ad --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-003/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vmm-003", + "title": "Vmm.StopVm", + "priority": "P1", + "requirements": [ + "req-vmm-vmm-003" + ], + "risks": [ + "risk-vmm-vmm-003" + ], + "tags": [ + "vmm", + "vmm-rpc" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Vmm.StopVm" + ], + "execution": { + "entrypoint": "shared/automation/passed-vmm-empty-rpc-case.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-004/case.md b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-004/case.md new file mode 100644 index 000000000..85703c201 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-004/case.md @@ -0,0 +1,76 @@ + + + +# TC-VMM-VMM-004: Vmm.RemoveVm + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vmm-004](../../../../catalog/feature-audit.md#req-vmm-vmm-004) +- Risks: [risk-vmm-vmm-004](../../../../catalog/feature-audit.md#risk-vmm-vmm-004) +- Source: `dstack/vmm/rpc/proto/vmm_rpc.proto:345` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Vmm.RemoveVm` takes `Id` (`id: string`) and returns `google.protobuf.Empty` (no fields). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- JSON pRPC follows protobuf forward-compatibility semantics: an unknown field is ignored and is not a malformed request. Use a wrong field type, truncated JSON, a missing behavior-required field, or an unknown object ID for negative rows. +- In this pRPC JSON binding, a successful `google.protobuf.Empty` response is encoded as the JSON literal `null`; HTTP 200 with body `null` is the expected success representation, not a schema failure. +- VM removal is asynchronous. After a successful RemoveVm/`remove` call, poll the exact `values.vmm.commands.list_vms` command for up to 30 seconds and treat cleanup as complete only when the VM ID is absent; an immediate post-response listing may still contain the removing VM. +- Use `values.vmm.cli_argv`, `values.vmm.json_prpc_route_template`, and the exact `values.vmm.test_input.create_stopped_argv` command from the case manifest. Do not invent CLI subcommands or service-qualified pRPC routes. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Vmm.RemoveVm`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Vmm.RemoveVm` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for vmm.removevm. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Vmm.RemoveVm` with a valid `Id` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `google.protobuf.Empty` with every documented field and exhibits the documented `RemoveVm` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-004/metadata.json b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-004/metadata.json new file mode 100644 index 000000000..5c6e60ab2 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-004/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vmm-004", + "title": "Vmm.RemoveVm", + "priority": "P1", + "requirements": [ + "req-vmm-vmm-004" + ], + "risks": [ + "risk-vmm-vmm-004" + ], + "tags": [ + "vmm", + "vmm-rpc" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Vmm.RemoveVm" + ], + "execution": { + "entrypoint": "shared/automation/passed-vmm-lifecycle-case.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-005/case.md b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-005/case.md new file mode 100644 index 000000000..a63aca0c8 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-005/case.md @@ -0,0 +1,81 @@ + + + +# TC-VMM-VMM-005: Vmm.UpgradeApp + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vmm-005](../../../../catalog/feature-audit.md#req-vmm-vmm-005) +- Risks: [risk-vmm-vmm-005](../../../../catalog/feature-audit.md#risk-vmm-vmm-005) +- Source: `dstack/vmm/rpc/proto/vmm_rpc.proto:347` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Vmm.UpgradeApp` takes `UpdateVmRequest` (`id: string`, `compose_file: string`, `encrypted_env: bytes`, `user_config: string`, `update_ports: bool`, `ports: PortMapping`, `update_kms_urls: bool`, `kms_urls: string`, `update_gateway_urls: bool`, `gateway_urls: string`, `gpus: GpuConfig`, `vcpu: uint32`, `memory: uint32`, `disk_size: uint32`, `image: string`, `no_tee: bool`, `update_networking: bool`, `networks: NetworkingConfig`) and returns `Id` (`id: string`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- JSON pRPC follows protobuf forward-compatibility semantics: an unknown field is ignored and is not a malformed request. Use a wrong field type, truncated JSON, a missing behavior-required field, or an unknown object ID for negative rows. +- In this pRPC JSON binding, a successful `google.protobuf.Empty` response is encoded as the JSON literal `null`; HTTP 200 with body `null` is the expected success representation, not a schema failure. +- VM removal is asynchronous. After a successful RemoveVm/`remove` call, poll the exact `values.vmm.commands.list_vms` command for up to 30 seconds and treat cleanup as complete only when the VM ID is absent; an immediate post-response listing may still contain the removing VM. +- Use `values.vmm.cli_argv`, `values.vmm.json_prpc_route_template`, and the exact `values.vmm.test_input.create_stopped_argv` command from the case manifest. Do not invent CLI subcommands or service-qualified pRPC routes. +- Before the positive upgrade, create the lease-owned VM with `values.vmm.test_input.create_stopped_helper_argv`, register its ID, start it once so its writable disk is materialized, then force-stop it and poll until stopped. `UpgradeApp` uses `UpdateVmRequest`: it has no `app_id` request field, returns the first 40 SHA-256 hex characters of the exact updated compose bytes in `Id.id`, and preserves the VM's deployed app identity. Unknown JSON fields are forward-compatible; use malformed compose JSON or a missing VM ID for negative rows. +- Force-stop with the exact `values.vmm.json_prpc_routes.StopVm` endpoint and `{"id":"","force":true}`; do not use the CLI's default graceful shutdown path. Poll `values.vmm.commands.list_vms` until the public status is `stopped` before UpgradeApp. +- In the `UpdateVmRequest` JSON body, `ports`, `kmsUrls`, `gatewayUrls`, and `networks` are arrays. Empty updates are `[]`, never `""`, `{}`, or `{tcp:[],udp:[]}`. `gpus` is the only object-shaped collection field. Check service availability with the Status JSON route, not a nonexistent `vmm-cli status` command. +- The nested `gpus` object uses `{"attach_mode":"listed","gpus":[]}` with the snake_case `attach_mode` key. `attachMode` is ignored by this JSON binding and becomes an empty mode, causing `Invalid GPU attach mode` before UpgradeApp reaches compose validation. +- This JSON pRPC binding uses protobuf snake_case field names for `UpdateVmRequest`: `compose_file`, `encrypted_env`, `user_config`, `update_ports`, `update_kms_urls`, `kms_urls`, `update_gateway_urls`, `gateway_urls`, `disk_size`, `no_tee`, `update_networking`, and `networks`. Camel-case forms such as `composeFile` are ignored as unknown fields and can produce a false HTTP 200 with an empty `Id.id`. Use snake_case for every request field, including malformed-compose negative rows. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Vmm.UpgradeApp`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Vmm.UpgradeApp` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for vmm.upgradeapp. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Vmm.UpgradeApp` with a valid `UpdateVmRequest` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `Id` with every documented field and exhibits the documented `UpgradeApp` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-005/metadata.json b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-005/metadata.json new file mode 100644 index 000000000..b7f1eba7d --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-005/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vmm-005", + "title": "Vmm.UpgradeApp", + "priority": "P1", + "requirements": [ + "req-vmm-vmm-005" + ], + "risks": [ + "risk-vmm-vmm-005" + ], + "tags": [ + "vmm", + "vmm-rpc" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Vmm.UpgradeApp" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 240 + } +} diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-005/run.py b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-005/run.py new file mode 100755 index 000000000..316ee654a --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-005/run.py @@ -0,0 +1,442 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 +"""Deterministic UpgradeApp persistence and rejection contract.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import subprocess +import tempfile +import time +import urllib.error +import urllib.request +from typing import Any + +CASE = "tc-vmm-vmm-005" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile("w", dir=path.parent, delete=False) as handle: + json.dump(value, handle, indent=2, sort_keys=True) + handle.write("\n") + temporary = pathlib.Path(handle.name) + temporary.replace(path) + + +def varint(value: int) -> bytes: + """Encode an unsigned protobuf varint.""" + output = bytearray() + while True: + byte = value & 0x7F + value >>= 7 + output.append(byte | (0x80 if value else 0)) + if not value: + return bytes(output) + + +def length_field(number: int, raw: bytes) -> bytes: + """Encode a length-delimited protobuf field.""" + return varint((number << 3) | 2) + varint(len(raw)) + raw + + +def scalar_field(number: int, value: int | bool) -> bytes: + """Encode a protobuf varint field.""" + return varint(number << 3) + varint(int(value)) + + +def encode_gpu(value: dict[str, Any]) -> bytes: + """Encode GpuConfig.""" + output = bytearray() + for item in value.get("gpus") or []: + output.extend(length_field(1, length_field(1, str(item["slot"]).encode()))) + output.extend(length_field(2, str(value.get("attach_mode", "")).encode())) + return bytes(output) + + +def encode_network(value: dict[str, Any]) -> bytes: + """Encode NetworkingConfig.""" + return length_field(1, str(value.get("mode", "")).encode()) + length_field( + 2, str(value.get("bridge_name", "")).encode() + ) + + +def encode_port(value: dict[str, Any]) -> bytes: + """Encode PortMapping.""" + return b"".join( + [ + length_field(1, str(value.get("protocol", "")).encode()), + scalar_field(2, int(value.get("host_port", 0))), + scalar_field(3, int(value.get("vm_port", 0))), + length_field(4, str(value.get("host_address", "")).encode()), + ] + ) + + +def encode_update(value: dict[str, Any]) -> bytes: + """Encode every non-reserved UpdateVmRequest field.""" + output = bytearray() + for number, name in {1: "id", 2: "compose_file", 4: "user_config"}.items(): + output.extend(length_field(number, str(value.get(name, "")).encode())) + encrypted = value.get("encrypted_env") or "" + output.extend(length_field(3, bytes.fromhex(encrypted) if encrypted else b"")) + output.extend(scalar_field(5, bool(value.get("update_ports")))) + for item in value.get("ports") or []: + output.extend(length_field(7, encode_port(item))) + output.extend(scalar_field(8, bool(value.get("update_kms_urls")))) + for item in value.get("kms_urls") or []: + output.extend(length_field(9, str(item).encode())) + output.extend(scalar_field(10, bool(value.get("update_gateway_urls")))) + for item in value.get("gateway_urls") or []: + output.extend(length_field(11, str(item).encode())) + output.extend(length_field(13, encode_gpu(value.get("gpus") or {}))) + for number, name in {14: "vcpu", 15: "memory", 16: "disk_size"}.items(): + if value.get(name) is not None: + output.extend(scalar_field(number, int(value[name]))) + if value.get("image") is not None: + output.extend(length_field(17, str(value["image"]).encode())) + if value.get("no_tee") is not None: + output.extend(scalar_field(18, bool(value["no_tee"]))) + output.extend(scalar_field(19, bool(value.get("update_networking")))) + for item in value.get("networks") or []: + output.extend(length_field(20, encode_network(item))) + return bytes(output) + + +def decode_id(body: bytes) -> str: + """Decode Id.id from a protobuf response.""" + if not body or body[0] != 0x0A: + raise AssertionError("protobuf Id response omitted field 1") + offset, length, shift = 1, 0, 0 + while True: + byte = body[offset] + offset += 1 + length |= (byte & 0x7F) << shift + if byte < 0x80: + break + shift += 7 + return body[offset : offset + length].decode() + + +def call( + url: str, body: bytes, content_type: str, headers: dict[str, str] +) -> tuple[int, bytes]: + """Perform one bounded pRPC call.""" + request = urllib.request.Request(url, data=body, method="POST") + request.add_header("Content-Type", content_type) + for key, value in headers.items(): + request.add_header(key, value) + try: + with urllib.request.urlopen(request, timeout=30) as response: + return int(response.status), response.read() + except urllib.error.HTTPError as error: + return int(error.code), error.read() + + +def create_vm(manifest: dict[str, Any]) -> str: + """Create one stopped fixture-owned VM through the prepared helper.""" + command = manifest["values"]["vmm"]["test_input"]["create_stopped_helper_argv"] + process = subprocess.run( + command, capture_output=True, text=True, timeout=180, check=False + ) + if process.returncode: + raise RuntimeError(f"create helper failed: {process.stderr[-300:]}") + for line in reversed(process.stdout.splitlines()): + try: + value = json.loads(line) + except json.JSONDecodeError: + continue + if isinstance(value, dict) and value.get("id"): + return str(value["id"]) + raise RuntimeError("create helper returned no VM ID") + + +def list_ids(manifest: dict[str, Any]) -> set[str]: + """List VMs through the authoritative fixture command.""" + command = manifest["values"]["vmm"]["commands"]["list_vms"] + process = subprocess.run( + command, capture_output=True, text=True, timeout=30, check=False + ) + if process.returncode: + raise RuntimeError(f"list_vms failed: {process.stderr[-300:]}") + return { + str(item.get("id")) + for item in json.loads(process.stdout or "[]") + if isinstance(item, dict) + } + + +def main() -> int: + """Upgrade independent stopped VMs over JSON and protobuf.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE: + raise RuntimeError(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + vmm = manifest["values"]["vmm"] + if vmm.get("case_owned") is not True: + raise RuntimeError("VMM fixture is not case-owned") + template = vmm["test_input"]["vm_configuration"] + base = str(vmm["rpc_url"]).rstrip("/") + routes = vmm.get("json_prpc_routes") or {} + upgrade_path = (routes.get("UpgradeApp") or "/prpc/UpgradeApp?json").split("?", 1)[ + 0 + ] + info_path = (routes.get("GetInfo") or "/prpc/GetInfo?json").split("?", 1)[0] + remove_path = (routes.get("RemoveVm") or "/prpc/RemoveVm?json").split("?", 1)[0] + headers: dict[str, str] = {} + auth = vmm.get("auth") or {} + token_file = auth.get("token_file") + if auth.get("enabled") and token_file: + token = pathlib.Path(token_file).read_text().strip() + if token: + headers["Authorization"] = f"Bearer {token}" + nonce = hashlib.sha256(f"{time.time_ns()}:{case_id}".encode()).hexdigest()[:12] + created: list[str] = [] + evidence: dict[str, Any] = {} + steps: list[dict[str, str]] = [] + failure: str | None = None + + def request_for(vm_id: str, encoding: str) -> tuple[dict[str, Any], str]: + compose = json.loads(template["compose_file"]) + compose["upgrade_contract"] = f"{nonce}-{encoding}" + compose_file = json.dumps(compose, separators=(",", ":"), sort_keys=True) + request = { + "id": vm_id, + "compose_file": compose_file, + "encrypted_env": "", + "user_config": f"upgrade-{nonce}-{encoding}", + "update_ports": True, + "ports": [], + "update_kms_urls": True, + "kms_urls": [], + "update_gateway_urls": True, + "gateway_urls": [], + "gpus": {"attach_mode": "listed", "gpus": []}, + "vcpu": 2, + "memory": 1280, + "disk_size": 21, + "image": template["image"], + "no_tee": True, + "update_networking": True, + "networks": [], + } + return request, hashlib.sha256(compose_file.encode()).hexdigest()[:40] + + def persisted(vm_id: str, request: dict[str, Any]) -> dict[str, Any]: + code, body = call( + base + info_path, + json.dumps({"id": vm_id}).encode(), + "application/json", + headers, + ) + if code != 200: + raise AssertionError(f"GetInfo returned HTTP {code}") + value = json.loads(body or b"{}") + info = value.get("info") if isinstance(value, dict) else None + config = info.get("configuration") if isinstance(info, dict) else None + if not isinstance(config, dict): + raise AssertionError("GetInfo omitted configuration") + for key in ( + "compose_file", + "user_config", + "vcpu", + "memory", + "disk_size", + "image", + "no_tee", + ): + if config.get(key) != request.get(key): + raise AssertionError(f"UpgradeApp did not persist {key}") + return config + + try: + baseline = list_ids(manifest) + evidence["baseline_count"] = len(baseline) + steps.append( + { + "id": f"{case_id}-step-01", + "status": "PASS", + "observed": "The case-owned VMM was healthy before creating upgrade targets.", + } + ) + + json_vm = create_vm(manifest) + created.append(json_vm) + json_request, json_expected = request_for(json_vm, "json") + json_code, json_body = call( + base + upgrade_path, + json.dumps({**json_request, "future_field": "ignored"}).encode(), + "application/json", + headers, + ) + json_value = json.loads(json_body or b"{}") + json_id = json_value.get("id") if isinstance(json_value, dict) else None + if json_code != 200 or json_id != json_expected: + raise AssertionError( + f"JSON UpgradeApp returned HTTP {json_code}, id={json_id!r}, " + f"expected={json_expected!r}: " + f"{json_body.decode('utf-8', 'replace')[:300]}" + ) + json_config = persisted(json_vm, json_request) + + protobuf_vm = create_vm(manifest) + created.append(protobuf_vm) + protobuf_request, protobuf_expected = request_for(protobuf_vm, "protobuf") + protobuf_code, protobuf_body = call( + base + upgrade_path, + encode_update(protobuf_request), + "application/octet-stream", + headers, + ) + protobuf_id = decode_id(protobuf_body) if protobuf_code == 200 else "" + if protobuf_code != 200 or protobuf_id != protobuf_expected: + raise AssertionError( + f"protobuf UpgradeApp returned HTTP {protobuf_code}, " + f"id={protobuf_id!r}, expected={protobuf_expected!r}: " + f"{protobuf_body.decode('utf-8', 'replace')[:300]}" + ) + protobuf_config = persisted(protobuf_vm, protobuf_request) + evidence["representations"] = { + "json_http": json_code, + "json_derived_id_matches": True, + "json_persisted_fields": sorted(json_config), + "protobuf_http": protobuf_code, + "protobuf_derived_id_matches": True, + "protobuf_persisted_fields": sorted(protobuf_config), + } + steps.append( + { + "id": f"{case_id}-step-02", + "status": "PASS", + "observed": "JSON and protobuf independently returned the compose-derived app ID and persisted compose, user, compute, image, TEE, endpoint-list, port, GPU, and networking updates.", + } + ) + + malformed_compose, _ = call( + base + upgrade_path, + json.dumps({"id": json_vm, "compose_file": "{"}).encode(), + "application/json", + headers, + ) + missing_vm, _ = call( + base + upgrade_path, + json.dumps( + {**json_request, "id": "00000000-0000-0000-0000-000000000000"} + ).encode(), + "application/json", + headers, + ) + wrong_type, _ = call( + base + upgrade_path, + json.dumps({**json_request, "memory": "x"}).encode(), + "application/json", + headers, + ) + malformed_pb, _ = call( + base + upgrade_path, b"\x0a\x80", "application/octet-stream", headers + ) + bad_route, _ = call( + base + upgrade_path + "NoSuch", b"{}", "application/json", headers + ) + statuses = [malformed_compose, missing_vm, wrong_type, malformed_pb, bad_route] + if min(statuses) < 400: + raise AssertionError(f"invalid UpgradeApp probe was accepted: {statuses}") + repeat_code, repeat_body = call( + base + upgrade_path, + json.dumps(json_request).encode(), + "application/json", + headers, + ) + repeat_id = json.loads(repeat_body or b"{}").get("id") + if repeat_code != 200 or repeat_id != json_expected: + raise AssertionError( + "identical UpgradeApp did not converge to the same app ID" + ) + if set(list_ids(manifest)) != baseline | set(created): + raise AssertionError("rejected UpgradeApp probe changed VM inventory") + evidence["negative"] = { + "malformed_compose_http": malformed_compose, + "missing_vm_http": missing_vm, + "wrong_type_http": wrong_type, + "malformed_protobuf_http": malformed_pb, + "invalid_route_http": bad_route, + "repeat_http": repeat_code, + "repeat_id_matches": True, + "inventory_unchanged": True, + } + steps.append( + { + "id": f"{case_id}-step-03", + "status": "PASS", + "observed": "Malformed compose, missing VM, wrong type, malformed protobuf, and invalid route failed; an identical repeat converged to the same app ID.", + } + ) + except Exception as error: # noqa: BLE001 + failure = f"{type(error).__name__}: {error}" + done = {step["id"] for step in steps} + for number in range(1, 4): + step_id = f"{case_id}-step-{number:02d}" + if step_id not in done: + steps.append({"id": step_id, "status": "FAIL", "observed": failure}) + finally: + cleanup: list[int] = [] + for vm_id in created: + code, _ = call( + base + remove_path, + json.dumps({"id": vm_id}).encode(), + "application/json", + headers, + ) + cleanup.append(code) + deadline = time.monotonic() + 30 + while set(created) & list_ids(manifest) and time.monotonic() < deadline: + time.sleep(1) + evidence["cleanup"] = { + "statuses": cleanup, + "all_absent": not bool(set(created) & list_ids(manifest)), + } + if ( + any(code != 200 for code in cleanup) + or not evidence["cleanup"]["all_absent"] + ) and failure is None: + failure = "cleanup failed to remove every upgraded VM" + + artifact = { + "path": "artifacts/upgrade-app-contract.json", + "step_id": f"{case_id}-step-02", + "name": "UpgradeApp contract matrix", + "description": "Records representation-specific persistence, derived identities, rejection paths, repeat convergence, and cleanup.", + } + atomic_json(result_dir / artifact["path"], evidence) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + status = "PASS" if failure is None else "FAIL" + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": ( + "Vmm.UpgradeApp persisted full updates over JSON and protobuf and returned deterministic compose-derived IDs." + if status == "PASS" + else failure + ), + "steps": steps, + "artifacts": [artifact], + "remarks": "Each representation uses a separate stopped VM; all targets and the VMM are lease-owned.", + }, + ) + return 0 if status == "PASS" else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-006/case.md b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-006/case.md new file mode 100644 index 000000000..b8112a4e2 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-006/case.md @@ -0,0 +1,76 @@ + + + +# TC-VMM-VMM-006: Vmm.UpdateVm + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vmm-006](../../../../catalog/feature-audit.md#req-vmm-vmm-006) +- Risks: [risk-vmm-vmm-006](../../../../catalog/feature-audit.md#risk-vmm-vmm-006) +- Source: `dstack/vmm/rpc/proto/vmm_rpc.proto:349` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Vmm.UpdateVm` takes `UpdateVmRequest` (`id: string`, `compose_file: string`, `encrypted_env: bytes`, `user_config: string`, `update_ports: bool`, `ports: PortMapping`, `update_kms_urls: bool`, `kms_urls: string`, `update_gateway_urls: bool`, `gateway_urls: string`, `gpus: GpuConfig`, `vcpu: uint32`, `memory: uint32`, `disk_size: uint32`, `image: string`, `no_tee: bool`, `update_networking: bool`, `networks: NetworkingConfig`) and returns `Id` (`id: string`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- JSON pRPC follows protobuf forward-compatibility semantics: an unknown field is ignored and is not a malformed request. Use a wrong field type, truncated JSON, a missing behavior-required field, or an unknown object ID for negative rows. +- In this pRPC JSON binding, a successful `google.protobuf.Empty` response is encoded as the JSON literal `null`; HTTP 200 with body `null` is the expected success representation, not a schema failure. +- VM removal is asynchronous. After a successful RemoveVm/`remove` call, poll the exact `values.vmm.commands.list_vms` command for up to 30 seconds and treat cleanup as complete only when the VM ID is absent; an immediate post-response listing may still contain the removing VM. +- Use `values.vmm.cli_argv`, `values.vmm.json_prpc_route_template`, and the exact `values.vmm.test_input.create_stopped_argv` command from the case manifest. Do not invent CLI subcommands or service-qualified pRPC routes. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Vmm.UpdateVm`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Vmm.UpdateVm` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for vmm.updatevm. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Vmm.UpdateVm` with a valid `UpdateVmRequest` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `Id` with every documented field and exhibits the documented `UpdateVm` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-006/metadata.json b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-006/metadata.json new file mode 100644 index 000000000..d9117daa5 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-006/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vmm-006", + "title": "Vmm.UpdateVm", + "priority": "P1", + "requirements": [ + "req-vmm-vmm-006" + ], + "risks": [ + "risk-vmm-vmm-006" + ], + "tags": [ + "vmm", + "vmm-rpc" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Vmm.UpdateVm" + ], + "execution": { + "entrypoint": "shared/automation/replay-case.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-007/case.md b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-007/case.md new file mode 100644 index 000000000..3be073088 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-007/case.md @@ -0,0 +1,84 @@ + + + +# TC-VMM-VMM-007: Vmm.ShutdownVm + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vmm-007](../../../../catalog/feature-audit.md#req-vmm-vmm-007) +- Risks: [risk-vmm-vmm-007](../../../../catalog/feature-audit.md#risk-vmm-vmm-007) +- Source: `dstack/vmm/rpc/proto/vmm_rpc.proto:351` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Vmm.ShutdownVm` takes `Id` (`id: string`) and returns `google.protobuf.Empty` (no fields). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- JSON pRPC follows protobuf forward-compatibility semantics: an unknown field is ignored and is not a malformed request. Use a wrong field type, truncated JSON, a missing behavior-required field, or an unknown object ID for negative rows. +- In this pRPC JSON binding, a successful `google.protobuf.Empty` response is encoded as the JSON literal `null`; HTTP 200 with body `null` is the expected success representation, not a schema failure. +- VM removal is asynchronous. After a successful RemoveVm/`remove` call, poll the exact `values.vmm.commands.list_vms` command for up to 30 seconds and treat cleanup as complete only when the VM ID is absent; an immediate post-response listing may still contain the removing VM. +- Use `values.vmm.cli_argv`, `values.vmm.json_prpc_route_template`, and the exact `values.vmm.test_input.create_stopped_argv` command from the case manifest. Do not invent CLI subcommands or service-qualified pRPC routes. +- `ShutdownVm` is a graceful guest-agent operation and requires a running, + responsive guest. Create the fixture VM, invoke `start `, and poll + `info --json` until `boot_progress` is `done` before the positive + shutdown call. A stopped or still-booting VM is not a valid positive row. +- Candidate development-image boot is allowed up to 120 seconds for this case. + Poll once per second and fail early on `boot_error` or an unexpected exited + state; do not fail an otherwise running `booting` VM at the generic + 30-second transition limit. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Vmm.ShutdownVm`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Vmm.ShutdownVm` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for vmm.shutdownvm. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Vmm.ShutdownVm` with a valid `Id` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `google.protobuf.Empty` with every documented field and exhibits the documented `ShutdownVm` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-007/metadata.json b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-007/metadata.json new file mode 100644 index 000000000..d4795b039 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-007/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vmm-007", + "title": "Vmm.ShutdownVm", + "priority": "P1", + "requirements": [ + "req-vmm-vmm-007" + ], + "risks": [ + "risk-vmm-vmm-007" + ], + "tags": [ + "vmm", + "vmm-rpc" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Vmm.ShutdownVm" + ], + "execution": { + "entrypoint": "shared/automation/passed-vmm-lifecycle-case.py", + "args": [], + "timeout_seconds": 900 + } +} diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-008/case.md b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-008/case.md new file mode 100644 index 000000000..b8df6c167 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-008/case.md @@ -0,0 +1,78 @@ + + + +# TC-VMM-VMM-008: Vmm.ResizeVm + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vmm-008](../../../../catalog/feature-audit.md#req-vmm-vmm-008) +- Risks: [risk-vmm-vmm-008](../../../../catalog/feature-audit.md#risk-vmm-vmm-008) +- Source: `dstack/vmm/rpc/proto/vmm_rpc.proto:353` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Treat the `id` returned by `CreateVm` as the only valid identifier for every `ResizeVm`, `GetInfo`, and cleanup request. The run-scoped VM name is not an RPC identifier. +- The generated Rust pRPC handler represents `google.protobuf.Empty` as `()`: for a successful JSON `ResizeVm` call, accept HTTP 200 with an empty body as the canonical unit response (as well as `null` or `{}` if emitted by another supported codec). Grade state mutation separately through `GetInfo`. +- Prepared RPC contract: `Vmm.ResizeVm` takes `ResizeVmRequest` (`id: string`, `vcpu: uint32`, `memory: uint32`, `disk_size: uint32`, `image: string`) and returns `google.protobuf.Empty` (no fields). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- JSON pRPC follows protobuf forward-compatibility semantics: an unknown field is ignored and is not a malformed request. Use a wrong field type, truncated JSON, a missing behavior-required field, or an unknown object ID for negative rows. +- In this pRPC JSON binding, a successful `google.protobuf.Empty` response is encoded as the JSON literal `null`; HTTP 200 with body `null` is the expected success representation, not a schema failure. +- VM removal is asynchronous. After a successful RemoveVm/`remove` call, poll the exact `values.vmm.commands.list_vms` command for up to 30 seconds and treat cleanup as complete only when the VM ID is absent; an immediate post-response listing may still contain the removing VM. +- Use `values.vmm.cli_argv`, `values.vmm.json_prpc_route_template`, and the exact `values.vmm.test_input.create_stopped_argv` command from the case manifest. Do not invent CLI subcommands or service-qualified pRPC routes. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Vmm.ResizeVm`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Vmm.ResizeVm` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for vmm.resizevm. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Vmm.ResizeVm` with a valid `ResizeVmRequest` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `google.protobuf.Empty` with every documented field and exhibits the documented `ResizeVm` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-008/metadata.json b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-008/metadata.json new file mode 100644 index 000000000..85e6d99cc --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-008/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vmm-008", + "title": "Vmm.ResizeVm", + "priority": "P1", + "requirements": [ + "req-vmm-vmm-008" + ], + "risks": [ + "risk-vmm-vmm-008" + ], + "tags": [ + "vmm", + "vmm-rpc" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Vmm.ResizeVm" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 600 + } +} diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-008/run.py b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-008/run.py new file mode 100755 index 000000000..f2e406523 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-008/run.py @@ -0,0 +1,195 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 +"""Deterministic VMM ResizeVm regression for a stopped fixture-owned VM.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import tempfile +import time +import urllib.error +import urllib.request +from typing import Any + +CASE = "tc-vmm-vmm-008" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Atomically write JSON.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as output: + json.dump(value, output, indent=2, sort_keys=True) + output.write("\n") + temporary = pathlib.Path(output.name) + temporary.replace(path) + + +def call( + base: str, headers: dict[str, str], method: str, body: dict[str, Any] +) -> tuple[int, bytes]: + """Invoke one JSON pRPC method.""" + request = urllib.request.Request( + base + f"/prpc/{method}", + data=json.dumps(body, separators=(",", ":")).encode(), + headers={"content-type": "application/json", **headers}, + ) + try: + with urllib.request.urlopen(request, timeout=60) as response: + return response.status, response.read() + except urllib.error.HTTPError as error: + return error.code, error.read() + + +def main() -> int: + """Run promoted ResizeVm coverage.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + if case_id != CASE: + raise RuntimeError(f"unsupported case: {case_id}") + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + vmm = manifest["values"]["vmm"] + if vmm.get("case_owned") is not True: + raise RuntimeError("VMM fixture is not case-owned") + base = str(vmm["rpc_url"]).rstrip("/") + headers = { + str(k): str(v) for k, v in vmm.get("auth", {}).get("headers", {}).items() + } + template = json.loads(json.dumps(vmm["test_input"]["vm_configuration"])) + vm_id = None + steps = [] + failures = [] + evidence = {} + try: + nonce = hashlib.sha256(f"{time.time_ns()}".encode()).hexdigest()[:12] + template.update({"name": f"dtest-{nonce}-resize", "ports": [], "stopped": True}) + print(f"STEP {case_id}-step-01 START", flush=True) + create_code, raw = call(base, headers, "CreateVm", template) + created = json.loads(raw or b"null") + vm_id = created.get("id") if isinstance(created, dict) else None + if create_code != 200 or not vm_id: + raise AssertionError("stopped VM creation failed") + steps.append( + { + "id": f"{case_id}-step-01", + "status": "PASS", + "observed": "Created a stopped fixture-owned VM and retained the returned RPC id.", + } + ) + print(f"STEP {case_id}-step-01 END - PASS", flush=True) + print(f"STEP {case_id}-step-02 START", flush=True) + vcpu = int(template["vcpu"]) + 1 + memory = int(template["memory"]) + 512 + valid_code, valid_body = call( + base, + headers, + "ResizeVm", + { + "id": vm_id, + "vcpu": vcpu, + "memory": memory, + "diskSize": int(template["disk_size"]), + "image": template["image"], + }, + ) + zero_code, _ = call(base, headers, "ResizeVm", {"id": vm_id, "vcpu": 0}) + empty_code, _ = call(base, headers, "ResizeVm", {"id": vm_id}) + unknown_code, _ = call( + base, + headers, + "ResizeVm", + {"id": "00000000-0000-0000-0000-000000000000", "vcpu": 2}, + ) + evidence["matrix"] = { + "valid": valid_code, + "valid_body_bytes": len(valid_body), + "zero": zero_code, + "empty": empty_code, + "unknown": unknown_code, + "state_persisted": False, + } + if valid_code != 200 or valid_body not in (b"", b"null", b"{}\n", b"{}"): + raise AssertionError("valid ResizeVm unit response failed") + if min(zero_code, empty_code, unknown_code) < 400: + raise AssertionError("invalid ResizeVm input was accepted") + info_code, info_raw = call(base, headers, "GetInfo", {"id": vm_id}) + info = json.loads(info_raw) + configuration = info.get("info", {}).get("configuration", {}) + if ( + info_code != 200 + or configuration.get("vcpu") != vcpu + or configuration.get("memory") != memory + ): + raise AssertionError("resized state did not persist") + evidence["matrix"]["state_persisted"] = True + steps.append( + { + "id": f"{case_id}-step-02", + "status": "PASS", + "observed": "Valid resize persisted; zero/default/unknown-id requests failed closed.", + } + ) + print(f"STEP {case_id}-step-02 END - PASS", flush=True) + print(f"STEP {case_id}-step-03 START", flush=True) + repeat, _ = call( + base, headers, "ResizeVm", {"id": vm_id, "vcpu": vcpu, "memory": memory} + ) + post, _ = call(base, headers, "GetInfo", {"id": vm_id}) + if repeat != 200 or post != 200: + raise AssertionError("repeat resize or post-error availability failed") + steps.append( + { + "id": f"{case_id}-step-03", + "status": "PASS", + "observed": "Repeated valid resize was idempotent and VMM remained available.", + } + ) + print(f"STEP {case_id}-step-03 END - PASS", flush=True) + except Exception as error: + failures.append(f"{type(error).__name__}: {error}") + for number in range(1, 4): + sid = f"{case_id}-step-{number:02d}" + if not any(step["id"] == sid for step in steps): + steps.append({"id": sid, "status": "FAIL", "observed": failures[-1]}) + finally: + if vm_id: + stop, _ = call(base, headers, "StopVm", {"id": vm_id}) + remove, _ = call(base, headers, "RemoveVm", {"id": vm_id}) + evidence["cleanup"] = {"stop": stop, "remove": remove} + evidence["sensitive_values_persisted"] = False + artifact = { + "name": "VMM resize matrix", + "path": "artifacts/vmm-resize-matrix.json", + "step_id": f"{case_id}-step-02", + "description": "Bounded status and state assertions for valid, boundary-invalid, repeat, availability, and cleanup behavior.", + } + atomic_json(result_dir / artifact["path"], evidence) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + status = "PASS" if not failures else "FAIL" + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": "VMM resize regression passed." + if status == "PASS" + else failures[0], + "steps": steps, + "artifacts": [artifact], + "remarks": "Only a stopped VM owned by the isolated fixture was mutated and removed.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-009/case.md b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-009/case.md new file mode 100644 index 000000000..764562333 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-009/case.md @@ -0,0 +1,79 @@ + + + +# TC-VMM-VMM-009: Vmm.GetComposeHash + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vmm-009](../../../../catalog/feature-audit.md#req-vmm-vmm-009) +- Risks: [risk-vmm-vmm-009](../../../../catalog/feature-audit.md#risk-vmm-vmm-009) +- Source: `dstack/vmm/rpc/proto/vmm_rpc.proto:355` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Vmm.GetComposeHash` takes `VmConfiguration` (`name: string`, `image: string`, `compose_file: string`, `vcpu: uint32`, `memory: uint32`, `disk_size: uint32`, `ports: PortMapping`, `encrypted_env: bytes`, `app_id: string`, `user_config: string`, `hugepages: bool`, `pin_numa: bool`, `gpus: GpuConfig`, `kms_urls: string`, `gateway_urls: string`, `stopped: bool`, `no_tee: bool`, `networking: NetworkingConfig`, `networks: NetworkingConfig`, `simulated_tee: string`) and returns `ComposeHash` (`hash: string`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- JSON pRPC follows protobuf forward-compatibility semantics: an unknown field is ignored and is not a malformed request. Use a wrong field type, truncated JSON, a missing behavior-required field, or an unknown object ID for negative rows. +- In this pRPC JSON binding, a successful `google.protobuf.Empty` response is encoded as the JSON literal `null`; HTTP 200 with body `null` is the expected success representation, not a schema failure. +- VM removal is asynchronous. After a successful RemoveVm/`remove` call, poll the exact `values.vmm.commands.list_vms` command for up to 30 seconds and treat cleanup as complete only when the VM ID is absent; an immediate post-response listing may still contain the removing VM. +- Use `values.vmm.cli_argv`, `values.vmm.json_prpc_route_template`, and the exact `values.vmm.test_input.create_stopped_argv` command from the case manifest. Do not invent CLI subcommands or service-qualified pRPC routes. +- `GetComposeHash` accepts a complete `VmConfiguration`, not an `Id`. Use + `values.vmm.test_input.vm_configuration` unchanged for the positive row and + compare the returned hash with SHA-256 of its exact `compose_file` bytes. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Vmm.GetComposeHash`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Vmm.GetComposeHash` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for vmm.getcomposehash. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Vmm.GetComposeHash` with a valid `VmConfiguration` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `ComposeHash` with every documented field and exhibits the documented `GetComposeHash` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-009/metadata.json b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-009/metadata.json new file mode 100644 index 000000000..ea1889d4d --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-009/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vmm-009", + "title": "Vmm.GetComposeHash", + "priority": "P1", + "requirements": [ + "req-vmm-vmm-009" + ], + "risks": [ + "risk-vmm-vmm-009" + ], + "tags": [ + "vmm", + "vmm-rpc" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Vmm.GetComposeHash" + ], + "execution": { + "entrypoint": "shared/automation/passed-vmm-empty-rpc-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-010/case.md b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-010/case.md new file mode 100644 index 000000000..ec1f267b9 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-010/case.md @@ -0,0 +1,79 @@ + + + +# TC-VMM-VMM-010: Vmm.Status + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vmm-010](../../../../catalog/feature-audit.md#req-vmm-vmm-010) +- Risks: [risk-vmm-vmm-010](../../../../catalog/feature-audit.md#risk-vmm-vmm-010) +- Source: `dstack/vmm/rpc/proto/vmm_rpc.proto:358` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Vmm.Status` takes `StatusRequest` (`ids: string`, `brief: bool`, `keyword: string`, `page: uint32`, `page_size: uint32`) and returns `StatusResponse` (`vms: VmInfo`, `port_mapping_enabled: bool`, `total: uint32`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- JSON pRPC follows protobuf forward-compatibility semantics: an unknown field is ignored and is not a malformed request. Use a wrong field type, truncated JSON, a missing behavior-required field, or an unknown object ID for negative rows. +- In this pRPC JSON binding, a successful `google.protobuf.Empty` response is encoded as the JSON literal `null`; HTTP 200 with body `null` is the expected success representation, not a schema failure. +- VM removal is asynchronous. After a successful RemoveVm/`remove` call, poll the exact `values.vmm.commands.list_vms` command for up to 30 seconds and treat cleanup as complete only when the VM ID is absent; an immediate post-response listing may still contain the removing VM. +- Use `values.vmm.cli_argv`, `values.vmm.json_prpc_route_template`, and the exact `values.vmm.test_input.create_stopped_argv` command from the case manifest. Do not invent CLI subcommands or service-qualified pRPC routes. +- `StatusRequest.ids` is a repeated string field. Supply `ids` as a JSON array + such as `{"ids": [""], "brief": false, "page": 0, + "page_size": 10}`; a scalar string is the wrong-type negative row. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Vmm.Status`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Vmm.Status` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for vmm.status. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Vmm.Status` with a valid `StatusRequest` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `StatusResponse` with every documented field and exhibits the documented `Status` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-010/metadata.json b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-010/metadata.json new file mode 100644 index 000000000..7449d96dc --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-010/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vmm-010", + "title": "Vmm.Status", + "priority": "P1", + "requirements": [ + "req-vmm-vmm-010" + ], + "risks": [ + "risk-vmm-vmm-010" + ], + "tags": [ + "vmm", + "vmm-rpc" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Vmm.Status" + ], + "execution": { + "entrypoint": "shared/automation/passed-vmm-empty-rpc-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-011/case.md b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-011/case.md new file mode 100644 index 000000000..f31f98d4f --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-011/case.md @@ -0,0 +1,77 @@ + + + +# TC-VMM-VMM-011: Vmm.ListImages + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vmm-011](../../../../catalog/feature-audit.md#req-vmm-vmm-011) +- Risks: [risk-vmm-vmm-011](../../../../catalog/feature-audit.md#risk-vmm-vmm-011) +- Source: `dstack/vmm/rpc/proto/vmm_rpc.proto:360` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Vmm.ListImages` takes `google.protobuf.Empty` (no fields) and returns `ImageListResponse` (`images: ImageInfo`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- Empty-input transport semantics: `prpc-build` intentionally generates a zero-argument handler for `google.protobuf.Empty` and does not decode the request body. Exercise empty and extraneous/malformed bodies as body-ignored compatibility inputs, and use an invalid pRPC route for the negative transport check; GET is an explicitly supported JSON transport and is not a negative case. Do not expect malformed body rejection from an Empty-input handler. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- JSON pRPC follows protobuf forward-compatibility semantics: an unknown field is ignored and is not a malformed request. Use a wrong field type, truncated JSON, a missing behavior-required field, or an unknown object ID for negative rows. +- In this pRPC JSON binding, a successful `google.protobuf.Empty` response is encoded as the JSON literal `null`; HTTP 200 with body `null` is the expected success representation, not a schema failure. +- VM removal is asynchronous. After a successful RemoveVm/`remove` call, poll the exact `values.vmm.commands.list_vms` command for up to 30 seconds and treat cleanup as complete only when the VM ID is absent; an immediate post-response listing may still contain the removing VM. +- Use `values.vmm.cli_argv`, `values.vmm.json_prpc_route_template`, and the exact `values.vmm.test_input.create_stopped_argv` command from the case manifest. Do not invent CLI subcommands or service-qualified pRPC routes. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Vmm.ListImages`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Vmm.ListImages` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. `google.protobuf.Empty` has no request fields. Exercise the documented empty JSON/protobuf encodings plus extraneous and malformed body bytes, which the generated zero-argument handler intentionally ignores, and validate every response field, nested message field, and presence bit. Use an invalid pRPC route for negative transport framing; GET is supported by `ra-rpc`. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for vmm.listimages. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Vmm.ListImages` with a valid `google.protobuf.Empty` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send extraneous and malformed request-body bytes to confirm Empty-input body-ignore semantics, exercise an invalid pRPC route, and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `ImageListResponse` with every documented field and exhibits the documented `ListImages` state and side effects; extraneous or malformed bodies are ignored without changing the response or state, invalid routing returns a structured transport error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with an invalid route or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid routing or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-011/metadata.json b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-011/metadata.json new file mode 100644 index 000000000..e10627f02 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-011/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vmm-011", + "title": "Vmm.ListImages", + "priority": "P1", + "requirements": [ + "req-vmm-vmm-011" + ], + "risks": [ + "risk-vmm-vmm-011" + ], + "tags": [ + "vmm", + "vmm-rpc" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Vmm.ListImages" + ], + "execution": { + "entrypoint": "shared/automation/passed-vmm-empty-rpc-case.py", + "args": [], + "timeout_seconds": 180 + } +} diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-012/case.md b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-012/case.md new file mode 100644 index 000000000..bb89bcf02 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-012/case.md @@ -0,0 +1,76 @@ + + + +# TC-VMM-VMM-012: Vmm.GetAppEnvEncryptPubKey + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vmm-012](../../../../catalog/feature-audit.md#req-vmm-vmm-012) +- Risks: [risk-vmm-vmm-012](../../../../catalog/feature-audit.md#risk-vmm-vmm-012) +- Source: `dstack/vmm/rpc/proto/vmm_rpc.proto:363` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Vmm.GetAppEnvEncryptPubKey` takes `AppId` (`app_id: bytes`) and returns `PublicKeyResponse` (`public_key: bytes`, `signature: bytes`, `timestamp: uint64`, `signature_v1: bytes`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- JSON pRPC follows protobuf forward-compatibility semantics: an unknown field is ignored and is not a malformed request. Use a wrong field type, truncated JSON, a missing behavior-required field, or an unknown object ID for negative rows. +- In this pRPC JSON binding, a successful `google.protobuf.Empty` response is encoded as the JSON literal `null`; HTTP 200 with body `null` is the expected success representation, not a schema failure. +- VM removal is asynchronous. After a successful RemoveVm/`remove` call, poll the exact `values.vmm.commands.list_vms` command for up to 30 seconds and treat cleanup as complete only when the VM ID is absent; an immediate post-response listing may still contain the removing VM. +- Use `values.vmm.cli_argv`, `values.vmm.json_prpc_route_template`, and the exact `values.vmm.test_input.create_stopped_argv` command from the case manifest. Do not invent CLI subcommands or service-qualified pRPC routes. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Vmm.GetAppEnvEncryptPubKey`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Vmm.GetAppEnvEncryptPubKey` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for vmm.getappenvencryptpubkey. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Vmm.GetAppEnvEncryptPubKey` with a valid `AppId` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `PublicKeyResponse` with every documented field and exhibits the documented `GetAppEnvEncryptPubKey` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-012/metadata.json b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-012/metadata.json new file mode 100644 index 000000000..ad4c455b5 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-012/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vmm-012", + "title": "Vmm.GetAppEnvEncryptPubKey", + "priority": "P1", + "requirements": [ + "req-vmm-vmm-012" + ], + "risks": [ + "risk-vmm-vmm-012" + ], + "tags": [ + "vmm", + "vmm-rpc" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Vmm.GetAppEnvEncryptPubKey" + ], + "execution": { + "entrypoint": "shared/automation/passed-vmm-empty-rpc-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-013/case.md b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-013/case.md new file mode 100644 index 000000000..2473c8697 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-013/case.md @@ -0,0 +1,76 @@ + + + +# TC-VMM-VMM-013: Vmm.GetInfo + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vmm-013](../../../../catalog/feature-audit.md#req-vmm-vmm-013) +- Risks: [risk-vmm-vmm-013](../../../../catalog/feature-audit.md#risk-vmm-vmm-013) +- Source: `dstack/vmm/rpc/proto/vmm_rpc.proto:366` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Vmm.GetInfo` takes `Id` (`id: string`) and returns `GetInfoResponse` (`found: bool`, `info: VmInfo`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- JSON pRPC follows protobuf forward-compatibility semantics: an unknown field is ignored and is not a malformed request. Use a wrong field type, truncated JSON, a missing behavior-required field, or an unknown object ID for negative rows. +- In this pRPC JSON binding, a successful `google.protobuf.Empty` response is encoded as the JSON literal `null`; HTTP 200 with body `null` is the expected success representation, not a schema failure. +- VM removal is asynchronous. After a successful RemoveVm/`remove` call, poll the exact `values.vmm.commands.list_vms` command for up to 30 seconds and treat cleanup as complete only when the VM ID is absent; an immediate post-response listing may still contain the removing VM. +- Use `values.vmm.cli_argv`, `values.vmm.json_prpc_route_template`, and the exact `values.vmm.test_input.create_stopped_argv` command from the case manifest. Do not invent CLI subcommands or service-qualified pRPC routes. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Vmm.GetInfo`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Vmm.GetInfo` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for vmm.getinfo. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Vmm.GetInfo` with a valid `Id` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `GetInfoResponse` with every documented field and exhibits the documented `GetInfo` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-013/metadata.json b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-013/metadata.json new file mode 100644 index 000000000..bbe5e4f13 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-013/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vmm-013", + "title": "Vmm.GetInfo", + "priority": "P1", + "requirements": [ + "req-vmm-vmm-013" + ], + "risks": [ + "risk-vmm-vmm-013" + ], + "tags": [ + "vmm", + "vmm-rpc" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Vmm.GetInfo" + ], + "execution": { + "entrypoint": "shared/automation/passed-vmm-empty-rpc-case.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-014/case.md b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-014/case.md new file mode 100644 index 000000000..3676a085b --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-014/case.md @@ -0,0 +1,77 @@ + + + +# TC-VMM-VMM-014: Vmm.Version + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vmm-014](../../../../catalog/feature-audit.md#req-vmm-vmm-014) +- Risks: [risk-vmm-vmm-014](../../../../catalog/feature-audit.md#risk-vmm-vmm-014) +- Source: `dstack/vmm/rpc/proto/vmm_rpc.proto:369` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Vmm.Version` takes `google.protobuf.Empty` (no fields) and returns `VersionResponse` (`version: string`, `rev: string`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- Empty-input transport semantics: `prpc-build` intentionally generates a zero-argument handler for `google.protobuf.Empty` and does not decode the request body. Exercise empty and extraneous/malformed bodies as body-ignored compatibility inputs, and use an invalid pRPC route for the negative transport check; GET is an explicitly supported JSON transport and is not a negative case. Do not expect malformed body rejection from an Empty-input handler. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- JSON pRPC follows protobuf forward-compatibility semantics: an unknown field is ignored and is not a malformed request. Use a wrong field type, truncated JSON, a missing behavior-required field, or an unknown object ID for negative rows. +- In this pRPC JSON binding, a successful `google.protobuf.Empty` response is encoded as the JSON literal `null`; HTTP 200 with body `null` is the expected success representation, not a schema failure. +- VM removal is asynchronous. After a successful RemoveVm/`remove` call, poll the exact `values.vmm.commands.list_vms` command for up to 30 seconds and treat cleanup as complete only when the VM ID is absent; an immediate post-response listing may still contain the removing VM. +- Use `values.vmm.cli_argv`, `values.vmm.json_prpc_route_template`, and the exact `values.vmm.test_input.create_stopped_argv` command from the case manifest. Do not invent CLI subcommands or service-qualified pRPC routes. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Vmm.Version`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Vmm.Version` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. `google.protobuf.Empty` has no request fields. Exercise the documented empty JSON/protobuf encodings plus extraneous and malformed body bytes, which the generated zero-argument handler intentionally ignores, and validate every response field, nested message field, and presence bit. Use an invalid pRPC route for negative transport framing; GET is supported by `ra-rpc`. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for vmm.version. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Vmm.Version` with a valid `google.protobuf.Empty` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send extraneous and malformed request-body bytes to confirm Empty-input body-ignore semantics, exercise an invalid pRPC route, and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `VersionResponse` with every documented field and exhibits the documented `Version` state and side effects; extraneous or malformed bodies are ignored without changing the response or state, invalid routing returns a structured transport error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with an invalid route or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid routing or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-014/metadata.json b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-014/metadata.json new file mode 100644 index 000000000..45e9ea85d --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-014/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vmm-014", + "title": "Vmm.Version", + "priority": "P1", + "requirements": [ + "req-vmm-vmm-014" + ], + "risks": [ + "risk-vmm-vmm-014" + ], + "tags": [ + "vmm", + "vmm-rpc" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Vmm.Version" + ], + "execution": { + "entrypoint": "shared/automation/passed-vmm-empty-rpc-case.py", + "args": [], + "timeout_seconds": 180 + } +} diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-015/case.md b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-015/case.md new file mode 100644 index 000000000..419060ab7 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-015/case.md @@ -0,0 +1,77 @@ + + + +# TC-VMM-VMM-015: Vmm.GetMeta + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vmm-015](../../../../catalog/feature-audit.md#req-vmm-vmm-015) +- Risks: [risk-vmm-vmm-015](../../../../catalog/feature-audit.md#risk-vmm-vmm-015) +- Source: `dstack/vmm/rpc/proto/vmm_rpc.proto:372` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Vmm.GetMeta` takes `google.protobuf.Empty` (no fields) and returns `GetMetaResponse` (`kms: KmsSettings`, `gateway: GatewaySettings`, `resources: ResourcesSettings`, `networking: NetworkingCapabilities`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- Empty-input transport semantics: `prpc-build` intentionally generates a zero-argument handler for `google.protobuf.Empty` and does not decode the request body. Exercise empty and extraneous/malformed bodies as body-ignored compatibility inputs, and use an invalid pRPC route for the negative transport check; GET is an explicitly supported JSON transport and is not a negative case. Do not expect malformed body rejection from an Empty-input handler. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- JSON pRPC follows protobuf forward-compatibility semantics: an unknown field is ignored and is not a malformed request. Use a wrong field type, truncated JSON, a missing behavior-required field, or an unknown object ID for negative rows. +- In this pRPC JSON binding, a successful `google.protobuf.Empty` response is encoded as the JSON literal `null`; HTTP 200 with body `null` is the expected success representation, not a schema failure. +- VM removal is asynchronous. After a successful RemoveVm/`remove` call, poll the exact `values.vmm.commands.list_vms` command for up to 30 seconds and treat cleanup as complete only when the VM ID is absent; an immediate post-response listing may still contain the removing VM. +- Use `values.vmm.cli_argv`, `values.vmm.json_prpc_route_template`, and the exact `values.vmm.test_input.create_stopped_argv` command from the case manifest. Do not invent CLI subcommands or service-qualified pRPC routes. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Vmm.GetMeta`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Vmm.GetMeta` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. `google.protobuf.Empty` has no request fields. Exercise the documented empty JSON/protobuf encodings plus extraneous and malformed body bytes, which the generated zero-argument handler intentionally ignores, and validate every response field, nested message field, and presence bit. Use an invalid pRPC route for negative transport framing; GET is supported by `ra-rpc`. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for vmm.getmeta. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Vmm.GetMeta` with a valid `google.protobuf.Empty` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send extraneous and malformed request-body bytes to confirm Empty-input body-ignore semantics, exercise an invalid pRPC route, and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `GetMetaResponse` with every documented field and exhibits the documented `GetMeta` state and side effects; extraneous or malformed bodies are ignored without changing the response or state, invalid routing returns a structured transport error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with an invalid route or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid routing or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-015/metadata.json b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-015/metadata.json new file mode 100644 index 000000000..12c7a8b0a --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-015/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vmm-015", + "title": "Vmm.GetMeta", + "priority": "P1", + "requirements": [ + "req-vmm-vmm-015" + ], + "risks": [ + "risk-vmm-vmm-015" + ], + "tags": [ + "vmm", + "vmm-rpc" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Vmm.GetMeta" + ], + "execution": { + "entrypoint": "shared/automation/passed-vmm-empty-rpc-case.py", + "args": [], + "timeout_seconds": 180 + } +} diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-016/case.md b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-016/case.md new file mode 100644 index 000000000..53b5748bc --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-016/case.md @@ -0,0 +1,85 @@ + + + +# TC-VMM-VMM-016: Vmm.ListGpus + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vmm-016](../../../../catalog/feature-audit.md#req-vmm-vmm-016) +- Risks: [risk-vmm-vmm-016](../../../../catalog/feature-audit.md#risk-vmm-vmm-016) +- Source: `dstack/vmm/rpc/proto/vmm_rpc.proto:375` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Vmm.ListGpus` takes `google.protobuf.Empty` (no fields) and returns `ListGpusResponse` (`gpus: GpuInfo`, `allow_attach_all: bool`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- Empty-input transport semantics: `prpc-build` intentionally generates a zero-argument handler for `google.protobuf.Empty` and does not decode the request body. Exercise empty and extraneous/malformed bodies as body-ignored compatibility inputs, and use an invalid pRPC route for the negative transport check; GET is an explicitly supported JSON transport and is not a negative case. Do not expect malformed body rejection from an Empty-input handler. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- JSON pRPC follows protobuf forward-compatibility semantics: an unknown field is ignored and is not a malformed request. Use a wrong field type, truncated JSON, a missing behavior-required field, or an unknown object ID for negative rows. +- In this pRPC JSON binding, a successful `google.protobuf.Empty` response is encoded as the JSON literal `null`; HTTP 200 with body `null` is the expected success representation, not a schema failure. +- VM removal is asynchronous. After a successful RemoveVm/`remove` call, poll the exact `values.vmm.commands.list_vms` command for up to 30 seconds and treat cleanup as complete only when the VM ID is absent; an immediate post-response listing may still contain the removing VM. +- Use `values.vmm.cli_argv`, `values.vmm.json_prpc_route_template`, and the exact `values.vmm.test_input.create_stopped_argv` command from the case manifest. Do not invent CLI subcommands or service-qualified pRPC routes. +- A host with no matching GPU is a valid `ListGpus` test target. The expected + positive result is a successful response with an empty `gpus` list and the + effective `allow_attach_all` policy; do not mark this RPC case BLOCKED merely + because the case-owned VMM has no assignable GPU. +- Invoke the exact `values.vmm.json_prpc_routes.ListGpus` route + (`/prpc/ListGpus?json` in this fixture). `/prpc/Vmm/ListGpus`, + `/prpc/Vmm.ListGpus`, and other service-qualified paths are invalid Rocket + routes and must never be used for the positive row. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Vmm.ListGpus`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Vmm.ListGpus` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. `google.protobuf.Empty` has no request fields. Exercise the documented empty JSON/protobuf encodings plus extraneous and malformed body bytes, which the generated zero-argument handler intentionally ignores, and validate every response field, nested message field, and presence bit. Use an invalid pRPC route for negative transport framing; GET is supported by `ra-rpc`. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for vmm.listgpus. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Vmm.ListGpus` with a valid `google.protobuf.Empty` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send extraneous and malformed request-body bytes to confirm Empty-input body-ignore semantics, exercise an invalid pRPC route, and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `ListGpusResponse` with every documented field and exhibits the documented `ListGpus` state and side effects; extraneous or malformed bodies are ignored without changing the response or state, invalid routing returns a structured transport error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with an invalid route or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid routing or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-016/metadata.json b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-016/metadata.json new file mode 100644 index 000000000..1964f73ed --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-016/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vmm-016", + "title": "Vmm.ListGpus", + "priority": "P1", + "requirements": [ + "req-vmm-vmm-016" + ], + "risks": [ + "risk-vmm-vmm-016" + ], + "tags": [ + "vmm", + "vmm-rpc" + ], + "fixture": { + "profile": "gpu-policy", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": true, + "simulation_allowed": false + }, + "actions_under_test": [ + "Vmm.ListGpus" + ], + "execution": { + "entrypoint": "shared/automation/capability-probe-case.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-017/case.md b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-017/case.md new file mode 100644 index 000000000..505ed18f6 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-017/case.md @@ -0,0 +1,77 @@ + + + +# TC-VMM-VMM-017: Vmm.ReloadVms + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vmm-017](../../../../catalog/feature-audit.md#req-vmm-vmm-017) +- Risks: [risk-vmm-vmm-017](../../../../catalog/feature-audit.md#risk-vmm-vmm-017) +- Source: `dstack/vmm/rpc/proto/vmm_rpc.proto:378` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Vmm.ReloadVms` takes `google.protobuf.Empty` (no fields) and returns `ReloadVmsResponse` (`loaded: uint32`, `updated: uint32`, `removed: uint32`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- Empty-input transport semantics: `prpc-build` intentionally generates a zero-argument handler for `google.protobuf.Empty` and does not decode the request body. Exercise empty and extraneous/malformed bodies as body-ignored compatibility inputs, and use an invalid pRPC route for the negative transport check; GET is an explicitly supported JSON transport and is not a negative case. Do not expect malformed body rejection from an Empty-input handler. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- JSON pRPC follows protobuf forward-compatibility semantics: an unknown field is ignored and is not a malformed request. Use a wrong field type, truncated JSON, a missing behavior-required field, or an unknown object ID for negative rows. +- In this pRPC JSON binding, a successful `google.protobuf.Empty` response is encoded as the JSON literal `null`; HTTP 200 with body `null` is the expected success representation, not a schema failure. +- VM removal is asynchronous. After a successful RemoveVm/`remove` call, poll the exact `values.vmm.commands.list_vms` command for up to 30 seconds and treat cleanup as complete only when the VM ID is absent; an immediate post-response listing may still contain the removing VM. +- Use `values.vmm.cli_argv`, `values.vmm.json_prpc_route_template`, and the exact `values.vmm.test_input.create_stopped_argv` command from the case manifest. Do not invent CLI subcommands or service-qualified pRPC routes. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Vmm.ReloadVms`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Vmm.ReloadVms` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. `google.protobuf.Empty` has no request fields. Exercise the documented empty JSON/protobuf encodings plus extraneous and malformed body bytes, which the generated zero-argument handler intentionally ignores, and validate every response field, nested message field, and presence bit. Use an invalid pRPC route for negative transport framing; GET is supported by `ra-rpc`. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for vmm.reloadvms. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Vmm.ReloadVms` with a valid `google.protobuf.Empty` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send extraneous and malformed request-body bytes to confirm Empty-input body-ignore semantics, exercise an invalid pRPC route, and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `ReloadVmsResponse` with every documented field and exhibits the documented `ReloadVms` state and side effects; extraneous or malformed bodies are ignored without changing the response or state, invalid routing returns a structured transport error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with an invalid route or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid routing or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-017/metadata.json b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-017/metadata.json new file mode 100644 index 000000000..3d8777f48 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-017/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vmm-017", + "title": "Vmm.ReloadVms", + "priority": "P1", + "requirements": [ + "req-vmm-vmm-017" + ], + "risks": [ + "risk-vmm-vmm-017" + ], + "tags": [ + "vmm", + "vmm-rpc" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Vmm.ReloadVms" + ], + "execution": { + "entrypoint": "shared/automation/passed-vmm-empty-rpc-case.py", + "args": [], + "timeout_seconds": 180 + } +} diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-018/case.md b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-018/case.md new file mode 100644 index 000000000..80de733ab --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-018/case.md @@ -0,0 +1,77 @@ + + + +# TC-VMM-VMM-018: Vmm.SvList + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vmm-018](../../../../catalog/feature-audit.md#req-vmm-vmm-018) +- Risks: [risk-vmm-vmm-018](../../../../catalog/feature-audit.md#risk-vmm-vmm-018) +- Source: `dstack/vmm/rpc/proto/vmm_rpc.proto:381` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Vmm.SvList` takes `google.protobuf.Empty` (no fields) and returns `SvListResponse` (`processes: SvProcessInfo`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- Empty-input transport semantics: `prpc-build` intentionally generates a zero-argument handler for `google.protobuf.Empty` and does not decode the request body. Exercise empty and extraneous/malformed bodies as body-ignored compatibility inputs, and use an invalid pRPC route for the negative transport check; GET is an explicitly supported JSON transport and is not a negative case. Do not expect malformed body rejection from an Empty-input handler. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- JSON pRPC follows protobuf forward-compatibility semantics: an unknown field is ignored and is not a malformed request. Use a wrong field type, truncated JSON, a missing behavior-required field, or an unknown object ID for negative rows. +- In this pRPC JSON binding, a successful `google.protobuf.Empty` response is encoded as the JSON literal `null`; HTTP 200 with body `null` is the expected success representation, not a schema failure. +- VM removal is asynchronous. After a successful RemoveVm/`remove` call, poll the exact `values.vmm.commands.list_vms` command for up to 30 seconds and treat cleanup as complete only when the VM ID is absent; an immediate post-response listing may still contain the removing VM. +- Use `values.vmm.cli_argv`, `values.vmm.json_prpc_route_template`, and the exact `values.vmm.test_input.create_stopped_argv` command from the case manifest. Do not invent CLI subcommands or service-qualified pRPC routes. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Vmm.SvList`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Vmm.SvList` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. `google.protobuf.Empty` has no request fields. Exercise the documented empty JSON/protobuf encodings plus extraneous and malformed body bytes, which the generated zero-argument handler intentionally ignores, and validate every response field, nested message field, and presence bit. Use an invalid pRPC route for negative transport framing; GET is supported by `ra-rpc`. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for vmm.svlist. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Vmm.SvList` with a valid `google.protobuf.Empty` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send extraneous and malformed request-body bytes to confirm Empty-input body-ignore semantics, exercise an invalid pRPC route, and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `SvListResponse` with every documented field and exhibits the documented `SvList` state and side effects; extraneous or malformed bodies are ignored without changing the response or state, invalid routing returns a structured transport error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with an invalid route or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid routing or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-018/metadata.json b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-018/metadata.json new file mode 100644 index 000000000..d0db782d6 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-018/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vmm-018", + "title": "Vmm.SvList", + "priority": "P1", + "requirements": [ + "req-vmm-vmm-018" + ], + "risks": [ + "risk-vmm-vmm-018" + ], + "tags": [ + "vmm", + "vmm-rpc" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Vmm.SvList" + ], + "execution": { + "entrypoint": "shared/automation/passed-vmm-empty-rpc-case.py", + "args": [], + "timeout_seconds": 180 + } +} diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-019/case.md b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-019/case.md new file mode 100644 index 000000000..a02b19fee --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-019/case.md @@ -0,0 +1,81 @@ + + + +# TC-VMM-VMM-019: Vmm.SvStop + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vmm-019](../../../../catalog/feature-audit.md#req-vmm-vmm-019) +- Risks: [risk-vmm-vmm-019](../../../../catalog/feature-audit.md#risk-vmm-vmm-019) +- Source: `dstack/vmm/rpc/proto/vmm_rpc.proto:383` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Vmm.SvStop` takes `Id` (`id: string`) and returns `google.protobuf.Empty` (no fields). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- JSON pRPC follows protobuf forward-compatibility semantics: an unknown field is ignored and is not a malformed request. Use a wrong field type, truncated JSON, a missing behavior-required field, or an unknown object ID for negative rows. +- In this pRPC JSON binding, a successful `google.protobuf.Empty` response is encoded as the JSON literal `null`; HTTP 200 with body `null` is the expected success representation, not a schema failure. +- VM removal is asynchronous. After a successful RemoveVm/`remove` call, poll the exact `values.vmm.commands.list_vms` command for up to 30 seconds and treat cleanup as complete only when the VM ID is absent; an immediate post-response listing may still contain the removing VM. +- Use `values.vmm.cli_argv`, `values.vmm.json_prpc_route_template`, and the exact `values.vmm.test_input.create_stopped_argv` command from the case manifest. Do not invent CLI subcommands or service-qualified pRPC routes. +- `SvStop` controls a supervisor process, not a persisted VM record. Create the lease-owned VM with `values.vmm.test_input.create_stopped_helper_argv`, start it with the prepared `StartVm` JSON route, and poll `SvList` until the process appears. Use the exact `SvListResponse.processes[].id` value as the positive `SvStop.id`; do not pass a stopped VM ID that is absent from `SvList`. +- The action-specific fixture disables VMM auto-restart. After successful + `SvStop`, require the same process ID to remain present in `SvList` with + `status == "stopped"`; `SvStop` does not remove the supervisor record, so + polling for absence is an invalid expectation. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Vmm.SvStop`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Vmm.SvStop` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for vmm.svstop. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Create and start a lease-owned VM, poll `Vmm.SvList` until its supervisor process appears, and invoke `Vmm.SvStop` with that process ID using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `google.protobuf.Empty` with every documented field and the targeted supervisor process transitions out of the running state without affecting unrelated processes; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-019/metadata.json b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-019/metadata.json new file mode 100644 index 000000000..9c221622c --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-019/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vmm-019", + "title": "Vmm.SvStop", + "priority": "P1", + "requirements": [ + "req-vmm-vmm-019" + ], + "risks": [ + "risk-vmm-vmm-019" + ], + "tags": [ + "vmm", + "vmm-rpc" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Vmm.SvStop" + ], + "execution": { + "entrypoint": "shared/automation/passed-vmm-lifecycle-case.py", + "args": [], + "timeout_seconds": 900 + } +} diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-020/case.md b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-020/case.md new file mode 100644 index 000000000..b30329a0f --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-020/case.md @@ -0,0 +1,81 @@ + + + +# TC-VMM-VMM-020: Vmm.SvRemove + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vmm-020](../../../../catalog/feature-audit.md#req-vmm-vmm-020) +- Risks: [risk-vmm-vmm-020](../../../../catalog/feature-audit.md#risk-vmm-vmm-020) +- Source: `dstack/vmm/rpc/proto/vmm_rpc.proto:385` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Vmm.SvRemove` takes `Id` (`id: string`) and returns `google.protobuf.Empty` (no fields). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- JSON pRPC follows protobuf forward-compatibility semantics: an unknown field is ignored and is not a malformed request. Use a wrong field type, truncated JSON, a missing behavior-required field, or an unknown object ID for negative rows. +- In this pRPC JSON binding, a successful `google.protobuf.Empty` response is encoded as the JSON literal `null`; HTTP 200 with body `null` is the expected success representation, not a schema failure. +- VM removal is asynchronous. After a successful RemoveVm/`remove` call, poll the exact `values.vmm.commands.list_vms` command for up to 30 seconds and treat cleanup as complete only when the VM ID is absent; an immediate post-response listing may still contain the removing VM. +- Use `values.vmm.cli_argv`, `values.vmm.json_prpc_route_template`, and the exact `values.vmm.test_input.create_stopped_argv` command from the case manifest. Do not invent CLI subcommands or service-qualified pRPC routes. +- `SvRemove` controls a supervisor process, not a persisted VM record. Create the lease-owned VM with `values.vmm.test_input.create_stopped_helper_argv`, start it with the prepared `StartVm` JSON route, and poll `SvList` until the process appears. Use the exact `SvListResponse.processes[].id` value as the positive `SvRemove.id`; do not pass a stopped VM ID that is absent from `SvList`. +- The action-specific fixture disables VMM auto-restart. Stop the supervisor + process first and require `status == "stopped"`; only the subsequent + successful `SvRemove` is expected to make the process ID disappear from + `SvList`. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Vmm.SvRemove`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Vmm.SvRemove` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for vmm.svremove. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Create and start a lease-owned VM, poll `Vmm.SvList` until its supervisor process appears, stop that process with `Vmm.SvStop`, and invoke `Vmm.SvRemove` with the same supervisor process ID using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `google.protobuf.Empty` with every documented field and the targeted stopped supervisor process disappears from `SvList` without affecting unrelated processes; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-020/metadata.json b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-020/metadata.json new file mode 100644 index 000000000..f5100b021 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-020/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vmm-020", + "title": "Vmm.SvRemove", + "priority": "P1", + "requirements": [ + "req-vmm-vmm-020" + ], + "risks": [ + "risk-vmm-vmm-020" + ], + "tags": [ + "vmm", + "vmm-rpc" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Vmm.SvRemove" + ], + "execution": { + "entrypoint": "shared/automation/passed-vmm-lifecycle-case.py", + "args": [], + "timeout_seconds": 900 + } +} diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-021/case.md b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-021/case.md new file mode 100644 index 000000000..1587acbc1 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-021/case.md @@ -0,0 +1,77 @@ + + + +# TC-VMM-VMM-021: Vmm.ListRegistryImages + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vmm-021](../../../../catalog/feature-audit.md#req-vmm-vmm-021) +- Risks: [risk-vmm-vmm-021](../../../../catalog/feature-audit.md#risk-vmm-vmm-021) +- Source: `dstack/vmm/rpc/proto/vmm_rpc.proto:388` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Vmm.ListRegistryImages` takes `google.protobuf.Empty` (no fields) and returns `RegistryImageListResponse` (`images: RegistryImageInfo`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- Empty-input transport semantics: `prpc-build` intentionally generates a zero-argument handler for `google.protobuf.Empty` and does not decode the request body. Exercise empty and extraneous/malformed bodies as body-ignored compatibility inputs, and use an invalid pRPC route for the negative transport check; GET is an explicitly supported JSON transport and is not a negative case. Do not expect malformed body rejection from an Empty-input handler. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- JSON pRPC follows protobuf forward-compatibility semantics: an unknown field is ignored and is not a malformed request. Use a wrong field type, truncated JSON, a missing behavior-required field, or an unknown object ID for negative rows. +- In this pRPC JSON binding, a successful `google.protobuf.Empty` response is encoded as the JSON literal `null`; HTTP 200 with body `null` is the expected success representation, not a schema failure. +- VM removal is asynchronous. After a successful RemoveVm/`remove` call, poll the exact `values.vmm.commands.list_vms` command for up to 30 seconds and treat cleanup as complete only when the VM ID is absent; an immediate post-response listing may still contain the removing VM. +- Use `values.vmm.cli_argv`, `values.vmm.json_prpc_route_template`, and the exact `values.vmm.test_input.create_stopped_argv` command from the case manifest. Do not invent CLI subcommands or service-qualified pRPC routes. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Vmm.ListRegistryImages`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Vmm.ListRegistryImages` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. `google.protobuf.Empty` has no request fields. Exercise the documented empty JSON/protobuf encodings plus extraneous and malformed body bytes, which the generated zero-argument handler intentionally ignores, and validate every response field, nested message field, and presence bit. Use an invalid pRPC route for negative transport framing; GET is supported by `ra-rpc`. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for vmm.listregistryimages. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Vmm.ListRegistryImages` with a valid `google.protobuf.Empty` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send extraneous and malformed request-body bytes to confirm Empty-input body-ignore semantics, exercise an invalid pRPC route, and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `RegistryImageListResponse` with every documented field and exhibits the documented `ListRegistryImages` state and side effects; extraneous or malformed bodies are ignored without changing the response or state, invalid routing returns a structured transport error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with an invalid route or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid routing or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-021/metadata.json b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-021/metadata.json new file mode 100644 index 000000000..8520f8473 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-021/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vmm-021", + "title": "Vmm.ListRegistryImages", + "priority": "P1", + "requirements": [ + "req-vmm-vmm-021" + ], + "risks": [ + "risk-vmm-vmm-021" + ], + "tags": [ + "vmm", + "vmm-rpc" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Vmm.ListRegistryImages" + ], + "execution": { + "entrypoint": "shared/automation/passed-vmm-empty-rpc-case.py", + "args": [], + "timeout_seconds": 180 + } +} diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-022/case.md b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-022/case.md new file mode 100644 index 000000000..7d91b6b82 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-022/case.md @@ -0,0 +1,78 @@ + + + +# TC-VMM-VMM-022: Vmm.PullRegistryImage + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vmm-022](../../../../catalog/feature-audit.md#req-vmm-vmm-022) +- Risks: [risk-vmm-vmm-022](../../../../catalog/feature-audit.md#risk-vmm-vmm-022) +- Source: `dstack/vmm/rpc/proto/vmm_rpc.proto:390` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Vmm.PullRegistryImage` takes `PullRegistryImageRequest` (`tag: string`) and returns `google.protobuf.Empty` (no fields). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- JSON pRPC follows protobuf forward-compatibility semantics: an unknown field is ignored and is not a malformed request. Use a wrong field type, truncated JSON, a missing behavior-required field, or an unknown object ID for negative rows. +- In this pRPC JSON binding, a successful `google.protobuf.Empty` response is encoded as the JSON literal `null`; HTTP 200 with body `null` is the expected success representation, not a schema failure. +- VM removal is asynchronous. After a successful RemoveVm/`remove` call, poll the exact `values.vmm.commands.list_vms` command for up to 30 seconds and treat cleanup as complete only when the VM ID is absent; an immediate post-response listing may still contain the removing VM. +- Use `values.vmm.cli_argv`, `values.vmm.json_prpc_route_template`, and the exact `values.vmm.test_input.create_stopped_argv` command from the case manifest. Do not invent CLI subcommands or service-qualified pRPC routes. +- Use `values.vmm.test_input.registry_tag` as the valid tag. The fixture configures the case-owned VMM with `values.vmm.test_input.registry`; do not substitute another registry or tag. `PullRegistryImage` starts an asynchronous pull, so poll `ListRegistryImages` for the selected tag until `pulling` is false and require `local=true` with an empty `error` before grading the positive path. +- Poll for up to 60 seconds at intervals of at least 2 seconds; the authenticated fixture token exchange can take about 30 seconds under the case-owned server. Complete the valid pull and observe its terminal state before sending absent, unknown-field, wrong-type, or nonexistent-tag rows so they cannot contend with or obscure the positive background task. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Vmm.PullRegistryImage`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Vmm.PullRegistryImage` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for vmm.pullregistryimage. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `Vmm.PullRegistryImage` with the fixture-provided valid registry tag using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations and poll the registry-image status to completion. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `google.protobuf.Empty` with every documented field and exhibits the documented `PullRegistryImage` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-022/metadata.json b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-022/metadata.json new file mode 100644 index 000000000..5af91c908 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-022/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vmm-022", + "title": "Vmm.PullRegistryImage", + "priority": "P1", + "requirements": [ + "req-vmm-vmm-022" + ], + "risks": [ + "risk-vmm-vmm-022" + ], + "tags": [ + "vmm", + "vmm-rpc" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Vmm.PullRegistryImage" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 180 + } +} diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-022/run.py b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-022/run.py new file mode 100755 index 000000000..b7b44477a --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-022/run.py @@ -0,0 +1,282 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 +"""Deterministic registry-image pull lifecycle for a lease-owned VMM.""" + +from __future__ import annotations + +import json +import os +import pathlib +import tempfile +import time +import urllib.error +import urllib.request +from typing import Any + +CASE = "tc-vmm-vmm-022" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile("w", dir=path.parent, delete=False) as handle: + json.dump(value, handle, indent=2, sort_keys=True) + handle.write("\n") + temporary = pathlib.Path(handle.name) + temporary.replace(path) + + +def varint(value: int) -> bytes: + """Encode a protobuf unsigned varint.""" + output = bytearray() + while True: + byte = value & 0x7F + value >>= 7 + output.append(byte | (0x80 if value else 0)) + if not value: + return bytes(output) + + +def encode_tag(tag: str) -> bytes: + """Encode PullRegistryImageRequest.tag (field 1).""" + raw = tag.encode() + return varint((1 << 3) | 2) + varint(len(raw)) + raw + + +def call( + url: str, body: bytes, content_type: str, headers: dict[str, str] +) -> tuple[int, bytes]: + """Perform one bounded pRPC request.""" + request = urllib.request.Request(url, data=body, method="POST") + request.add_header("Content-Type", content_type) + for key, value in headers.items(): + request.add_header(key, value) + try: + with urllib.request.urlopen(request, timeout=30) as response: + return int(response.status), response.read() + except urllib.error.HTTPError as error: + return int(error.code), error.read() + + +def image_status( + base: str, routes: dict[str, str], headers: dict[str, str], tag: str +) -> dict[str, Any]: + """Return the exact registry row for the fixture tag.""" + path = (routes.get("ListRegistryImages") or "/prpc/ListRegistryImages?json").split( + "?", 1 + )[0] + code, body = call(base + path, b"{}", "application/json", headers) + if code != 200: + raise RuntimeError(f"ListRegistryImages returned HTTP {code}") + value = json.loads(body or b"{}") + rows = value.get("images") if isinstance(value, dict) else None + matches = [row for row in (rows or []) if row.get("tag") == tag] + if len(matches) != 1: + raise AssertionError(f"registry tag {tag!r} had {len(matches)} rows") + return matches[0] + + +def await_local( + base: str, + routes: dict[str, str], + headers: dict[str, str], + tag: str, + wanted: bool, + timeout: int = 60, +) -> dict[str, Any]: + """Poll until the fixture tag reaches its requested local state.""" + deadline = time.monotonic() + timeout + observed: dict[str, Any] = {} + while time.monotonic() < deadline: + observed = image_status(base, routes, headers, tag) + error = str(observed.get("error") or "") + if error: + raise AssertionError(f"registry pull failed: {error[:300]}") + if bool(observed.get("local")) is wanted and not observed.get("pulling"): + return observed + time.sleep(1) + raise AssertionError(f"registry tag did not reach local={wanted}: {observed}") + + +def main() -> int: + """Exercise PullRegistryImage over JSON and protobuf.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE: + raise RuntimeError(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + vmm = manifest["values"]["vmm"] + if vmm.get("case_owned") is not True: + raise RuntimeError("VMM fixture is not case-owned") + tag = str(vmm["test_input"].get("registry_tag") or "") + registry = str(vmm["test_input"].get("registry") or "") + if not tag or not registry: + raise RuntimeError("fixture did not provide a registry and disposable tag") + base = str(vmm["rpc_url"]).rstrip("/") + routes = vmm.get("json_prpc_routes") or {} + headers: dict[str, str] = {} + auth = vmm.get("auth") or {} + token_file = auth.get("token_file") + if auth.get("enabled") and token_file: + token = pathlib.Path(token_file).read_text().strip() + if token: + headers["Authorization"] = f"Bearer {token}" + pull_path = ( + routes.get("PullRegistryImage") or "/prpc/PullRegistryImage?json" + ).split("?", 1)[0] + delete_path = (routes.get("DeleteImage") or "/prpc/DeleteImage?json").split("?", 1)[ + 0 + ] + evidence: dict[str, Any] = {"tag": tag, "registry_configured": True} + steps: list[dict[str, str]] = [] + failure: str | None = None + + def delete_local() -> int: + code, body = call( + base + delete_path, + json.dumps({"id": tag}).encode(), + "application/json", + headers, + ) + if code != 200: + raise AssertionError( + f"DeleteImage cleanup returned HTTP {code}: " + f"{body.decode('utf-8', 'replace')[:300]}" + ) + await_local(base, routes, headers, tag, False) + return code + + try: + baseline = image_status(base, routes, headers, tag) + if baseline.get("pulling") or baseline.get("error"): + raise AssertionError( + f"fixture registry baseline was not healthy: {baseline}" + ) + evidence["baseline"] = baseline + steps.append( + { + "id": f"{case_id}-step-01", + "status": "PASS", + "observed": "The case-owned registry exposed exactly one healthy fixture tag.", + } + ) + + json_code, json_body = call( + base + pull_path, + json.dumps({"tag": tag, "future_field": "ignored"}).encode(), + "application/json", + headers, + ) + if json_code != 200 or json_body not in (b"", b"null"): + raise AssertionError( + f"JSON pull returned HTTP {json_code} and {len(json_body)} bytes" + ) + json_final = await_local(base, routes, headers, tag, True) + between_delete = delete_local() + protobuf_code, protobuf_body = call( + base + pull_path, encode_tag(tag), "application/octet-stream", headers + ) + if protobuf_code != 200 or protobuf_body: + raise AssertionError( + f"protobuf pull returned HTTP {protobuf_code} and " + f"{len(protobuf_body)} bytes" + ) + protobuf_final = await_local(base, routes, headers, tag, True) + evidence["representations"] = { + "json_http": json_code, + "json_final": json_final, + "between_delete_http": between_delete, + "protobuf_http": protobuf_code, + "protobuf_final": protobuf_final, + } + steps.append( + { + "id": f"{case_id}-step-02", + "status": "PASS", + "observed": "JSON and protobuf pulls independently downloaded the fixture tag and reached local=true without an error.", + } + ) + + wrong_type, _ = call( + base + pull_path, + json.dumps({"tag": 7}).encode(), + "application/json", + headers, + ) + malformed, _ = call( + base + pull_path, b"\x0a\x80", "application/octet-stream", headers + ) + bad_route, _ = call( + base + pull_path + "NoSuch", b"{}", "application/json", headers + ) + if min(wrong_type, malformed, bad_route) < 400: + raise AssertionError( + f"invalid probes were accepted: {wrong_type}, {malformed}, {bad_route}" + ) + healthy = image_status(base, routes, headers, tag) + if not healthy.get("local") or healthy.get("pulling") or healthy.get("error"): + raise AssertionError( + f"invalid probes disturbed the pulled image: {healthy}" + ) + evidence["negative"] = { + "wrong_type_http": wrong_type, + "malformed_protobuf_http": malformed, + "invalid_route_http": bad_route, + "healthy_after": healthy, + } + steps.append( + { + "id": f"{case_id}-step-03", + "status": "PASS", + "observed": "Wrong-typed JSON, malformed protobuf, and an invalid route were rejected without disturbing the pulled image.", + } + ) + except Exception as error: # noqa: BLE001 + failure = f"{type(error).__name__}: {error}" + done = {step["id"] for step in steps} + for number in range(1, 4): + step_id = f"{case_id}-step-{number:02d}" + if step_id not in done: + steps.append({"id": step_id, "status": "FAIL", "observed": failure}) + finally: + try: + if image_status(base, routes, headers, tag).get("local"): + evidence["cleanup_http"] = delete_local() + except Exception as error: # noqa: BLE001 + if failure is None: + failure = f"cleanup {type(error).__name__}: {error}" + + artifact = { + "path": "artifacts/registry-pull-lifecycle.json", + "step_id": f"{case_id}-step-02", + "name": "Registry pull lifecycle", + "description": "Records the fixture tag state across JSON/protobuf pulls, rejection probes, and cleanup.", + } + atomic_json(result_dir / artifact["path"], evidence) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + status = "PASS" if failure is None else "FAIL" + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": ( + "Vmm.PullRegistryImage downloaded the fixture tag over JSON and protobuf and rejected malformed requests." + if status == "PASS" + else failure + ), + "steps": steps, + "artifacts": [artifact], + "remarks": "The pulled image is deleted after verification; the mock registry and image store are lease-owned.", + }, + ) + return 0 if status == "PASS" else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-023/case.md b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-023/case.md new file mode 100644 index 000000000..b794d749b --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-023/case.md @@ -0,0 +1,77 @@ + + + +# TC-VMM-VMM-023: Vmm.DeleteImage + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vmm-023](../../../../catalog/feature-audit.md#req-vmm-vmm-023) +- Risks: [risk-vmm-vmm-023](../../../../catalog/feature-audit.md#risk-vmm-vmm-023) +- Source: `dstack/vmm/rpc/proto/vmm_rpc.proto:392` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `Vmm.DeleteImage` takes `Id` (`id: string`) and returns `google.protobuf.Empty` (no fields). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- JSON pRPC follows protobuf forward-compatibility semantics: an unknown field is ignored and is not a malformed request. Use a wrong field type, truncated JSON, a missing behavior-required field, or an unknown object ID for negative rows. +- In this pRPC JSON binding, a successful `google.protobuf.Empty` response is encoded as the JSON literal `null`; HTTP 200 with body `null` is the expected success representation, not a schema failure. +- VM removal is asynchronous. After a successful RemoveVm/`remove` call, poll the exact `values.vmm.commands.list_vms` command for up to 30 seconds and treat cleanup as complete only when the VM ID is absent; an immediate post-response listing may still contain the removing VM. +- Use `values.vmm.cli_argv`, `values.vmm.json_prpc_route_template`, and the exact `values.vmm.test_input.create_stopped_argv` command from the case manifest. Do not invent CLI subcommands or service-qualified pRPC routes. +- Delete only `values.vmm.test_input.deletable_image`. It is a disposable regular directory in the case-owned image store. Candidate images are exposed through read-only source symlinks and must never be deletion targets. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `Vmm.DeleteImage`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `Vmm.DeleteImage` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for vmm.deleteimage. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Confirm the fixture-provided disposable image is listed, invoke `Vmm.DeleteImage` with that exact image ID using valid service-specific authentication and attestation context, and capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `google.protobuf.Empty` with every documented field and exhibits the documented `DeleteImage` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-023/metadata.json b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-023/metadata.json new file mode 100644 index 000000000..6e3358ffa --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-023/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vmm-023", + "title": "Vmm.DeleteImage", + "priority": "P1", + "requirements": [ + "req-vmm-vmm-023" + ], + "risks": [ + "risk-vmm-vmm-023" + ], + "tags": [ + "vmm", + "vmm-rpc" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Vmm.DeleteImage" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-023/run.py b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-023/run.py new file mode 100755 index 000000000..2e33be756 --- /dev/null +++ b/test-suites/cases/02-vmm/01-rpc-vmm/tc-vmm-vmm-023/run.py @@ -0,0 +1,241 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 +"""Deterministic DeleteImage state transitions in a lease-owned image store.""" + +from __future__ import annotations + +import json +import os +import pathlib +import tempfile +import urllib.error +import urllib.request +from typing import Any + +CASE = "tc-vmm-vmm-023" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile("w", dir=path.parent, delete=False) as handle: + json.dump(value, handle, indent=2, sort_keys=True) + handle.write("\n") + temporary = pathlib.Path(handle.name) + temporary.replace(path) + + +def varint(value: int) -> bytes: + """Encode a protobuf unsigned varint.""" + output = bytearray() + while True: + byte = value & 0x7F + value >>= 7 + output.append(byte | (0x80 if value else 0)) + if not value: + return bytes(output) + + +def encode_id(image_id: str) -> bytes: + """Encode Id.id (field 1).""" + raw = image_id.encode() + return varint((1 << 3) | 2) + varint(len(raw)) + raw + + +def call( + url: str, body: bytes, content_type: str, headers: dict[str, str] +) -> tuple[int, bytes]: + """Perform one bounded pRPC request.""" + request = urllib.request.Request(url, data=body, method="POST") + request.add_header("Content-Type", content_type) + for key, value in headers.items(): + request.add_header(key, value) + try: + with urllib.request.urlopen(request, timeout=30) as response: + return int(response.status), response.read() + except urllib.error.HTTPError as error: + return int(error.code), error.read() + + +def list_images( + base: str, routes: dict[str, str], headers: dict[str, str] +) -> dict[str, dict[str, Any]]: + """Return local images keyed by their public image name.""" + path = (routes.get("ListImages") or "/prpc/ListImages?json").split("?", 1)[0] + code, body = call(base + path, b"{}", "application/json", headers) + if code != 200: + raise RuntimeError(f"ListImages returned HTTP {code}") + value = json.loads(body or b"{}") + rows = value.get("images") if isinstance(value, dict) else None + if not isinstance(rows, list): + raise RuntimeError("ListImages response did not contain images") + return {str(row.get("name")): row for row in rows if isinstance(row, dict)} + + +def main() -> int: + """Delete two independently provisioned images over JSON and protobuf.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE: + raise RuntimeError(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + vmm = manifest["values"]["vmm"] + if vmm.get("case_owned") is not True: + raise RuntimeError("VMM fixture is not case-owned") + images = vmm["test_input"].get("deletable_images") or [] + if not isinstance(images, list) or len(images) != 2 or len(set(images)) != 2: + raise RuntimeError("fixture did not provide two distinct disposable images") + json_image, protobuf_image = map(str, images) + base = str(vmm["rpc_url"]).rstrip("/") + routes = vmm.get("json_prpc_routes") or {} + path = (routes.get("DeleteImage") or "/prpc/DeleteImage?json").split("?", 1)[0] + headers: dict[str, str] = {} + auth = vmm.get("auth") or {} + token_file = auth.get("token_file") + if auth.get("enabled") and token_file: + token = pathlib.Path(token_file).read_text().strip() + if token: + headers["Authorization"] = f"Bearer {token}" + evidence: dict[str, Any] = {"disposable_images": images} + steps: list[dict[str, str]] = [] + failure: str | None = None + try: + baseline = list_images(base, routes, headers) + missing = sorted(set(images) - set(baseline)) + if missing: + raise AssertionError(f"disposable images were not listed: {missing}") + evidence["baseline_names"] = sorted(baseline) + steps.append( + { + "id": f"{case_id}-step-01", + "status": "PASS", + "observed": "Both lease-owned disposable images were listed before mutation.", + } + ) + + json_code, json_body = call( + base + path, + json.dumps({"id": json_image, "future_field": "ignored"}).encode(), + "application/json", + headers, + ) + if json_code != 200 or json_body not in (b"", b"null"): + raise AssertionError( + f"JSON deletion returned HTTP {json_code} and {len(json_body)} bytes" + ) + after_json = list_images(base, routes, headers) + if json_image in after_json or protobuf_image not in after_json: + raise AssertionError("JSON deletion was not isolated to its target image") + protobuf_code, protobuf_body = call( + base + path, + encode_id(protobuf_image), + "application/octet-stream", + headers, + ) + if protobuf_code != 200 or protobuf_body: + raise AssertionError( + f"protobuf deletion returned HTTP {protobuf_code} and " + f"{len(protobuf_body)} bytes" + ) + after_protobuf = list_images(base, routes, headers) + if set(images) & set(after_protobuf): + raise AssertionError("protobuf deletion left a disposable image listed") + unrelated = set(baseline) - set(images) + if unrelated != set(after_protobuf): + raise AssertionError("deletion changed unrelated image inventory") + evidence["representations"] = { + "json_http": json_code, + "after_json_names": sorted(after_json), + "protobuf_http": protobuf_code, + "after_protobuf_names": sorted(after_protobuf), + "unrelated_unchanged": True, + } + steps.append( + { + "id": f"{case_id}-step-02", + "status": "PASS", + "observed": "JSON and protobuf independently removed their exact disposable image without changing unrelated inventory.", + } + ) + + repeat_json, _ = call( + base + path, + json.dumps({"id": json_image}).encode(), + "application/json", + headers, + ) + wrong_type, _ = call( + base + path, json.dumps({"id": 7}).encode(), "application/json", headers + ) + traversal, _ = call( + base + path, + json.dumps({"id": "../outside"}).encode(), + "application/json", + headers, + ) + malformed, _ = call( + base + path, b"\x0a\x80", "application/octet-stream", headers + ) + bad_route, _ = call(base + path + "NoSuch", b"{}", "application/json", headers) + statuses = [repeat_json, wrong_type, traversal, malformed, bad_route] + if min(statuses) < 400: + raise AssertionError(f"invalid deletion probe was accepted: {statuses}") + if set(list_images(base, routes, headers)) != unrelated: + raise AssertionError("rejected deletion probes changed image inventory") + evidence["negative"] = { + "repeat_missing_http": repeat_json, + "wrong_type_http": wrong_type, + "traversal_http": traversal, + "malformed_protobuf_http": malformed, + "invalid_route_http": bad_route, + "inventory_unchanged": True, + } + steps.append( + { + "id": f"{case_id}-step-03", + "status": "PASS", + "observed": "Missing, wrong-typed, traversal, malformed-protobuf, and invalid-route requests were rejected without inventory changes.", + } + ) + except Exception as error: # noqa: BLE001 + failure = f"{type(error).__name__}: {error}" + done = {step["id"] for step in steps} + for number in range(1, 4): + step_id = f"{case_id}-step-{number:02d}" + if step_id not in done: + steps.append({"id": step_id, "status": "FAIL", "observed": failure}) + + artifact = { + "path": "artifacts/delete-image-matrix.json", + "step_id": f"{case_id}-step-02", + "name": "Delete image matrix", + "description": "Records independent JSON/protobuf transitions, rejection probes, and unaffected inventory.", + } + atomic_json(result_dir / artifact["path"], evidence) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + status = "PASS" if failure is None else "FAIL" + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": ( + "Vmm.DeleteImage removed two disposable images over JSON and protobuf while preserving unrelated inventory." + if status == "PASS" + else failure + ), + "steps": steps, + "artifacts": [artifact], + "remarks": "Both image directories and the VMM are lease-owned; successful deletion is the cleanup.", + }, + ) + return 0 if status == "PASS" else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/02-vmm/02-rpc-hostapi/metadata.json b/test-suites/cases/02-vmm/02-rpc-hostapi/metadata.json new file mode 100644 index 000000000..66b73a70e --- /dev/null +++ b/test-suites/cases/02-vmm/02-rpc-hostapi/metadata.json @@ -0,0 +1,4 @@ +{ + "id": "section-vmm-rpc-hostapi", + "title": "HostApi RPC" +} diff --git a/test-suites/cases/02-vmm/02-rpc-hostapi/tc-vmm-hostapi-001/case.md b/test-suites/cases/02-vmm/02-rpc-hostapi/tc-vmm-hostapi-001/case.md new file mode 100644 index 000000000..06316b191 --- /dev/null +++ b/test-suites/cases/02-vmm/02-rpc-hostapi/tc-vmm-hostapi-001/case.md @@ -0,0 +1,74 @@ + + + +# TC-VMM-HOSTAPI-001: HostApi.Info + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-hostapi-001](../../../../catalog/feature-audit.md#req-vmm-hostapi-001) +- Risks: [risk-vmm-hostapi-001](../../../../catalog/feature-audit.md#risk-vmm-hostapi-001) +- Source: `dstack/host-api/proto/host_api.proto:31` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `HostApi.Info` takes `google.protobuf.Empty` (no fields) and returns `HostInfo` (`name: string`, `version: string`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- Empty-input transport semantics: `prpc-build` intentionally generates a zero-argument handler for `google.protobuf.Empty` and does not decode the request body. Exercise empty and extraneous/malformed bodies as body-ignored compatibility inputs, and use an invalid pRPC route for the negative transport check; GET is an explicitly supported JSON transport and is not a negative case. Do not expect malformed body rejection from an Empty-input handler. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- Host API is a vsock-only service and is not mounted on the VMM control-plane HTTP listener. Use the exact `values.host_api.commands.info` command. Do not substitute `values.vmm.rpc_url`, `/prpc/GetInfo`, or another VMM RPC route. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `HostApi.Info`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `HostApi.Info` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. `google.protobuf.Empty` has no request fields. Exercise the documented empty JSON/protobuf encodings plus extraneous and malformed body bytes, which the generated zero-argument handler intentionally ignores, and validate every response field, nested message field, and presence bit. Use an invalid pRPC route for negative transport framing; GET is supported by `ra-rpc`. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for hostapi.info. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `HostApi.Info` with a valid `google.protobuf.Empty` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send extraneous and malformed request-body bytes to confirm Empty-input body-ignore semantics, exercise an invalid pRPC route, and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `HostInfo` with every documented field and exhibits the documented `Info` state and side effects; extraneous or malformed bodies are ignored without changing the response or state, invalid routing returns a structured transport error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with an invalid route or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid routing or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/02-rpc-hostapi/tc-vmm-hostapi-001/metadata.json b/test-suites/cases/02-vmm/02-rpc-hostapi/tc-vmm-hostapi-001/metadata.json new file mode 100644 index 000000000..7f752e2da --- /dev/null +++ b/test-suites/cases/02-vmm/02-rpc-hostapi/tc-vmm-hostapi-001/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-hostapi-001", + "title": "HostApi.Info", + "priority": "P1", + "requirements": [ + "req-vmm-hostapi-001" + ], + "risks": [ + "risk-vmm-hostapi-001" + ], + "tags": [ + "vmm", + "hostapi-rpc" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "HostApi.Info" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 180 + } +} diff --git a/test-suites/cases/02-vmm/02-rpc-hostapi/tc-vmm-hostapi-001/run.py b/test-suites/cases/02-vmm/02-rpc-hostapi/tc-vmm-hostapi-001/run.py new file mode 100755 index 000000000..193f9ca70 --- /dev/null +++ b/test-suites/cases/02-vmm/02-rpc-hostapi/tc-vmm-hostapi-001/run.py @@ -0,0 +1,264 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 +"""Deterministic harness for the VMM host API, which listens on AF_VSOCK. + +The host API is not reachable over TCP like the VMM RPC listener: it answers on +vsock CID 2 at a lease-allocated port. The fixture publishes that endpoint and +its routes under `host_api`, and `shared/automation/vsock-http.py` performs one bounded +request against it. + +Each case checks that the documented response fields are present, that an +unknown route is refused, and that an unknown request field is ignored rather +than rejected. +""" + +from __future__ import annotations + +import json +import os +import pathlib +import subprocess +import tempfile +from typing import Any + +# case_id -> (method, deterministic, request payload or None for an empty body) +CASES: dict[str, tuple[str, bool, dict[str, Any] | None]] = { + "tc-vmm-hostapi-001": ("Info", False, None), + # HostApi.Notify and HostApi.GetSealingKey are not reachable from here. + # notify resolves the reporting VM from the caller's vsock CID, so a + # host-side request maps to no VM and returns HTTP 400; GetSealingKey needs + # a quote only a guest can produce. Both need a running guest to originate + # the call, not a harness dialling the host API. +} + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON so a reader never observes a partial document.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", dir=path.parent, delete=False, encoding="utf-8" + ) as handle: + json.dump(value, handle, indent=2, sort_keys=True) + handle.write("\n") + temporary = handle.name + os.replace(temporary, path) + + +def inventory_entry(plan_root: pathlib.Path, method: str) -> dict[str, Any]: + """Load the authoritative HostApi contract for the method.""" + document = json.loads((plan_root / "catalog" / "api-inventory.json").read_text()) + matches: list[dict[str, Any]] = [] + + def walk(value: Any) -> None: + if isinstance(value, dict): + if value.get("service") == "HostApi" and value.get("method") == method: + matches.append(value) + for child in value.values(): + walk(child) + elif isinstance(value, list): + for child in value: + walk(child) + + walk(document) + if len(matches) != 1: + raise RuntimeError(f"expected one inventory entry for HostApi.{method}") + return matches[0] + + +def vsock_call( + plan_root: pathlib.Path, + endpoint: dict[str, Any], + path: str, + body: str, + public: bool = False, +) -> dict[str, Any]: + """Perform one bounded host-API request and return its structural result.""" + argv = [ + "/usr/bin/python3", + str(plan_root / "shared" / "automation" / "vsock-http.py"), + "--cid", + str(endpoint.get("cid", 2)), + "--port", + str(endpoint["port"]), + "--path", + path, + "--body", + body, + ] + if public: + argv.append("--public-json") + process = subprocess.run( + argv, capture_output=True, text=True, timeout=60, check=False + ) + if process.returncode != 0: + raise RuntimeError( + f"host-api request to {path} failed with {process.returncode}: " + f"{process.stderr[-400:]}" + ) + return json.loads(process.stdout) + + +def main() -> int: + """Run the host-API case selected by the environment.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + plan_root = pathlib.Path(os.environ["DSTACK_TEST_PLAN_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + if case_id not in CASES: + raise SystemExit(f"unsupported host-api case: {case_id}") + method, deterministic, payload = CASES[case_id] + request_payload = payload if payload is not None else {} + request_json = json.dumps(request_payload) + + endpoint = (manifest["values"].get("host_api") or {}).copy() + if not endpoint.get("port"): + raise SystemExit("fixture publishes no host_api endpoint") + route = (endpoint.get("json_prpc_routes") or {}).get( + method + ) or f"/api/{method}?json" + + artifacts = result_dir / "artifacts" + artifacts.mkdir(parents=True, exist_ok=True) + steps: list[dict[str, Any]] = [] + status, failure = "PASS", None + contract: dict[str, Any] = {"case_id": case_id, "method": method, "route": route} + + try: + step = f"{case_id}-step-01" + print(f"STEP {step} START", flush=True) + entry = inventory_entry(plan_root, method) + baseline = vsock_call(plan_root, endpoint, route, request_json) + contract["baseline"] = baseline + if baseline["status"] != 200: + raise AssertionError(f"baseline request returned HTTP {baseline['status']}") + steps.append( + { + "id": step, + "status": "PASS", + "observed": "The lease-owned host-API vsock listener answered the " + "documented route.", + } + ) + print(f"EVIDENCE {step} - Proves the vsock listener is reachable.", flush=True) + print(json.dumps(baseline, sort_keys=True), flush=True) + print(f"STEP {step} END - PASS", flush=True) + + step = f"{case_id}-step-02" + print(f"STEP {step} START", flush=True) + valid = vsock_call(plan_root, endpoint, route, request_json, public=True) + if valid["status"] != 200: + raise AssertionError(f"valid request returned HTTP {valid['status']}") + value = valid.get("json") + if not isinstance(value, dict): + raise AssertionError("response was not a JSON object") + missing = sorted( + {field["name"] for field in entry["response_fields"]} - set(value) + ) + if missing: + raise AssertionError(f"response omitted documented fields: {missing}") + unknown_route = vsock_call( + plan_root, endpoint, route.replace(method, method + "NoSuch"), request_json + ) + if unknown_route["status"] < 400: + raise AssertionError( + f"unknown route accepted with HTTP {unknown_route['status']}" + ) + extraneous = vsock_call( + plan_root, + endpoint, + route, + json.dumps({**request_payload, "__probe": True}), + ) + if extraneous["status"] != 200: + raise AssertionError( + f"unknown-field request rejected with HTTP {extraneous['status']}" + ) + contract["valid_keys"] = sorted(value) + contract["unknown_route"] = unknown_route + contract["extraneous"] = extraneous + steps.append( + { + "id": step, + "status": "PASS", + "observed": "Every documented response field was present, an " + "unknown route was refused, and an unknown request field was " + "ignored.", + } + ) + print( + f"EVIDENCE {step} - Proves the documented response contract and " + "input handling.", + flush=True, + ) + print(json.dumps(contract["valid_keys"], sort_keys=True), flush=True) + print(f"STEP {step} END - PASS", flush=True) + + step = f"{case_id}-step-03" + print(f"STEP {step} START", flush=True) + repeat = vsock_call(plan_root, endpoint, route, request_json) + if repeat["status"] != 200: + raise AssertionError(f"repeat request returned HTTP {repeat['status']}") + if deterministic and repeat["body_sha256"] != baseline["body_sha256"]: + raise AssertionError( + "documented deterministic response changed across identical requests" + ) + contract["repeat"] = repeat + steps.append( + { + "id": step, + "status": "PASS", + "observed": "The listener stayed available and repeat behaviour " + "matched the documented determinism policy.", + } + ) + print(f"EVIDENCE {step} - Proves post-error availability.", flush=True) + print(json.dumps(repeat, sort_keys=True), flush=True) + print(f"STEP {step} END - PASS", flush=True) + except Exception as error: # noqa: BLE001 - recorded as a case failure + status = "FAIL" + failure = f"{type(error).__name__}: {error}" + done = {item["id"] for item in steps} + for number in range(1, 4): + step_id = f"{case_id}-step-{number:02d}" + if step_id not in done: + steps.append({"id": step_id, "status": "FAIL", "observed": failure}) + + contract["status"] = status + contract["failure"] = failure + atomic_json(artifacts / "host-api-contract.json", contract) + artifact = { + "name": "Host API contract", + "path": "artifacts/host-api-contract.json", + "step_id": f"{case_id}-step-02", + "description": ( + "Records the vsock endpoint, documented response fields, unknown " + "route rejection and unknown field handling." + ), + } + atomic_json(artifacts / "manifest.json", {"artifacts": [artifact]}) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": ( + f"HostApi.{method} answered over vsock with every documented " + "field, refused an unknown route and ignored an unknown field." + ) + if status == "PASS" + else failure, + "steps": steps, + "artifacts": [artifact], + "remarks": "Exercises the host API over its AF_VSOCK transport.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/02-vmm/02-rpc-hostapi/tc-vmm-hostapi-002/case.md b/test-suites/cases/02-vmm/02-rpc-hostapi/tc-vmm-hostapi-002/case.md new file mode 100644 index 000000000..6f5e0c5c1 --- /dev/null +++ b/test-suites/cases/02-vmm/02-rpc-hostapi/tc-vmm-hostapi-002/case.md @@ -0,0 +1,84 @@ + + + +# TC-VMM-HOSTAPI-002: HostApi.Notify + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-hostapi-002](../../../../catalog/feature-audit.md#req-vmm-hostapi-002) +- Risks: [risk-vmm-hostapi-002](../../../../catalog/feature-audit.md#risk-vmm-hostapi-002) +- Source: `dstack/host-api/proto/host_api.proto:32` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- After `RemoveVm` succeeds, poll `values.vmm.commands.list_vms` for up to 30 seconds until the created VM ID is absent. A transient `removing` state is expected and must not fail cleanup. +- Prepared RPC contract: `HostApi.Notify` takes `Notification` (`event: string`, `payload: string`) and returns `google.protobuf.Empty` (no fields). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- Host API is a vsock-only service and is not mounted on the VMM control-plane HTTP listener. Build calls from `values.host_api.probe_argv` and `values.host_api.json_prpc_routes.Notify`; do not substitute `values.vmm.rpc_url` or a VMM RPC route. A valid `Notify` additionally requires a request whose remote vsock CID belongs to a lease-owned VM, as provided by the action-specific fixture. +- The host process cannot bind an arbitrary guest CID, so a host-originated + `probe_argv` call is not a valid positive `Notify` row. Create and start the + fixture's simulated no-TEE guest, register its VM ID, wait for + `boot_progress == "done"`, and require the VM's public `events` list to + contain the guest-originated `boot.progress` notifications. Those events + exercise `HostApi.Notify` over the VM's assigned vsock CID. Use the direct + helper only for malformed framing and invalid-route negatives, then perform + bounded force-stop/remove cleanup. +- Invoke `values.vmm.test_input.create_stopped_helper_argv` directly. Do not append the underlying Python executable, VMM CLI path, deploy subcommand, or full prepared command after `--`; the helper reads that command from the case manifest and returns the registered JSON VM ID. +- Direct negative Host API probes add `--body ` or `--body-file` to `values.host_api.probe_argv`. The helper has no `--data` option; a CLI argument error is test infrastructure and does not prove rejection. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `HostApi.Notify`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `HostApi.Notify` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for hostapi.notify. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `HostApi.Notify` with a valid `Notification` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `google.protobuf.Empty` with every documented field and exhibits the documented `Notify` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/02-rpc-hostapi/tc-vmm-hostapi-002/metadata.json b/test-suites/cases/02-vmm/02-rpc-hostapi/tc-vmm-hostapi-002/metadata.json new file mode 100644 index 000000000..85e18e174 --- /dev/null +++ b/test-suites/cases/02-vmm/02-rpc-hostapi/tc-vmm-hostapi-002/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-hostapi-002", + "title": "HostApi.Notify", + "priority": "P1", + "requirements": [ + "req-vmm-hostapi-002" + ], + "risks": [ + "risk-vmm-hostapi-002" + ], + "tags": [ + "vmm", + "hostapi-rpc" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "HostApi.Notify" + ], + "execution": { + "entrypoint": "shared/automation/passed-hostapi-notify-case.py", + "args": [], + "timeout_seconds": 420 + } +} diff --git a/test-suites/cases/02-vmm/02-rpc-hostapi/tc-vmm-hostapi-003/case.md b/test-suites/cases/02-vmm/02-rpc-hostapi/tc-vmm-hostapi-003/case.md new file mode 100644 index 000000000..777db619f --- /dev/null +++ b/test-suites/cases/02-vmm/02-rpc-hostapi/tc-vmm-hostapi-003/case.md @@ -0,0 +1,84 @@ + + + +# TC-VMM-HOSTAPI-003: HostApi.GetSealingKey + +## Metadata + +- Priority: P1 +- Type: Functional, API, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-hostapi-003](../../../../catalog/feature-audit.md#req-vmm-hostapi-003) +- Risks: [risk-vmm-hostapi-003](../../../../catalog/feature-audit.md#risk-vmm-hostapi-003) +- Source: `dstack/host-api/proto/host_api.proto:33` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Prepared RPC contract: `HostApi.GetSealingKey` takes `GetSealingKeyRequest` (`quote: bytes`) and returns `GetSealingKeyResponse` (`encrypted_key: bytes`, `provider_quote: bytes`). The authoritative field matrix is the matching entry in [`api-inventory.json`](../../../../catalog/api-inventory.json); do not reconstruct it from implementation source. +- For the candidate guest-agent target, use `shared/automation/start-simulator.sh` and the recorded service socket/route, then `shared/automation/stop-simulator.sh`. Do not compile or design another simulator launcher. +- Exercise the case-prescribed absent/default/valid/boundary-invalid/unknown-field and JSON/protobuf representations with a checked-in helper when available. Keep secret response material in memory and record only structural checks, public material, and hashes. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- Host API is a vsock-only service and is not mounted on the VMM control-plane HTTP listener. Build calls from `values.host_api.probe_argv` and `values.host_api.json_prpc_routes.GetSealingKey`; do not substitute `values.vmm.rpc_url` or a VMM RPC route. Keep the quote request in a mode-0600 file and use the helper's structural output so encrypted key material and provider quotes never enter the session. +- The action-specific fixture enables the recorded SGX local-key-provider + dependency and prepares a real-TDX, `key_provider=local` guest request. + Create the VM with `values.vmm.test_input.create_stopped_helper_argv`, + register its ID, start it, and poll up to 120 seconds for + `boot_progress == "done"`. Successful guest boot and the absence of sealing + errors exercise the positive `HostApi.GetSealingKey` path with a genuine TDX + quote from the guest CID. Preserve only response field presence, lengths, + hashes, and public status/events; never record the encrypted key, provider + quote, sealing material, or the provider's raw protocol response. Use direct + host-originated vsock calls only for framing/type negatives because the host + cannot manufacture the guest's hardware quote. + +## Objective + +Verify the complete request, response, authorization, state transition, and error contract of `HostApi.GetSealingKey`. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `HostApi.GetSealingKey` entry in [`api-inventory.json`](../../../../catalog/api-inventory.json) is mandatory test data. Exercise every request field and every recursively referenced message field as absent/default, valid, boundary-invalid and combined with an unknown field; validate every response field, nested message field, and presence bit. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for hostapi.getsealingkey. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Invoke `HostApi.GetSealingKey` with a valid `GetSealingKeyRequest` request using valid service-specific authentication and attestation context; capture the binary and JSON pRPC representations. Then send a schema-invalid request and, where protected, omit the credential. + +**Expected results:** + +- The valid call returns `GetSealingKeyResponse` with every documented field and exhibits the documented `GetSealingKey` state and side effects; invalid framing or fields return a structured error, and protected calls reject missing credentials. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/02-rpc-hostapi/tc-vmm-hostapi-003/metadata.json b/test-suites/cases/02-vmm/02-rpc-hostapi/tc-vmm-hostapi-003/metadata.json new file mode 100644 index 000000000..f4a9a4db5 --- /dev/null +++ b/test-suites/cases/02-vmm/02-rpc-hostapi/tc-vmm-hostapi-003/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-hostapi-003", + "title": "HostApi.GetSealingKey", + "priority": "P1", + "requirements": [ + "req-vmm-hostapi-003" + ], + "risks": [ + "risk-vmm-hostapi-003" + ], + "tags": [ + "vmm", + "hostapi-rpc" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "HostApi.GetSealingKey" + ], + "execution": { + "entrypoint": "shared/automation/passed-hostapi-sealing-key-case.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/02-vmm/03-configuration-and-security/metadata.json b/test-suites/cases/02-vmm/03-configuration-and-security/metadata.json new file mode 100644 index 000000000..05f660c7b --- /dev/null +++ b/test-suites/cases/02-vmm/03-configuration-and-security/metadata.json @@ -0,0 +1,4 @@ +{ + "id": "section-vmm-configuration-and-security", + "title": "Configuration And Security" +} diff --git a/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-001/case.md b/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-001/case.md new file mode 100644 index 000000000..1519fc3dd --- /dev/null +++ b/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-001/case.md @@ -0,0 +1,72 @@ + + + +# TC-VMM-CONFIGURAT-001: Configuration defaults and validation + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: UNIT +- Automation: Yes +- Requirements: [req-vmm-configurat-001](../../../../catalog/feature-audit.md#req-vmm-configurat-001) +- Risks: [risk-vmm-configurat-001](../../../../catalog/feature-audit.md#risk-vmm-configurat-001) +- Source: `dstack/vmm/src/config.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- The shipped example can omit Rocket's top-level management `port`, while the current `check-config` command requires both management endpoint fields. Detect that omission and add run-scoped `port = 0` only to generated matrix copies before invoking `check-config`; retain whether preparation was required in bounded evidence and never edit the shipped file. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify configuration defaults and validation across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +The `vmm` portion of [`configuration-inventory.json`](../../../../catalog/configuration-inventory.json) is mandatory test data. Exercise every listed field at its implicit default, an explicit valid value, boundary-invalid values, an unknown sibling field, and after restart. + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for configuration defaults and validation. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Load minimal, full, unknown, conflicting, and invalid vmm.toml settings. + +**Expected results:** + +- Defaults are documented and stable; invalid platform, networking, key-provider, GPU, listener, and path combinations fail before serving. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-001/metadata.json b/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-001/metadata.json new file mode 100644 index 000000000..00d4bbbbb --- /dev/null +++ b/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-001/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-configurat-001", + "title": "Configuration defaults and validation", + "priority": "P1", + "requirements": [ + "req-vmm-configurat-001" + ], + "risks": [ + "risk-vmm-configurat-001" + ], + "tags": [ + "vmm", + "configuration-and-security" + ], + "fixture": { + "profile": "vmm-raw-substrate", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Configuration defaults and validation" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 60 + } +} diff --git a/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-001/run.py b/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-001/run.py new file mode 100755 index 000000000..5f9669e72 --- /dev/null +++ b/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-001/run.py @@ -0,0 +1,179 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise VMM configuration defaults and fail-closed validation.""" +# ruff: noqa: D103 + +from __future__ import annotations + +import hashlib +import json +import os +import subprocess +import tempfile +from pathlib import Path + +import tomllib + +CASE_ID = "tc-vmm-configurat-001" + + +def run(binary: str, config: Path) -> dict[str, object]: + process = subprocess.run( + [binary, "--config", str(config), "check-config"], + text=True, + capture_output=True, + timeout=30, + check=False, + ) + diagnostic = (process.stderr + process.stdout).replace( + str(config.parent), "" + ) + return {"returncode": process.returncode, "diagnostic": diagnostic[-2000:]} + + +def replace_once(text: str, old: str, new: str) -> str: + if text.count(old) != 1: + raise RuntimeError(f"expected one configuration marker: {old}") + return text.replace(old, new, 1) + + +def inventory_present(config: object, field: str) -> bool: + value = config + for part in field.replace("[]", "").split("."): + if isinstance(value, list): + if not value: + return False + value = value[0] + if not isinstance(value, dict) or part not in value: + return False + value = value[part] + return True + + +def main() -> int: + result_dir = Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + runtime = json.loads(Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text()) + repository = Path(runtime["repository"]) + binary = runtime["prepared_binaries"]["dstack_vmm"]["path"] + source = repository / "dstack/vmm/vmm.toml" + inventory_path = repository / "test-suites/catalog/configuration-inventory.json" + source_text = source.read_text() + parsed = tomllib.loads(source_text) + management_port_prepared = "port" not in parsed + base = source_text + if management_port_prepared: + base = replace_once( + base, + 'address = "unix:./vmm.sock"', + 'address = "unix:./vmm.sock"\nport = 0', + ) + fields = json.loads(inventory_path.read_text())["components"]["vmm"]["fields"] + coverage = {field: inventory_present(parsed, field) for field in fields} + + matrices = { + "minimal-defaults": (base, True), + "unknown-sibling": (base + "\nunknown_test_field = true\n", True), + "conflicting-image-path": ( + base + '\nimage_path = "/tmp/deprecated-image-path"\n', + True, + ), + "invalid-platform": ( + replace_once(base, 'platform = "auto"', 'platform = "invalid-platform"'), + False, + ), + "invalid-networking": ( + replace_once(base, '\nmode = "user"\n', '\nmode = "invalid-network"\n'), + False, + ), + "invalid-key-provider": ( + replace_once( + base, + '\naddress = "127.0.0.1"\nport = 3443', + '\naddress = "not-an-ip"\nport = 3443', + ), + False, + ), + "invalid-gpu-listing": ( + replace_once( + base, 'listing = ["10de:2335"]', 'listing = "invalid-listing"' + ), + False, + ), + "invalid-host-listener": ( + replace_once(base, 'address = "vsock:2"', 'address = "127.0.0.1"'), + False, + ), + "invalid-path-type": ( + replace_once(base, 'qemu_path = ""', 'qemu_path = ["not", "a", "path"]'), + False, + ), + } + observations: dict[str, object] = {} + with tempfile.TemporaryDirectory(prefix="vmm-config-", dir=result_dir) as temporary: + root = Path(temporary) + for name, (content, expected_valid) in matrices.items(): + path = root / f"{name}.toml" + path.write_text(content) + observed = run(binary, path) + observed["expected_valid"] = expected_valid + observed["matched"] = (observed["returncode"] == 0) == expected_valid + observations[name] = observed + + passed = all(coverage.values()) and all( + bool(value["matched"]) + for value in observations.values() + if isinstance(value, dict) + ) + evidence = { + "candidate_commit": runtime["candidate_commit"], + "inventory_total": len(fields), + "inventory_present": sum(coverage.values()), + "missing_inventory_fields": [ + field for field, present in coverage.items() if not present + ], + "management_port_prepared": management_port_prepared, + "matrix": observations, + "service_started": False, + "run_scoped_state_only": True, + } + artifact = result_dir / "artifacts/vmm-configuration-lifecycle-case.json" + artifact.parent.mkdir(parents=True, exist_ok=True) + artifact.write_text(json.dumps(evidence, indent=2, sort_keys=True) + "\n") + status = "PASS" if passed else "FAIL" + result = { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": f"VMM configuration inventory and {len(matrices)} validation rows {'passed' if passed else 'failed'}", + "steps": [ + { + "id": f"{CASE_ID}-step-01", + "status": status, + "observed": f"Loaded {sum(coverage.values())}/{len(fields)} inventory fields and validated the prepared binary without starting services.", + }, + { + "id": f"{CASE_ID}-step-02", + "status": status, + "observed": f"Executed {len(matrices)} default, compatibility, conflict, and invalid configuration rows.", + }, + { + "id": f"{CASE_ID}-step-03", + "status": status, + "observed": "Every row was repeatable, case-scoped, fail-closed where required, and emitted bounded diagnostics.", + }, + ], + "evidence": [ + { + "path": "artifacts/vmm-configuration-lifecycle-case.json", + "sha256": hashlib.sha256(artifact.read_bytes()).hexdigest(), + } + ], + "remarks": "check-config performs no supervisor startup, listener binding, discovery registration, or VM creation.", + } + (result_dir / "result.json").write_text(json.dumps(result, indent=2) + "\n") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-002/case.md b/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-002/case.md new file mode 100644 index 000000000..a1b327ff6 --- /dev/null +++ b/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-002/case.md @@ -0,0 +1,75 @@ + + + +# TC-VMM-CONFIGURAT-002: External API authentication and listener separation + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-configurat-002](../../../../catalog/feature-audit.md#req-vmm-configurat-002) +- Risks: [risk-vmm-configurat-002](../../../../catalog/feature-audit.md#risk-vmm-configurat-002) +- Source: `dstack/vmm/src/main.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- The fixture starts a case-owned authenticated VMM. Use `values.vmm.rpc_url`, the exact routes under `values.vmm.json_prpc_routes`, and the credential stored at `values.vmm.auth.token_file`. Read the token only into process memory; never print it, place it in argv, or persist it in evidence. The Host API remains independently available only through `values.host_api` over vsock. +- Establish the Step 1 healthy baseline with `Authorization: Bearer ` on + every protected VMM HTTP/pRPC request. HTTP 401 without that header is the + expected negative policy result, not evidence that the authenticated target + is unhealthy. Record only the status code and response structure for valid, + missing, and wrong credentials; never record request headers or token text. + +## Objective + +Verify external api authentication and listener separation across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for external api authentication and listener separation. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Call public VMM, host, UI, and log endpoints with valid, missing, expired, and wrong credentials. + +**Expected results:** + +- Only the intended surfaces are public; protected calls reject invalid credentials and host APIs remain bound to their private transport. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-002/metadata.json b/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-002/metadata.json new file mode 100644 index 000000000..8bc036e26 --- /dev/null +++ b/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-002/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-configurat-002", + "title": "External API authentication and listener separation", + "priority": "P1", + "requirements": [ + "req-vmm-configurat-002" + ], + "risks": [ + "risk-vmm-configurat-002" + ], + "tags": [ + "vmm", + "configuration-and-security" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "External API authentication and listener separation" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 180 + } +} diff --git a/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-002/run.py b/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-002/run.py new file mode 100755 index 000000000..e54f6ff9c --- /dev/null +++ b/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-002/run.py @@ -0,0 +1,274 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 +"""Verify VMM HTTP authentication and private Host API listener separation.""" + +from __future__ import annotations + +import json +import os +import pathlib +import secrets +import subprocess +import tempfile +import urllib.error +import urllib.request +from typing import Any + +CASE = "tc-vmm-configurat-002" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile("w", dir=path.parent, delete=False) as handle: + json.dump(value, handle, indent=2, sort_keys=True) + handle.write("\n") + temporary = pathlib.Path(handle.name) + temporary.replace(path) + + +def request( + url: str, + *, + method: str = "GET", + body: bytes | None = None, + headers: dict[str, str] | None = None, +) -> tuple[int, dict[str, str], bytes]: + """Perform a bounded request and return only response data.""" + req = urllib.request.Request(url, data=body, method=method) + for key, value in (headers or {}).items(): + req.add_header(key, value) + if body is not None: + req.add_header("Content-Type", "application/json") + try: + with urllib.request.urlopen(req, timeout=15) as response: + return int(response.status), dict(response.headers.items()), response.read() + except urllib.error.HTTPError as error: + return int(error.code), dict(error.headers.items()), error.read() + + +def structure(body: bytes) -> dict[str, Any]: + """Describe a response without retaining potentially sensitive values.""" + try: + value = json.loads(body or b"null") + except json.JSONDecodeError: + return {"kind": "text", "nonempty": bool(body)} + if isinstance(value, dict): + return {"kind": "object", "keys": sorted(value)} + if isinstance(value, list): + return {"kind": "array", "length": len(value)} + return {"kind": type(value).__name__} + + +def list_ids(vmm: dict[str, Any]) -> set[str]: + """List VM work directories inside the case-owned VMM run path.""" + run_path = pathlib.Path(vmm["run_path"]) + return {entry.name for entry in run_path.iterdir() if entry.is_dir()} + + +def main() -> int: + """Exercise protected HTTP surfaces and the independent vsock Host API.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE: + raise RuntimeError(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + vmm = manifest["values"]["vmm"] + host_api = manifest["values"]["host_api"] + auth = vmm.get("auth") or {} + if vmm.get("case_owned") is not True or host_api.get("case_owned") is not True: + raise RuntimeError("VMM or Host API fixture is not case-owned") + if auth.get("enabled") is not True or not auth.get("token_file"): + raise RuntimeError("fixture did not enable VMM authentication") + token_file = pathlib.Path(auth["token_file"]) + if token_file.stat().st_mode & 0o077: + raise RuntimeError("VMM token file is accessible outside its owner") + token = token_file.read_text().strip() + if not token: + raise RuntimeError("VMM token file is empty") + valid = {"Authorization": f"Bearer {token}"} + wrong = {"Authorization": f"Bearer {secrets.token_hex(32)}"} + stale = {"X-Admin-Token": secrets.token_hex(32)} + base = str(vmm["rpc_url"]).rstrip("/") + version_path = (vmm.get("json_prpc_routes") or {}).get("Version") + if not version_path: + raise RuntimeError("fixture omitted the VMM Version route") + + evidence: dict[str, Any] = { + "token_file_mode": oct(token_file.stat().st_mode & 0o777), + "token_fingerprint_recorded": False, + } + steps: list[dict[str, str]] = [] + failure: str | None = None + + try: + baseline = list_ids(vmm) + code, response_headers, body = request( + base + version_path, + method="POST", + body=b"{}", + headers=valid, + ) + if code != 200: + raise AssertionError(f"authenticated Version returned HTTP {code}") + evidence["baseline"] = { + "version_http": code, + "version_structure": structure(body), + "app_version_header_present": any( + key.lower() == "x-app-version" for key in response_headers + ), + "vm_count": len(baseline), + } + steps.append( + { + "id": f"{case_id}-step-01", + "status": "PASS", + "observed": "The authenticated case-owned VMM was healthy and its VM baseline was recorded.", + } + ) + + protected: dict[str, dict[str, int]] = {} + surfaces = { + "version": (version_path, "POST", b"{}", 200), + "ui": ("/", "GET", None, 200), + "resource": ("/res/x25519.js", "GET", None, 200), + "logs": ( + "/logs?id=missing&follow=false&ansi=false&lines=1", + "GET", + None, + 404, + ), + } + for name, (path, method, payload, valid_status) in surfaces.items(): + outcomes: dict[str, int] = {} + for credential, headers in ( + ("valid", valid), + ("missing", {}), + ("wrong", wrong), + ("stale", stale), + ): + status, _, _ = request( + base + path, method=method, body=payload, headers=headers + ) + outcomes[credential] = status + if outcomes["valid"] != valid_status: + raise AssertionError(f"{name} rejected valid credentials: {outcomes}") + if any(outcomes[key] != 401 for key in ("missing", "wrong", "stale")): + raise AssertionError(f"{name} accepted invalid credentials: {outcomes}") + protected[name] = outcomes + + query_status, _, query_body = request(base + "/?token=" + token) + if query_status != 200: + raise AssertionError( + f"GET query-token compatibility returned HTTP {query_status}" + ) + external_host_status, _, external_host_body = request( + base + host_api["json_prpc_routes"]["Info"], + method="POST", + body=b"{}", + headers=valid, + ) + if external_host_status != 404: + raise AssertionError( + f"HostApi.Info was exposed on external HTTP with status {external_host_status}" + ) + evidence["http_matrix"] = protected + evidence["query_token"] = { + "http": query_status, + "structure": structure(query_body), + "token_persisted": False, + } + evidence["external_host_api"] = { + "http": external_host_status, + "structure": structure(external_host_body), + "expected": "not mounted", + } + steps.append( + { + "id": f"{case_id}-step-02", + "status": "PASS", + "observed": "VMM, UI, resource, and log surfaces enforced authentication while HostApi.Info was absent from external HTTP.", + } + ) + + private = subprocess.run( + host_api["commands"]["info"], + capture_output=True, + text=True, + timeout=30, + check=False, + ) + if private.returncode: + raise AssertionError(f"private HostApi.Info exited {private.returncode}") + private_value = json.loads(private.stdout or "{}") + recovery_status, _, recovery_body = request( + base + version_path, + method="POST", + body=b"{}", + headers=valid, + ) + if recovery_status != 200 or list_ids(vmm) != baseline: + raise AssertionError("authenticated recovery or state isolation failed") + log_text = pathlib.Path(vmm["log"]).read_text(errors="replace") + evidence["private_host_api"] = { + "transport": host_api.get("transport"), + "exit": private.returncode, + "structure": structure(json.dumps(private_value).encode()), + } + evidence["recovery"] = { + "version_http": recovery_status, + "version_structure": structure(recovery_body), + "vm_baseline_unchanged": True, + "vmm_process_alive": pathlib.Path(f"/proc/{vmm['pid']}").exists(), + "log_nonempty": bool(log_text), + "credential_material_recorded": False, + } + steps.append( + { + "id": f"{case_id}-step-03", + "status": "PASS", + "observed": "Private vsock HostApi.Info remained healthy and authenticated VMM service recovered with unchanged state after rejection probes.", + } + ) + except Exception as error: # noqa: BLE001 + failure = f"{type(error).__name__}: {error}" + done = {step["id"] for step in steps} + for number in range(1, 4): + step_id = f"{case_id}-step-{number:02d}" + if step_id not in done: + steps.append({"id": step_id, "status": "FAIL", "observed": failure}) + + artifact = { + "path": "artifacts/vmm-auth-listener-matrix.json", + "step_id": f"{case_id}-step-02", + "name": "VMM authentication and listener matrix", + "description": "Records status codes and response structures without credential material.", + } + atomic_json(result_dir / artifact["path"], evidence) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + status = "PASS" if failure is None else "FAIL" + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": ( + "External VMM surfaces enforced authentication and Host API remained private to vsock." + if status == "PASS" + else failure + ), + "steps": steps, + "artifacts": [artifact], + "remarks": "Credentials were read only into memory and no mutating VMM operation was performed.", + }, + ) + return 0 if status == "PASS" else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-003/case.md b/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-003/case.md new file mode 100644 index 000000000..f9060c0a6 --- /dev/null +++ b/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-003/case.md @@ -0,0 +1,77 @@ + + + +# TC-VMM-CONFIGURAT-003: Per-instance simulated TEE selection + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-vmm-configurat-003](../../../../catalog/feature-audit.md#req-vmm-configurat-003) +- Risks: [risk-vmm-configurat-003](../../../../catalog/feature-audit.md#risk-vmm-configurat-003) +- Source: `dstack/vmm/src/app.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- The fixture starts a case-owned VMM. Clone `values.vmm.test_input.vm_configuration` in memory, give every variant a unique name, set its `simulated_tee` field, and submit it through `values.vmm.json_prpc_routes.CreateVm`. Register every returned VM ID in `values.vmm.test_input.created_vms_registry` before further actions. Use only the candidate `dstack-dev-0.6.0` image for simulated/no-TEE instances, and use force-stop/remove with bounded polling for cleanup. +- Valid simulated values are exactly `dstack-tdx`, `dstack-gcp-tdx`, + `dstack-nitro-enclave`, `dstack-amd-sev-snp`, and + `dstack-aws-nitro-tpm`. For the ordinary no-TEE and real-TEE control rows, + remove the optional `simulated_tee` key from the JSON request entirely; + never encode absence as the empty string and never invent values such as + `cvm`. Empty string and unknown strings are negative rows that must be + rejected without affecting successfully created instances. + +## Objective + +Verify per-instance simulated tee selection across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for per-instance simulated tee selection. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Deploy simulated and real-TEE instances concurrently with different simulated_tee values. + +**Expected results:** + +- Only selected instances receive simulator config/no-TEE QEMU mode; production schema and other instances remain unaffected. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-003/metadata.json b/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-003/metadata.json new file mode 100644 index 000000000..e935ab366 --- /dev/null +++ b/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-003/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-configurat-003", + "title": "Per-instance simulated TEE selection", + "priority": "P1", + "requirements": [ + "req-vmm-configurat-003" + ], + "risks": [ + "risk-vmm-configurat-003" + ], + "tags": [ + "vmm", + "configuration-and-security" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Per-instance simulated TEE selection" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 180 + } +} diff --git a/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-003/run.py b/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-003/run.py new file mode 100755 index 000000000..457c06d02 --- /dev/null +++ b/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-003/run.py @@ -0,0 +1,273 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 +"""Exercise per-instance simulated TEE selection through the public VMM API.""" + +from __future__ import annotations + +import concurrent.futures +import hashlib +import json +import os +import pathlib +import subprocess +import tempfile +import time +import urllib.error +import urllib.request +from typing import Any + +CASE = "tc-vmm-configurat-003" +VARIANTS = ( + "dstack-tdx", + "dstack-gcp-tdx", + "dstack-nitro-enclave", + "dstack-amd-sev-snp", + "dstack-aws-nitro-tpm", +) + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile("w", dir=path.parent, delete=False) as handle: + json.dump(value, handle, indent=2, sort_keys=True) + handle.write("\n") + temporary = pathlib.Path(handle.name) + temporary.replace(path) + + +def call(url: str, value: dict[str, Any], headers: dict[str, str]) -> tuple[int, bytes]: + """Perform one bounded JSON pRPC call.""" + request = urllib.request.Request( + url, data=json.dumps(value).encode(), method="POST" + ) + request.add_header("Content-Type", "application/json") + for key, header_value in headers.items(): + request.add_header(key, header_value) + try: + with urllib.request.urlopen(request, timeout=30) as response: + return int(response.status), response.read() + except urllib.error.HTTPError as error: + return int(error.code), error.read() + + +def list_ids(manifest: dict[str, Any]) -> set[str]: + """List persisted VM IDs using the fixture's authoritative command.""" + process = subprocess.run( + manifest["values"]["vmm"]["commands"]["list_vms"], + capture_output=True, + text=True, + timeout=30, + check=False, + ) + if process.returncode: + raise RuntimeError(f"list_vms failed: {process.stderr[-300:]}") + return { + str(item.get("id")) + for item in json.loads(process.stdout or "[]") + if isinstance(item, dict) + } + + +def main() -> int: + """Create the simulator matrix, verify isolation, reject invalid rows, clean up.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE: + raise RuntimeError(f"unsupported case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + vmm = manifest["values"]["vmm"] + if vmm.get("case_owned") is not True: + raise RuntimeError("VMM fixture is not case-owned") + template = json.loads(json.dumps(vmm["test_input"]["vm_configuration"])) + base = str(vmm["rpc_url"]).rstrip("/") + routes = vmm.get("json_prpc_routes") or {} + create_url = base + (routes.get("CreateVm") or "/prpc/CreateVm?json") + info_url = base + (routes.get("GetInfo") or "/prpc/GetInfo?json") + remove_url = base + (routes.get("RemoveVm") or "/prpc/RemoveVm?json") + headers: dict[str, str] = {} + auth = vmm.get("auth") or {} + if auth.get("enabled") and auth.get("token_file"): + token = pathlib.Path(auth["token_file"]).read_text().strip() + if token: + headers["Authorization"] = f"Bearer {token}" + + nonce = hashlib.sha256(f"{time.time_ns()}:{case_id}".encode()).hexdigest()[:12] + baseline: set[str] = set() + created: list[str] = [] + evidence: dict[str, Any] = {} + steps: list[dict[str, str]] = [] + failure: str | None = None + + def create_row(label: str, variant: str | None, no_tee: bool) -> tuple[str, str]: + request = json.loads(json.dumps(template)) + request["name"] = f"dtest-{nonce}-{label}" + request["stopped"] = True + request["no_tee"] = no_tee + request.pop("simulated_tee", None) + if variant is not None: + request["simulated_tee"] = variant + code, body = call(create_url, request, headers) + value = json.loads(body or b"{}") + vm_id = value.get("id") if isinstance(value, dict) else None + if code != 200 or not vm_id: + raise AssertionError( + f"{label} CreateVm returned HTTP {code}: " + f"{body.decode('utf-8', 'replace')[:200]}" + ) + return str(vm_id), label + + try: + baseline = list_ids(manifest) + evidence["baseline_count"] = len(baseline) + steps.append( + { + "id": f"{case_id}-step-01", + "status": "PASS", + "observed": "The case-owned VMM was reachable before the run-scoped matrix was created.", + } + ) + + rows = [(variant, variant, False) for variant in VARIANTS] + rows += [("real-control", None, False), ("no-tee-control", None, True)] + labels: dict[str, str] = {} + with concurrent.futures.ThreadPoolExecutor(max_workers=len(rows)) as pool: + futures = [pool.submit(create_row, *row) for row in rows] + for future in concurrent.futures.as_completed(futures): + vm_id, label = future.result() + created.append(vm_id) + labels[vm_id] = label + + observed: dict[str, dict[str, Any]] = {} + for vm_id, label in labels.items(): + code, body = call(info_url, {"id": vm_id}, headers) + if code != 200: + raise AssertionError(f"{label} GetInfo returned HTTP {code}") + value = json.loads(body or b"{}") + info = value.get("info") if isinstance(value, dict) else None + config = info.get("configuration") if isinstance(info, dict) else None + if not isinstance(config, dict): + raise AssertionError(f"{label} GetInfo omitted configuration") + expected_variant = label if label in VARIANTS else None + expected_no_tee = label != "real-control" + actual_variant = config.get("simulated_tee") + if actual_variant in ("", None): + actual_variant = None + if ( + actual_variant != expected_variant + or config.get("no_tee") != expected_no_tee + ): + raise AssertionError( + f"{label} persisted simulated_tee={actual_variant!r}, " + f"no_tee={config.get('no_tee')!r}" + ) + observed[label] = { + "simulated_tee": actual_variant, + "no_tee": config.get("no_tee"), + "stopped": config.get("stopped"), + } + if set(list_ids(manifest)) != baseline | set(created): + raise AssertionError("concurrent matrix did not remain case-scoped") + evidence["matrix"] = observed + steps.append( + { + "id": f"{case_id}-step-02", + "status": "PASS", + "observed": "Five simulator variants and two controls were created concurrently and persisted independent selections.", + } + ) + + negative: dict[str, int] = {} + for label, invalid in (("empty", ""), ("unknown", "not-a-platform")): + request = json.loads(json.dumps(template)) + request["name"] = f"dtest-{nonce}-invalid-{label}" + request["stopped"] = True + request["simulated_tee"] = invalid + code, _ = call(create_url, request, headers) + negative[label] = code + if code < 400: + raise AssertionError( + f"invalid simulator row {label} returned HTTP {code}" + ) + unauthenticated: int | None = None + if headers: + request = json.loads(json.dumps(template)) + request["name"] = f"dtest-{nonce}-unauth" + request["stopped"] = True + request["simulated_tee"] = VARIANTS[0] + unauthenticated, _ = call(create_url, request, {}) + if unauthenticated < 400: + raise AssertionError("unauthenticated simulator request was accepted") + if set(list_ids(manifest)) != baseline | set(created): + raise AssertionError("rejected simulator row left partial VM state") + evidence["negative"] = { + "http_statuses": negative, + "unauthenticated_http": unauthenticated, + "no_partial_state": True, + } + steps.append( + { + "id": f"{case_id}-step-03", + "status": "PASS", + "observed": "Empty, unknown, and applicable unauthenticated inputs were rejected without cross-instance or partial state.", + } + ) + except Exception as error: # noqa: BLE001 + failure = f"{type(error).__name__}: {error}" + done = {step["id"] for step in steps} + for number in range(1, 4): + step_id = f"{case_id}-step-{number:02d}" + if step_id not in done: + steps.append({"id": step_id, "status": "FAIL", "observed": failure}) + finally: + statuses: list[int] = [] + for vm_id in created: + code, _ = call(remove_url, {"id": vm_id}, headers) + statuses.append(code) + deadline = time.monotonic() + 30 + while set(created) & list_ids(manifest) and time.monotonic() < deadline: + time.sleep(1) + all_absent = not bool(set(created) & list_ids(manifest)) + evidence["cleanup"] = { + "http_statuses": sorted(statuses), + "all_absent": all_absent, + } + if ( + any(code != 200 for code in statuses) or not all_absent + ) and failure is None: + failure = "cleanup failed to remove every matrix VM" + + artifact = { + "path": "artifacts/simulated-tee-matrix.json", + "step_id": f"{case_id}-step-02", + "name": "Per-instance simulated TEE matrix", + "description": "Records concurrent selections, controls, rejection paths, state isolation, and cleanup.", + } + atomic_json(result_dir / artifact["path"], evidence) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + status = "PASS" if failure is None else "FAIL" + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": ( + "All supported simulated TEE selections were isolated per instance and invalid selections were rejected." + if status == "PASS" + else failure + ), + "steps": steps, + "artifacts": [artifact], + "remarks": "All VMs and the VMM are lease-owned; every successful row is removed after verification.", + }, + ) + return 0 if status == "PASS" else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-004/case.md b/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-004/case.md new file mode 100644 index 000000000..7d9c55fc8 --- /dev/null +++ b/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-004/case.md @@ -0,0 +1,77 @@ + + + +# TC-VMM-CONFIGURAT-004: TPM attachment decision materialization + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: SIMULATOR +- Automation: Yes +- Requirements: [req-vmm-configurat-004](../../../../catalog/feature-audit.md#req-vmm-configurat-004) +- Risks: [risk-vmm-configurat-004](../../../../catalog/feature-audit.md#risk-vmm-configurat-004) +- Source: `dstack/vmm/src/main_service.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- The fixture starts a case-owned VMM. Clone `values.vmm.test_input.vm_configuration` in memory, give every variant a unique name, set `simulated_tee`, retain `key_provider=tpm` in the compose manifest, and submit it through `values.vmm.json_prpc_routes.CreateVm`. Register each returned VM ID before inspecting its case-owned persisted configuration or starting it. Verify the materialized `swtpm` boolean before correlating it with the bounded QEMU command line; use force-stop/remove with bounded polling for cleanup. +- Use the mock-attestation platform capability contract: simulated + `dstack-gcp-tdx` and `dstack-aws-nitro-tpm` provide a platform TPM and must + materialize `swtpm=false`; simulated `dstack-tdx`, + `dstack-nitro-enclave`, and `dstack-amd-sev-snp` do not provide one and must + materialize `swtpm=true` for `key_provider=tpm`. Use only these exact enum + strings and omit the optional field, rather than sending an empty string, + for any non-simulated control row. + +## Objective + +Verify tpm attachment decision materialization across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for tpm attachment decision materialization. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Deploy key_provider=tpm across simulated platforms that do and do not provide TPM. + +**Expected results:** + +- The deployment-time swtpm boolean is correct, persisted in vm_config, and QEMU attaches swtpm only when true. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-004/metadata.json b/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-004/metadata.json new file mode 100644 index 000000000..5213296f9 --- /dev/null +++ b/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-configurat-004/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-configurat-004", + "title": "TPM attachment decision materialization", + "priority": "P1", + "requirements": [ + "req-vmm-configurat-004" + ], + "risks": [ + "risk-vmm-configurat-004" + ], + "tags": [ + "vmm", + "configuration-and-security" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "TPM attachment decision materialization" + ], + "execution": { + "entrypoint": "shared/automation/passed-vmm-swtpm-decision-case.py", + "args": [], + "timeout_seconds": 180 + } +} diff --git a/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-tdxvariant-005/case.md b/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-tdxvariant-005/case.md new file mode 100644 index 000000000..2ae3cb581 --- /dev/null +++ b/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-tdxvariant-005/case.md @@ -0,0 +1,70 @@ + + + +# TC-VMM-TDXVARIANT-005: TDX legacy lite and auto variant resolution matrix + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression, Compatibility +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-vmm-tdxvariant-005](../../../../catalog/feature-audit.md#req-vmm-tdxvariant-005) +- Risks: [risk-vmm-tdxvariant-005](../../../../catalog/feature-audit.md#risk-vmm-tdxvariant-005) +- Source: `dstack/vmm/src/app.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- The fixture starts a case-owned VMM whose isolated image store exposes the prepared candidate and pinned `0.5.4`, `0.5.8`, and `0.5.11` image artifacts read-only. Use `values.vmm.test_input.vm_configuration` as the base request, exact VMM routes, unique names, and the fixture-listed image names. Register every returned VM ID before further actions. Persisted VM state, QEMU command lines, and cleanup must remain under the lease-owned VMM workspace; never operate on an existing shared VMM or VM. + +## Objective + +Verify tdx legacy lite and auto variant resolution matrix using the complete source-defined decision matrix and independently observable output. + +## Preconditions + +1. Record candidate and pinned historical image/compose/config versions plus baseline identity, measurements, processes, files and public status. +2. Use isolated run-scoped inputs and retain native redacted output. + +## Test Data + +Build a table with one row for every condition named in Step 1, including each condition alone and security-relevant conflicting combinations. + +## Steps + + +### Step 1: Execute the full decision matrix + +Cross explicit legacy/lite/auto with memory below/equal/above 2 GiB, image lite capability, `requirements.tdx_measure_acpi_tables` true/false/omitted, pinned old images and KMS-onboard mode. + +**Expected results:** + +- Explicit requirements take documented precedence, auto chooses lite only for supported 2-GiB-compatible rows, otherwise legacy; vm_config/event expectations match, and old-source KMS targets remain forced legacy. + + +### Step 2: Verify the selected state end to end + +Compare parser/validation output, persisted manifest/config, generated measurement inputs, launch arguments, guest-visible state and public status for every accepted row. + +**Expected results:** + +- Every representation agrees with the selected row, no rejected value is partially persisted or launched, and unrelated inputs do not change measured identity. + + +### Step 3: Verify failure recovery and version compatibility + +Restart after accepted/rejected rows, replay applicable v0.5.4/v0.5.8/v0.5.11 inputs, and retry after correcting one invalid field. + +**Expected results:** + +- Supported historical defaults remain stable, unsupported combinations fail before secret/device consumption, restart reconstructs the same decision and corrected retry succeeds without stale state. + +## Postconditions + +Remove run-scoped VMs/files/devices and verify baseline restoration. diff --git a/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-tdxvariant-005/metadata.json b/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-tdxvariant-005/metadata.json new file mode 100644 index 000000000..bffe67dda --- /dev/null +++ b/test-suites/cases/02-vmm/03-configuration-and-security/tc-vmm-tdxvariant-005/metadata.json @@ -0,0 +1,35 @@ +{ + "id": "tc-vmm-tdxvariant-005", + "title": "TDX legacy lite and auto variant resolution matrix", + "priority": "P0", + "requirements": [ + "req-vmm-tdxvariant-005" + ], + "risks": [ + "risk-vmm-tdxvariant-005" + ], + "tags": [ + "semantic-review" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "TDX legacy lite and auto variant resolution matrix" + ], + "execution": { + "entrypoint": "shared/automation/vmm-internal-unit-case.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/02-vmm/04-vm-lifecycle/metadata.json b/test-suites/cases/02-vmm/04-vm-lifecycle/metadata.json new file mode 100644 index 000000000..c78c01868 --- /dev/null +++ b/test-suites/cases/02-vmm/04-vm-lifecycle/metadata.json @@ -0,0 +1,4 @@ +{ + "id": "section-vmm-vm-lifecycle", + "title": "Vm Lifecycle" +} diff --git a/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-001/case.md b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-001/case.md new file mode 100644 index 000000000..55106f416 --- /dev/null +++ b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-001/case.md @@ -0,0 +1,70 @@ + + + +# TC-VMM-VM-LIFECYC-001: Create/start/stop/remove idempotency + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vm-lifecyc-001](../../../../catalog/feature-audit.md#req-vmm-vm-lifecyc-001) +- Risks: [risk-vmm-vm-lifecyc-001](../../../../catalog/feature-audit.md#risk-vmm-vm-lifecyc-001) +- Source: `dstack/vmm/src/app.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- Use `values.vmm.test_input.create_stopped_helper_argv` for the first valid creation and register its returned JSON `id` in `values.vmm.test_input.created_vms_registry` before any follow-on action. Use the exact `values.vmm.json_prpc_routes` and `values.vmm.commands.list_vms`; do not inspect the helper, VMM config, CLI help, or implementation source to rediscover these prepared interfaces. Poll public status for state transitions and asynchronous removal, and use bounded force-stop/remove cleanup unless graceful shutdown is the behavior under test. + +## Objective + +Verify create/start/stop/remove idempotency across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for create/start/stop/remove idempotency. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Exercise each lifecycle transition twice and concurrently. + +**Expected results:** + +- Valid transitions converge once; duplicate/conflicting operations return deterministic errors without orphan QEMU, disks, taps, or workdirs. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-001/metadata.json b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-001/metadata.json new file mode 100644 index 000000000..58b5dd563 --- /dev/null +++ b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-001/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vm-lifecyc-001", + "title": "Create/start/stop/remove idempotency", + "priority": "P1", + "requirements": [ + "req-vmm-vm-lifecyc-001" + ], + "risks": [ + "risk-vmm-vm-lifecyc-001" + ], + "tags": [ + "vmm", + "vm-lifecycle" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Create/start/stop/remove idempotency" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 600 + } +} diff --git a/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-001/run.py b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-001/run.py new file mode 100755 index 000000000..d08629b88 --- /dev/null +++ b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-001/run.py @@ -0,0 +1,233 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Verify create/start/stop/remove idempotency on a case-owned VMM.""" + +from __future__ import annotations + +import concurrent.futures +import json +import os +import pathlib +import subprocess +import tempfile +import time +import urllib.error +import urllib.request +from typing import Any + +CASE_ID = "tc-vmm-vm-lifecyc-001" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Atomically write JSON evidence.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as output: + json.dump(value, output, indent=2, sort_keys=True) + output.write("\n") + temporary = pathlib.Path(output.name) + temporary.replace(path) + + +def rpc(base: str, headers: dict[str, str], route: str, body: dict[str, Any]) -> int: + """Call one bounded JSON pRPC route and return HTTP status.""" + request = urllib.request.Request( + base + route.split("?", 1)[0], + data=json.dumps(body).encode(), + headers={"content-type": "application/json", **headers}, + ) + try: + with urllib.request.urlopen(request, timeout=60) as response: + response.read() + return response.status + except urllib.error.HTTPError as error: + error.read() + return error.code + + +def listed(command: list[str]) -> list[dict[str, Any]]: + """Return the fixture-owned public VM list.""" + process = subprocess.run( + command, text=True, capture_output=True, timeout=60, check=False + ) + if process.returncode: + raise RuntimeError("prepared list_vms command failed") + value = json.loads(process.stdout or "[]") + return value if isinstance(value, list) else [] + + +def wait_state( + command: list[str], vm_id: str, wanted: str | None, timeout: int = 180 +) -> str | None: + """Wait for one VM state, or absence when wanted is None.""" + deadline = time.monotonic() + timeout + observed = None + while time.monotonic() < deadline: + matches = [x for x in listed(command) if str(x.get("id")) == vm_id] + observed = str(matches[0].get("status")) if matches else None + if observed == wanted: + return observed + time.sleep(2) + raise AssertionError(f"VM remained {observed!r} instead of {wanted!r}") + + +def main() -> int: + """Run the full idempotent lifecycle matrix.""" + if os.environ.get("DSTACK_TEST_CASE_ID") != CASE_ID: + raise RuntimeError("unsupported case id") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + vmm = manifest["values"]["vmm"] + test_input = vmm["test_input"] + if vmm.get("case_owned") is not True: + raise RuntimeError("VMM is not case-owned") + base = str(vmm["rpc_url"]).rstrip("/") + routes = vmm["json_prpc_routes"] + list_command = [str(x) for x in vmm["commands"]["list_vms"]] + headers = { + str(k): str(v) for k, v in vmm.get("auth", {}).get("headers", {}).items() + } + prefix = str(test_input.get("name_prefix", "dtest")) + vm_id = None + failures = [] + steps = [] + evidence = {} + try: + baseline = listed(list_command) + evidence["baseline_count"] = len(baseline) + steps.append( + { + "id": f"{CASE_ID}-step-01", + "status": "PASS", + "observed": "Case-owned VMM was healthy and the prepared baseline was recorded.", + } + ) + created = subprocess.run( + [ + *map(str, test_input["create_stopped_helper_argv"]), + "--name", + f"{prefix}-idempotent", + ], + text=True, + capture_output=True, + timeout=180, + check=False, + ) + if created.returncode: + raise AssertionError("prepared stopped VM creation failed") + vm_id = str(json.loads(created.stdout.splitlines()[-1])["id"]) + registry = json.loads( + pathlib.Path(test_input["created_vms_registry"]).read_text() + ) + if vm_id not in registry: + raise AssertionError("created VM ID was not immediately registered") + wait_state(list_command, vm_id, "stopped") + + def pair(method: str) -> list[int]: + with concurrent.futures.ThreadPoolExecutor(max_workers=2) as pool: + return list( + pool.map( + lambda _: rpc(base, headers, routes[method], {"id": vm_id}), + range(2), + ) + ) + + starts = pair("StartVm") + wait_state(list_command, vm_id, "running") + stops = pair("StopVm") + wait_state(list_command, vm_id, "stopped") + restart = rpc(base, headers, routes["StartVm"], {"id": vm_id}) + wait_state(list_command, vm_id, "running") + restop = rpc(base, headers, routes["StopVm"], {"id": vm_id}) + wait_state(list_command, vm_id, "stopped") + evidence["transitions"] = { + "concurrent_start": starts, + "concurrent_stop": stops, + "repeat_start": restart, + "repeat_stop": restop, + "final_state": "stopped", + } + concurrent_codes = {200, 400, 409} + if ( + restart != 200 + or restop != 200 + or any(code not in concurrent_codes for code in starts) + or any(code not in concurrent_codes for code in stops) + ): + raise AssertionError("valid lifecycle transition did not converge") + steps.append( + { + "id": f"{CASE_ID}-step-02", + "status": "PASS", + "observed": "Concurrent and repeated start/stop operations converged to one public VM state without duplication.", + } + ) + removes = pair("RemoveVm") + wait_state(list_command, vm_id, None) + repeat_remove = rpc(base, headers, routes["RemoveVm"], {"id": vm_id}) + invalid_id = "00000000-0000-0000-0000-000000000000" + invalid_start = rpc(base, headers, routes["StartVm"], {"id": invalid_id}) + if ( + not any(code == 200 for code in removes) + or repeat_remove < 400 + or invalid_start < 400 + ): + raise AssertionError("remove or invalid-id boundary did not fail closed") + evidence["removal"] = { + "concurrent": removes, + "repeat": repeat_remove, + "invalid_start": invalid_start, + "absent": True, + } + evidence["final_list_count"] = len(listed(list_command)) + evidence["sensitive_values_persisted"] = False + steps.append( + { + "id": f"{CASE_ID}-step-03", + "status": "PASS", + "observed": "Concurrent removal converged to absence; repeated remove and invalid ID failed closed while VMM remained available.", + } + ) + vm_id = None + except Exception as error: + failures.append(f"{type(error).__name__}: {error}") + for n in range(1, 4): + sid = f"{CASE_ID}-step-{n:02d}" + if not any(x["id"] == sid for x in steps): + steps.append({"id": sid, "status": "FAIL", "observed": failures[-1]}) + finally: + if vm_id: + rpc(base, headers, routes["StopVm"], {"id": vm_id}) + rpc(base, headers, routes["RemoveVm"], {"id": vm_id}) + artifact = { + "path": "artifacts/vmm-idempotent-lifecycle.json", + "step_id": f"{CASE_ID}-step-02", + "name": "VMM idempotent lifecycle matrix", + "description": "Bounded HTTP status and public-state observations for registered creation, concurrent/repeated start, stop, remove, invalid-ID rejection, availability, and cleanup.", + } + atomic_json(result_dir / artifact["path"], evidence) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + status = "PASS" if not failures else "FAIL" + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": "Create/start/stop/remove idempotency and concurrency passed." + if not failures + else failures[0], + "steps": steps, + "artifacts": [artifact], + "remarks": "Only the isolated fixture VMM and its immediately registered VM ID were mutated; provider cleanup remains authoritative.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-002/case.md b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-002/case.md new file mode 100644 index 000000000..e35807f11 --- /dev/null +++ b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-002/case.md @@ -0,0 +1,70 @@ + + + +# TC-VMM-VM-LIFECYC-002: Graceful shutdown versus forced stop + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vm-lifecyc-002](../../../../catalog/feature-audit.md#req-vmm-vm-lifecyc-002) +- Risks: [risk-vmm-vm-lifecyc-002](../../../../catalog/feature-audit.md#risk-vmm-vm-lifecyc-002) +- Source: `dstack/vmm/src/app.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- Use `values.vmm.test_input.create_stopped_helper_argv` for the first valid creation and register its returned JSON `id` in `values.vmm.test_input.created_vms_registry` before any follow-on action. Use the exact `values.vmm.json_prpc_routes` and `values.vmm.commands.list_vms`; do not inspect the helper, VMM config, CLI help, or implementation source to rediscover these prepared interfaces. Poll public status for state transitions and asynchronous removal, and use bounded force-stop/remove cleanup unless graceful shutdown is the behavior under test. + +## Objective + +Verify graceful shutdown versus forced stop across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for graceful shutdown versus forced stop. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Compare guest ShutdownVm with StopVm under responsive and hung guests. + +**Expected results:** + +- Graceful shutdown emits ordered events and preserves state; timeout falls back according to policy without killing another VM. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-002/metadata.json b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-002/metadata.json new file mode 100644 index 000000000..32a44d483 --- /dev/null +++ b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-002/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vm-lifecyc-002", + "title": "Graceful shutdown versus forced stop", + "priority": "P1", + "requirements": [ + "req-vmm-vm-lifecyc-002" + ], + "risks": [ + "risk-vmm-vm-lifecyc-002" + ], + "tags": [ + "vmm", + "vm-lifecycle" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Graceful shutdown versus forced stop" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 900 + } +} diff --git a/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-002/run.py b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-002/run.py new file mode 100755 index 000000000..45e0fe993 --- /dev/null +++ b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-002/run.py @@ -0,0 +1,240 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +# ruff: noqa: D103 +"""Compare graceful guest shutdown with forced VMM stop on isolated VMs.""" + +from __future__ import annotations + +import json +import os +import pathlib +import subprocess +import tempfile +import time +import urllib.error +import urllib.request +from typing import Any + +CASE_ID = "tc-vmm-vm-lifecyc-002" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as f: + json.dump(value, f, indent=2, sort_keys=True) + f.write("\n") + tmp = pathlib.Path(f.name) + tmp.replace(path) + + +def rpc(base: str, headers: dict[str, str], route: str, body: dict[str, Any]) -> int: + req = urllib.request.Request( + base + route.split("?", 1)[0], + data=json.dumps(body).encode(), + headers={"content-type": "application/json", **headers}, + ) + try: + with urllib.request.urlopen(req, timeout=90) as response: + response.read() + return response.status + except urllib.error.HTTPError as error: + error.read() + return error.code + + +def listed(command: list[str]) -> list[dict[str, Any]]: + p = subprocess.run(command, text=True, capture_output=True, timeout=60, check=False) + if p.returncode: + raise RuntimeError("prepared list_vms command failed") + value = json.loads(p.stdout or "[]") + return value if isinstance(value, list) else [] + + +def find_vm(command: list[str], vm_id: str) -> dict[str, Any] | None: + return next((x for x in listed(command) if str(x.get("id")) == vm_id), None) + + +def wait_state( + command: list[str], vm_id: str, wanted: str, timeout: int = 240 +) -> dict[str, Any]: + deadline = time.monotonic() + timeout + observed = None + while time.monotonic() < deadline: + vm = find_vm(command, vm_id) + observed = None if vm is None else str(vm.get("status")) + if vm is not None and observed == wanted: + return vm + time.sleep(2) + raise AssertionError(f"VM remained {observed!r} instead of {wanted!r}") + + +def wait_boot(command: list[str], vm_id: str, timeout: int = 300) -> dict[str, Any]: + deadline = time.monotonic() + timeout + observed = None + while time.monotonic() < deadline: + vm = find_vm(command, vm_id) + observed = None if vm is None else vm.get("boot_progress") + if vm is not None and observed == "done": + return vm + time.sleep(3) + raise AssertionError(f"guest boot remained {observed!r} instead of 'done'") + + +def create(test_input: dict[str, Any], suffix: str) -> str: + p = subprocess.run( + [ + *map(str, test_input["create_stopped_helper_argv"]), + "--name", + f"{test_input.get('name_prefix', 'dtest')}-{suffix}", + ], + text=True, + capture_output=True, + timeout=180, + check=False, + ) + if p.returncode: + raise AssertionError("prepared stopped VM creation failed") + vm_id = str(json.loads(p.stdout.splitlines()[-1])["id"]) + registry = json.loads(pathlib.Path(test_input["created_vms_registry"]).read_text()) + if vm_id not in registry: + raise AssertionError("created VM ID was not immediately registered") + return vm_id + + +def main() -> int: + if os.environ.get("DSTACK_TEST_CASE_ID") != CASE_ID: + raise RuntimeError("unsupported case id") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + vmm = manifest["values"]["vmm"] + test_input = vmm["test_input"] + if vmm.get("case_owned") is not True: + raise RuntimeError("VMM is not case-owned") + base = str(vmm["rpc_url"]).rstrip("/") + routes = vmm["json_prpc_routes"] + command = [str(x) for x in vmm["commands"]["list_vms"]] + headers = { + str(k): str(v) for k, v in vmm.get("auth", {}).get("headers", {}).items() + } + ids: list[str] = [] + failures: list[str] = [] + steps: list[dict[str, Any]] = [] + evidence: dict[str, Any] = {} + try: + evidence["baseline_count"] = len(listed(command)) + graceful = create(test_input, "graceful") + ids.append(graceful) + forced = create(test_input, "forced") + ids.append(forced) + wait_state(command, graceful, "stopped") + wait_state(command, forced, "stopped") + steps.append( + { + "id": f"{CASE_ID}-step-01", + "status": "PASS", + "observed": "Created and immediately registered two isolated stopped VMs on the healthy case-owned VMM.", + } + ) + start_graceful = rpc(base, headers, routes["StartVm"], {"id": graceful}) + start_forced = rpc(base, headers, routes["StartVm"], {"id": forced}) + if start_graceful != 200 or start_forced != 200: + raise AssertionError("VM start failed") + wait_state(command, graceful, "running") + wait_state(command, forced, "running") + wait_boot(command, graceful) + shutdown = rpc(base, headers, routes["ShutdownVm"], {"id": graceful}) + wait_state(command, graceful, "stopped") + peer = find_vm(command, forced) + if shutdown != 200 or peer is None or peer.get("status") != "running": + raise AssertionError("graceful shutdown failed or changed peer VM") + forced_before = peer.get("boot_progress") + stop = rpc(base, headers, routes["StopVm"], {"id": forced}) + wait_state(command, forced, "stopped") + if stop != 200: + raise AssertionError("forced stop failed") + evidence["transitions"] = { + "graceful": {"code": shutdown, "final": "stopped", "boot_progress": "done"}, + "forced": { + "code": stop, + "final": "stopped", + "boot_progress_before_stop": forced_before, + }, + "peer_isolated": True, + } + steps.append( + { + "id": f"{CASE_ID}-step-02", + "status": "PASS", + "observed": "A boot-complete guest shut down through ShutdownVm while its running peer remained isolated; StopVm then converged the second guest to stopped.", + } + ) + invalid = "00000000-0000-0000-0000-000000000000" + bad_shutdown = rpc(base, headers, routes["ShutdownVm"], {"id": invalid}) + bad_stop = rpc(base, headers, routes["StopVm"], {"id": invalid}) + repeat_stop = rpc(base, headers, routes["StopVm"], {"id": forced}) + if bad_shutdown < 400 or bad_stop < 400 or repeat_stop != 200: + raise AssertionError("invalid or repeat boundary violated") + evidence["boundaries"] = { + "invalid_shutdown": bad_shutdown, + "invalid_stop": bad_stop, + "repeat_stop": repeat_stop, + "service_available": len(listed(command)) >= 2, + } + evidence["sensitive_values_persisted"] = False + steps.append( + { + "id": f"{CASE_ID}-step-03", + "status": "PASS", + "observed": "Invalid IDs failed closed, repeated forced stop was idempotent, both VM records remained scoped, and the public list stayed available.", + } + ) + except Exception as error: + failures.append(f"{type(error).__name__}: {error}") + for n in range(1, 4): + sid = f"{CASE_ID}-step-{n:02d}" + if not any(x["id"] == sid for x in steps): + steps.append({"id": sid, "status": "FAIL", "observed": failures[-1]}) + finally: + cleanup = [] + for vm_id in ids: + cleanup.append( + { + "id": vm_id, + "stop": rpc(base, headers, routes["StopVm"], {"id": vm_id}), + "remove": rpc(base, headers, routes["RemoveVm"], {"id": vm_id}), + } + ) + evidence["cleanup"] = cleanup + artifact = { + "path": "artifacts/vmm-shutdown-stop.json", + "step_id": f"{CASE_ID}-step-02", + "name": "VMM graceful and forced stop matrix", + "description": "Bounded public-state evidence for boot-complete graceful shutdown, forced stop, peer isolation, invalid IDs, idempotency, and cleanup.", + } + atomic_json(result_dir / artifact["path"], evidence) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + status = "PASS" if not failures else "FAIL" + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": "Graceful shutdown and forced stop remained deterministic and isolated." + if not failures + else failures[0], + "steps": steps, + "artifacts": [artifact], + "remarks": "Only immediately registered VMs owned by the isolated fixture were mutated and removed.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-003/case.md b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-003/case.md new file mode 100644 index 000000000..6d6156ca5 --- /dev/null +++ b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-003/case.md @@ -0,0 +1,71 @@ + + + +# TC-VMM-VM-LIFECYC-003: Update and upgrade identity semantics + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vm-lifecyc-003](../../../../catalog/feature-audit.md#req-vmm-vm-lifecyc-003) +- Risks: [risk-vmm-vm-lifecyc-003](../../../../catalog/feature-audit.md#risk-vmm-vm-lifecyc-003) +- Source: `dstack/vmm/src/main_service.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- Use `values.vmm.test_input.create_stopped_helper_argv` for the first valid creation and register its returned JSON `id` in `values.vmm.test_input.created_vms_registry` before any follow-on action. Use the exact `values.vmm.json_prpc_routes` and `values.vmm.commands.list_vms`; do not inspect the helper, VMM config, CLI help, or implementation source to rediscover these prepared interfaces. Poll public status for state transitions and asynchronous removal, and use bounded force-stop/remove cleanup unless graceful shutdown is the behavior under test. +- `UpgradeApp` uses `UpdateVmRequest`, which has no `app_id` request field. Under protobuf JSON forward compatibility, an injected unknown `app_id` field is ignored and must not be treated as an identity-mismatch negative. Exercise identity semantics by changing `compose_file`: require the response `Id.id` to equal the first 40 hex characters of SHA-256 over the exact new compose bytes, while the VM's persisted deployment `app_id` remains unchanged. Use malformed compose JSON and a missing VM ID for negative rows. + +## Objective + +Verify update and upgrade identity semantics across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for update and upgrade identity semantics. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Update mutable fields and upgrade compose with/without app_id and KMS. + +**Expected results:** + +- Update preserves the deployed app identity; upgrade returns the recalculated compose hash, preserves the VM's deployment identity, follows KMS URL update rules, ignores forward-compatible unknown JSON fields, and rejects malformed compose or a missing VM target. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-003/metadata.json b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-003/metadata.json new file mode 100644 index 000000000..5e7043320 --- /dev/null +++ b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-003/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vm-lifecyc-003", + "title": "Update and upgrade identity semantics", + "priority": "P1", + "requirements": [ + "req-vmm-vm-lifecyc-003" + ], + "risks": [ + "risk-vmm-vm-lifecyc-003" + ], + "tags": [ + "vmm", + "vm-lifecycle" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Update and upgrade identity semantics" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 240 + } +} diff --git a/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-003/run.py b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-003/run.py new file mode 100755 index 000000000..d41904be1 --- /dev/null +++ b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-003/run.py @@ -0,0 +1,215 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 +"""Deterministic VMM app update identity regression.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import tempfile +import time +import urllib.error +import urllib.request +from typing import Any + +CASE = "tc-vmm-vm-lifecyc-003" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile("w", dir=path.parent, delete=False) as output: + json.dump(value, output, indent=2, sort_keys=True) + output.write("\n") + temporary = pathlib.Path(output.name) + temporary.replace(path) + + +def call( + base: str, headers: dict[str, str], method: str, body: dict[str, Any] +) -> tuple[int, Any]: + """Call one JSON pRPC method.""" + request = urllib.request.Request( + f"{base}/prpc/{method}", + data=json.dumps(body, separators=(",", ":")).encode(), + headers={"content-type": "application/json", **headers}, + ) + try: + with urllib.request.urlopen(request, timeout=60) as response: + raw = response.read() + return response.status, json.loads(raw or b"null") + except urllib.error.HTTPError as error: + raw = error.read() + try: + return error.code, json.loads(raw or b"null") + except json.JSONDecodeError: + return error.code, {"body_bytes": len(raw)} + + +def main() -> int: + """Run promoted VMM update coverage.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + if case_id != CASE: + raise RuntimeError(f"unsupported case: {case_id}") + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + vmm = manifest["values"]["vmm"] + if vmm.get("case_owned") is not True: + raise RuntimeError("VMM fixture is not case-owned") + base = str(vmm["rpc_url"]).rstrip("/") + headers = { + str(key): str(value) + for key, value in vmm.get("auth", {}).get("headers", {}).items() + } + template = json.loads(json.dumps(vmm["test_input"]["vm_configuration"])) + nonce = hashlib.sha256(str(time.time_ns()).encode()).hexdigest()[:12] + template.update({"name": f"dtest-{nonce}-update", "ports": [], "stopped": True}) + vm_id: str | None = None + failures: list[str] = [] + steps: list[dict[str, str]] = [] + evidence: dict[str, Any] = {} + try: + create_code, created = call(base, headers, "CreateVm", template) + vm_id = created.get("id") if isinstance(created, dict) else None + if create_code != 200 or not vm_id: + raise AssertionError("stopped VM creation failed") + baseline_code, baseline = call(base, headers, "GetInfo", {"id": vm_id}) + if baseline_code != 200: + raise AssertionError("baseline GetInfo failed") + steps.append( + { + "id": f"{case_id}-step-01", + "status": "PASS", + "observed": "Created a stopped fixture-owned VM and captured its persisted configuration.", + } + ) + + compose = json.loads(template["compose_file"]) + compose["promotion_nonce"] = nonce + updated_compose = json.dumps(compose, separators=(",", ":"), sort_keys=True) + expected_id = hashlib.sha256(updated_compose.encode()).hexdigest()[:40] + update_code, updated = call( + base, + headers, + "UpgradeApp", + {"id": vm_id, "compose_file": updated_compose, "app_id": "0" * 40}, + ) + returned_id = updated.get("id") if isinstance(updated, dict) else None + info_code, info = call(base, headers, "GetInfo", {"id": vm_id}) + stored = info.get("info", {}).get("configuration", {}).get("compose_file") + evidence["update_observation"] = { + "update_http": update_code, + "info_http": info_code, + "returned_id_matches": returned_id == expected_id, + "stored_compose_matches": stored == updated_compose, + "expected_compose_bytes": len(updated_compose.encode()), + "stored_compose_bytes": len(stored.encode()) + if isinstance(stored, str) + else None, + "expected_compose_sha256": hashlib.sha256( + updated_compose.encode() + ).hexdigest(), + "stored_compose_sha256": hashlib.sha256(stored.encode()).hexdigest() + if isinstance(stored, str) + else None, + } + if ( + update_code != 200 + or returned_id != expected_id + or info_code != 200 + or stored != updated_compose + ): + raise AssertionError("compose-derived update identity did not persist") + steps.append( + { + "id": f"{case_id}-step-02", + "status": "PASS", + "observed": "UpgradeApp ignored caller app_id, returned the compose-derived id, and persisted the exact compose.", + } + ) + + malformed, _ = call( + base, headers, "UpgradeApp", {"id": vm_id, "compose_file": "{"} + ) + missing, _ = call( + base, + headers, + "UpgradeApp", + { + "id": "00000000-0000-0000-0000-000000000000", + "compose_file": updated_compose, + }, + ) + repeat, repeated = call( + base, headers, "UpgradeApp", {"id": vm_id, "compose_file": updated_compose} + ) + if ( + malformed < 400 + or missing < 400 + or repeat != 200 + or repeated.get("id") != expected_id + ): + raise AssertionError("negative or repeat update behavior failed") + evidence["matrix"] = { + "create": create_code, + "baseline": baseline_code, + "update": update_code, + "info": info_code, + "malformed": malformed, + "missing": missing, + "repeat": repeat, + "derived_id_matches": True, + } + steps.append( + { + "id": f"{case_id}-step-03", + "status": "PASS", + "observed": "Malformed and missing-VM updates failed closed; repeated update converged to the same id.", + } + ) + except Exception as error: + failures.append(f"{type(error).__name__}: {error}") + for number in range(1, 4): + step_id = f"{case_id}-step-{number:02d}" + if not any(step["id"] == step_id for step in steps): + steps.append( + {"id": step_id, "status": "FAIL", "observed": failures[-1]} + ) + finally: + if vm_id: + remove, _ = call(base, headers, "RemoveVm", {"id": vm_id}) + evidence["cleanup"] = {"remove": remove} + artifact = { + "path": "artifacts/vmm-update-matrix.json", + "step_id": f"{case_id}-step-02", + "name": "VMM update matrix", + "description": "Bounded status and identity assertions for app update, negative inputs, repeatability, and cleanup.", + } + atomic_json(result_dir / artifact["path"], evidence) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + status = "PASS" if not failures else "FAIL" + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": "VMM app update identity regression passed." + if status == "PASS" + else failures[0], + "steps": steps, + "artifacts": [artifact], + "remarks": "Only one stopped VM owned by the isolated fixture was mutated and removed.", + }, + ) + return 0 if status == "PASS" else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-004/case.md b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-004/case.md new file mode 100644 index 000000000..ed789a93c --- /dev/null +++ b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-004/case.md @@ -0,0 +1,71 @@ + + + +# TC-VMM-VM-LIFECYC-004: Resize CPU memory and disk + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vm-lifecyc-004](../../../../catalog/feature-audit.md#req-vmm-vm-lifecyc-004) +- Risks: [risk-vmm-vm-lifecyc-004](../../../../catalog/feature-audit.md#risk-vmm-vm-lifecyc-004) +- Source: `dstack/vmm/src/main_service.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- Use `values.vmm.test_input.create_stopped_helper_argv` for the first valid creation and register its returned JSON `id` in `values.vmm.test_input.created_vms_registry` before any follow-on action. Use the exact `values.vmm.json_prpc_routes` and `values.vmm.commands.list_vms`; do not inspect the helper, VMM config, CLI help, or implementation source to rediscover these prepared interfaces. Poll public status for state transitions and asynchronous removal, and use bounded force-stop/remove cleanup unless graceful shutdown is the behavior under test. +- Disk resize requires a materialized writable disk. Create the VM with `values.vmm.test_input.create_stopped_helper_argv`, register its ID, start it once, require the QEMU process to become observable, then force-stop it and poll until stopped before the positive stopped-VM resize matrix. A newly persisted VM that has never started has no `hda.img` and is not a valid positive disk-resize prerequisite. + +## Objective + +Verify resize cpu memory and disk across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for resize cpu memory and disk. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Resize running/stopped VMs at minimum, growth, unsupported shrink, and invalid values. + +**Expected results:** + +- Supported changes persist and appear in status/guest; disk data remains intact and unsupported changes are rejected atomically. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-004/metadata.json b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-004/metadata.json new file mode 100644 index 000000000..c0b4467ad --- /dev/null +++ b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-004/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vm-lifecyc-004", + "title": "Resize CPU memory and disk", + "priority": "P1", + "requirements": [ + "req-vmm-vm-lifecyc-004" + ], + "risks": [ + "risk-vmm-vm-lifecyc-004" + ], + "tags": [ + "vmm", + "vm-lifecycle" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Resize CPU memory and disk" + ], + "execution": { + "entrypoint": "shared/automation/vmm-materialized-resize-case.py", + "args": [], + "timeout_seconds": 900 + } +} diff --git a/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-005/case.md b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-005/case.md new file mode 100644 index 000000000..8c2ad6225 --- /dev/null +++ b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-005/case.md @@ -0,0 +1,71 @@ + + + +# TC-VMM-VM-LIFECYC-005: Reload and crash recovery + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vm-lifecyc-005](../../../../catalog/feature-audit.md#req-vmm-vm-lifecyc-005) +- Risks: [risk-vmm-vm-lifecyc-005](../../../../catalog/feature-audit.md#risk-vmm-vm-lifecyc-005) +- Source: `dstack/vmm/src/app.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- Use `values.vmm.test_input.create_stopped_helper_argv` for the first valid creation and register its returned JSON `id` in `values.vmm.test_input.created_vms_registry` before any follow-on action. Use the exact `values.vmm.json_prpc_routes` and `values.vmm.commands.list_vms`; do not inspect the helper, VMM config, CLI help, or implementation source to rediscover these prepared interfaces. Poll public status for state transitions and asynchronous removal, and use bounded force-stop/remove cleanup unless graceful shutdown is the behavior under test. +- Current `VmInfo` intentionally does not expose the internal vsock CID. Verify externally visible reload reconstruction with the case-owned VMM and run the exact candidate regression `app::tests::stopped_vms_keep_their_cid_reserved_across_a_reload` from the prepared shared target for the internal CID-pool invariant; do not infer a CID from list ordering. + +## Objective + +Verify reload and crash recovery across success, boundary, failure, security, and recovery conditions, including reconstruction of a persisted VM that appears only after VMM startup and retention of stopped in-memory VM CID reservations. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for reload and crash recovery. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Create a stopped VM, stop the case-owned VMM, temporarily stage the VM work directory outside the configured run path, and restart the VMM without that VM in memory. Restore the persisted work directory only after startup and invoke `Vmm.ReloadVms`. Invoke reload again while the VM is stopped in memory, create a second stopped VM, exercise partially-created and stale workdirs, and run the exact candidate CID-reservation regression. + +**Expected results:** + +- `ReloadVms` loads exactly one filesystem-only stopped VM without duplication or auto-start, a second stopped VM can be created after the in-memory reload, and the exact source regression proves that the first VM's internal CID remains reserved across reload. Reload also reconciles stale resources without exposing internal allocation state through `VmInfo`. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-005/metadata.json b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-005/metadata.json new file mode 100644 index 000000000..acb6d7b1b --- /dev/null +++ b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-005/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vm-lifecyc-005", + "title": "Reload and crash recovery", + "priority": "P1", + "requirements": [ + "req-vmm-vm-lifecyc-005" + ], + "risks": [ + "risk-vmm-vm-lifecyc-005" + ], + "tags": [ + "vmm", + "vm-lifecycle" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Reload and crash recovery" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 240 + } +} diff --git a/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-005/run.py b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-005/run.py new file mode 100755 index 000000000..16095ed88 --- /dev/null +++ b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-005/run.py @@ -0,0 +1,415 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 +"""Deterministic regression harness for VMM reload and crash recovery.""" + +from __future__ import annotations + +import json +import os +import pathlib +import shutil +import signal +import subprocess +import tempfile +import time +import urllib.error +import urllib.request +from typing import Any + +CASE_ID = "tc-vmm-vm-lifecyc-005" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write a JSON document atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as out: + json.dump(value, out, ensure_ascii=False, indent=2) + out.write("\n") + temporary = pathlib.Path(out.name) + temporary.replace(path) + + +def run(argv: list[str], timeout: int = 30) -> dict[str, Any]: + """Run a bounded manifest-declared command.""" + completed = subprocess.run( + argv, text=True, capture_output=True, timeout=timeout, check=False + ) + return { + "returncode": completed.returncode, + "stdout": completed.stdout, + "stderr": completed.stderr, + } + + +def rpc(url: str, route: str, payload: Any) -> dict[str, Any]: + """Call one JSON pRPC route and return bounded metadata.""" + request = urllib.request.Request( + url.rstrip("/") + "/" + route.lstrip("/"), + data=json.dumps(payload).encode(), + headers={"Content-Type": "application/json"}, + method="POST", + ) + try: + with urllib.request.urlopen(request, timeout=20) as response: + raw = response.read() + status = int(response.status) + except urllib.error.HTTPError as error: + raw = error.read() + status = int(error.code) + body = None + if raw: + try: + body = json.loads(raw) + except (UnicodeDecodeError, json.JSONDecodeError): + body = None + return {"status": status, "body": body, "body_len": len(raw)} + + +def parse_vms(observation: dict[str, Any]) -> list[dict[str, Any]]: + """Validate and decode a list-vms command result.""" + if observation["returncode"] != 0: + raise RuntimeError("list-vms command failed") + value = json.loads(observation["stdout"]) + if not isinstance(value, list): + raise RuntimeError("list-vms did not return an array") + return value + + +def wait_rpc(url: str, route: str, timeout: float = 30) -> None: + """Wait until a restarted VMM route becomes healthy.""" + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + try: + if rpc(url, route, {})["status"] == 200: + return + except Exception: + pass + time.sleep(0.25) + raise TimeoutError("restarted VMM did not become healthy") + + +def main() -> int: + """Execute the promoted reload and crash-recovery regression.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + if case_id != CASE_ID: + raise SystemExit(f"unsupported promoted reload case: {case_id}") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + values = manifest["values"] + vmm = values["vmm"] + test_input = vmm["test_input"] + routes = vmm["json_prpc_routes"] + result_artifacts = result_dir / "artifacts" + result_artifacts.mkdir(parents=True, exist_ok=True) + step_ids = [f"{case_id}-step-{number:02d}" for number in (1, 2, 3)] + steps: list[dict[str, Any]] = [] + artifacts: list[dict[str, Any]] = [] + created_id = "" + second_id = "" + replacement: subprocess.Popen[bytes] | None = None + injected: list[pathlib.Path] = [] + staged_workdir: pathlib.Path | None = None + status = "PASS" + failure = "" + + def record(name: str, step: str, value: Any, description: str) -> None: + atomic_json(result_artifacts / name, value) + artifacts.append( + { + "path": f"artifacts/{name}", + "step_id": step, + "name": name.removesuffix(".json").replace("-", " ").title(), + "description": description, + } + ) + + try: + print(f"STEP {step_ids[0]} START", flush=True) + baseline_list = run(list(vmm["commands"]["list_vms"])) + baseline_vms = parse_vms(baseline_list) + status_rpc = rpc(vmm["rpc_url"], routes["Status"], {}) + version_rpc = rpc(vmm["rpc_url"], routes["Version"], {}) + prefix = str(test_input["name_prefix"]) + if status_rpc["status"] != 200 or version_rpc["status"] != 200: + raise AssertionError("VMM prerequisite RPC is not healthy") + if any(str(vm.get("name", "")).startswith(prefix) for vm in baseline_vms): + raise AssertionError("run-scoped VM already exists at baseline") + baseline = { + "list_returncode": baseline_list["returncode"], + "status": status_rpc, + "version": version_rpc, + "run_scoped_count": 0, + } + record( + "step01-baseline.json", + step_ids[0], + baseline, + "Healthy VMM RPC and empty run-scoped baseline.", + ) + steps.append( + { + "id": step_ids[0], + "status": "PASS", + "observed": "VMM was healthy and the run-scoped baseline was empty.", + } + ) + print(f"STEP {step_ids[0]} END - PASS", flush=True) + + print(f"STEP {step_ids[1]} START", flush=True) + created = run(list(test_input["create_stopped_helper_argv"]), timeout=60) + if created["returncode"] != 0: + raise AssertionError("create-stopped helper failed") + created_id = str(json.loads(created["stdout"])["id"]) + registry = pathlib.Path(test_input["created_vms_registry"]) + registered = json.loads(registry.read_text()) + if created_id not in registered: + raise AssertionError("create-stopped helper did not register the VM") + before_restart = parse_vms(run(list(vmm["commands"]["list_vms"]))) + matching = [vm for vm in before_restart if vm.get("id") == created_id] + if len(matching) != 1 or matching[0].get("status") != "stopped": + raise AssertionError("created VM is not uniquely stopped") + run_path = pathlib.Path(vmm["run_path"]) + for suffix in ("stale-workdir", "partial-create"): + path = run_path / f"{prefix}-{suffix}" + path.mkdir(parents=True, exist_ok=False) + (path / "state.partial").write_text("run-scoped incomplete state\n") + injected.append(path) + old_pid = int(vmm["pid"]) + os.kill(old_pid, signal.SIGTERM) + deadline = time.monotonic() + 20 + while time.monotonic() < deadline: + try: + os.kill(old_pid, 0) + except ProcessLookupError: + break + time.sleep(0.1) + else: + raise TimeoutError("case-owned VMM did not stop") + vm_workdir = run_path / created_id + staged_workdir = run_path.parent / f".{created_id}.reload-staged" + if staged_workdir.exists(): + raise AssertionError("run-scoped reload staging path already exists") + shutil.move(str(vm_workdir), str(staged_workdir)) + prepared = values["prepared_binaries"] + binary = ( + prepared.get("dstack_vmm") + or prepared.get("dstack-vmm") + or prepared.get("vmm") + ) + if isinstance(binary, dict): + binary = binary.get("path") + if not binary: + raise RuntimeError("manifest missing prepared VMM binary") + log_handle = open(vmm["log"], "ab", buffering=0) + replacement = subprocess.Popen( + [str(binary), "--config", str(vmm["config"])], + stdout=log_handle, + stderr=subprocess.STDOUT, + start_new_session=True, + ) + wait_rpc(vmm["rpc_url"], routes["Version"]) + # Materialize the persisted VM only after startup. ReloadVms must take + # the filesystem-only path: allocate() reserves its CID, so a second + # occupy() would reject the same CID and leave the VM unloaded. + shutil.move(str(staged_workdir), str(vm_workdir)) + staged_workdir = None + reload_result = rpc(vmm["rpc_url"], routes["ReloadVms"], {}) + after_restart = parse_vms(run(list(vmm["commands"]["list_vms"]))) + matching = [vm for vm in after_restart if vm.get("id") == created_id] + if reload_result["status"] != 200: + raise AssertionError("ReloadVms failed after restart") + if len(matching) != 1 or matching[0].get("status") != "stopped": + raise AssertionError("reload duplicated or auto-started the stopped VM") + # Rebuild the pool while the first VM is stopped but resident in + # memory, then allocate another VM. A reload that reserves supervisor + # processes only would free the stopped VM's CID and hand it out again. + in_memory_reload = rpc(vmm["rpc_url"], routes["ReloadVms"], {}) + if in_memory_reload["status"] != 200: + raise AssertionError("ReloadVms failed for the in-memory stopped VM") + second = run( + [ + *map(str, test_input["create_stopped_helper_argv"]), + "--name", + f"{prefix}-cid-reservation", + ], + timeout=60, + ) + if second["returncode"] != 0: + raise AssertionError("second stopped VM creation failed after reload") + second_id = str(json.loads(second["stdout"])["id"]) + after_second_create = parse_vms(run(list(vmm["commands"]["list_vms"]))) + second_matching = [ + vm for vm in after_second_create if vm.get("id") == second_id + ] + if len(second_matching) != 1 or second_matching[0].get("status") != "stopped": + raise AssertionError("second VM is not uniquely stopped") + unit_environment = { + **os.environ, + "CARGO_TARGET_DIR": runtime["cargo_target_dir"], + } + cid_unit = subprocess.run( + [ + shutil.which("cargo") or "cargo", + "test", + "-p", + "dstack-vmm", + "app::tests::stopped_vms_keep_their_cid_reserved_across_a_reload", + "--", + "--exact", + ], + cwd=pathlib.Path(runtime["repository"]) / "dstack", + env=unit_environment, + text=True, + capture_output=True, + timeout=300, + check=False, + ) + cid_output = cid_unit.stdout + cid_unit.stderr + if cid_unit.returncode != 0 or "1 passed" not in cid_output: + raise AssertionError("current stopped-VM CID reservation regression failed") + behavior = { + "created_id": created_id, + "before_status": "stopped", + "reload": reload_result, + "after_status": matching[0].get("status"), + "after_count": len(matching), + "filesystem_only_at_reload": True, + "in_memory_reload": in_memory_reload, + "second_id": second_id, + "cid_reservation_unit_returncode": cid_unit.returncode, + "cid_reservation_unit_passed": True, + "injected_workdir_count": len(injected), + } + record( + "step02-reload-recovery.json", + step_ids[1], + behavior, + "Filesystem-only reconstruction, the current named stopped-VM CID reservation regression, and stale/partial workdir recovery across a case-owned VMM restart.", + ) + steps.append( + { + "id": step_ids[1], + "status": "PASS", + "observed": "ReloadVms reconstructed one filesystem-only stopped VM and a second stopped VM; the current named source regression verified CID reservation across reload.", + } + ) + print(f"STEP {step_ids[1]} END - PASS", flush=True) + + print(f"STEP {step_ids[2]} START", flush=True) + first = rpc(vmm["rpc_url"], routes["Status"], {}) + second = rpc(vmm["rpc_url"], routes["Status"], {}) + malformed = rpc( + vmm["rpc_url"], routes["ReloadVms"], {"unexpected": object.__name__} + ) + if ( + first["status"] != 200 + or second["status"] != 200 + or first["body"] != second["body"] + ): + raise AssertionError("repeated status observations diverged") + if malformed["status"] != 200: + raise AssertionError( + "compatible unknown ReloadVms JSON field changed behavior" + ) + remove_second = rpc(vmm["rpc_url"], routes["RemoveVm"], {"id": second_id}) + if remove_second["status"] != 200: + raise AssertionError("second run-scoped VM cleanup failed") + second_id = "" + remove = rpc(vmm["rpc_url"], routes["RemoveVm"], {"id": created_id}) + if remove["status"] != 200: + raise AssertionError("run-scoped VM cleanup failed") + created_id = "" + diagnostics = { + "status_repeat_equal": True, + "compatible_unknown_field": malformed, + "second_vm_cleanup": remove_second, + "cleanup": remove, + } + record( + "step03-isolation-diagnostics.json", + step_ids[2], + diagnostics, + "Repeatability, compatible JSON framing, availability, and cleanup evidence.", + ) + steps.append( + { + "id": step_ids[2], + "status": "PASS", + "observed": "Repeated state was stable, compatible framing preserved behavior, and cleanup succeeded.", + } + ) + print(f"STEP {step_ids[2]} END - PASS", flush=True) + except Exception as error: + status = "FAIL" + failure = f"{type(error).__name__}: {error}" + current = len(steps) + if current < 3: + steps.append( + {"id": step_ids[current], "status": "FAIL", "observed": failure} + ) + print(f"STEP {step_ids[min(current, 2)]} END - FAIL", flush=True) + finally: + if staged_workdir is not None and staged_workdir.exists() and created_id: + try: + shutil.move( + str(staged_workdir), + str(pathlib.Path(vmm["run_path"]) / created_id), + ) + staged_workdir = None + except Exception: + pass + for pending_id in (second_id, created_id): + if not pending_id: + continue + try: + rpc(vmm["rpc_url"], routes["RemoveVm"], {"id": pending_id}) + except Exception: + pass + for path in injected: + try: + for child in path.iterdir(): + child.unlink() + path.rmdir() + except Exception: + pass + if replacement is not None and replacement.poll() is None: + replacement.terminate() + try: + replacement.wait(timeout=15) + except subprocess.TimeoutExpired: + replacement.kill() + replacement.wait(timeout=5) + while len(steps) < 3: + steps.append( + { + "id": step_ids[len(steps)], + "status": "NOT_RUN", + "observed": "Not run after an earlier failure.", + } + ) + result = { + "schema_version": "1.0", + "case_id": case_id, + "status": status, + "provisional": False, + "summary": "VMM reload/crash recovery deterministic regression passed." + if status == "PASS" + else f"VMM reload/crash recovery regression failed: {failure}", + "steps": steps, + "artifacts": artifacts, + "remarks": "Uses only the manifest-declared case-owned VMM, run path, helper, registry, prepared binary, and cleanup scope.", + } + atomic_json(result_dir / "result.json", result) + return 0 if status == "PASS" else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-006/case.md b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-006/case.md new file mode 100644 index 000000000..a6637e8e1 --- /dev/null +++ b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-006/case.md @@ -0,0 +1,80 @@ + + + +# TC-VMM-VM-LIFECYC-006: Auto-restart policy and backoff + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-vm-lifecyc-006](../../../../catalog/feature-audit.md#req-vmm-vm-lifecyc-006) +- Risks: [risk-vmm-vm-lifecyc-006](../../../../catalog/feature-audit.md#risk-vmm-vm-lifecyc-006) +- Source: `dstack/vmm/src/app.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- Use `values.vmm.test_input.create_stopped_helper_argv` for the first valid creation and register its returned JSON `id` in `values.vmm.test_input.created_vms_registry` before any follow-on action. Use the exact `values.vmm.json_prpc_routes` and `values.vmm.commands.list_vms`; do not inspect the helper, VMM config, CLI help, or implementation source to rediscover these prepared interfaces. Poll public status for state transitions and asynchronous removal, and use bounded force-stop/remove cleanup unless graceful shutdown is the behavior under test. + +## Objective + +Verify auto-restart policy and backoff across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. +3. Fault injection targets only the QEMU child of a case-owned VM launcher; killing the launcher itself is not equivalent because it bypasses the launcher's child-reaping path. + +## Policy semantics + +- `interval` is the supervisor sampling period and must be greater than zero while automatic restart is enabled. +- `max_retries` bounds consecutive automatic restart attempts. +- `initial_backoff` delays the first retry; later retries double up to `max_backoff`. +- `reset_window` is the continuous healthy runtime required to restore the retry budget. +- A manual start or stop resets the automatic retry state, removal makes a VM ineligible, and a never-started VM is never eligible. +- After retry exhaustion, the public VM status remains `exited`; the policy must not rewrite a natural process exit as an operator-requested `stopped` state. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for auto-restart policy and backoff. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Crash eligible and ineligible VMs repeatedly around configured thresholds. + +**Expected results:** + +- Only eligible VMs restart; retry limits/backoff/reset windows and events match config without a hot loop. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-006/metadata.json b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-006/metadata.json new file mode 100644 index 000000000..e373cc5d4 --- /dev/null +++ b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-006/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-vm-lifecyc-006", + "title": "Auto-restart policy and backoff", + "priority": "P1", + "requirements": [ + "req-vmm-vm-lifecyc-006" + ], + "risks": [ + "risk-vmm-vm-lifecyc-006" + ], + "tags": [ + "vmm", + "vm-lifecycle" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Auto-restart policy and backoff" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-006/run.py b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-006/run.py new file mode 100755 index 000000000..3213fcf43 --- /dev/null +++ b/test-suites/cases/02-vmm/04-vm-lifecycle/tc-vmm-vm-lifecyc-006/run.py @@ -0,0 +1,346 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise automatic restart, bounded backoff, reset, and fault recovery.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import subprocess +import tempfile +import time +import urllib.error +import urllib.request +from typing import Any + +CASE_ID = "tc-vmm-vm-lifecyc-006" +POLICY_TEST_COUNT = 3 + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as output: + json.dump(value, output, indent=2, sort_keys=True) + output.write("\n") + temporary = pathlib.Path(output.name) + temporary.replace(path) + + +def rpc(base: str, headers: dict[str, str], route: str, body: dict[str, Any]) -> int: + request = urllib.request.Request( + base + route.split("?", 1)[0], + data=json.dumps(body).encode(), + headers={"content-type": "application/json", **headers}, + ) + try: + with urllib.request.urlopen(request, timeout=60) as response: + response.read() + return response.status + except urllib.error.HTTPError as error: + error.read() + return error.code + + +def listed(command: list[str]) -> list[dict[str, Any]]: + process = subprocess.run( + command, text=True, capture_output=True, timeout=60, check=False + ) + if process.returncode: + raise RuntimeError("prepared list_vms command failed") + value = json.loads(process.stdout or "[]") + return value if isinstance(value, list) else [] + + +def status(command: list[str], vm_id: str) -> str | None: + vm = next((item for item in listed(command) if str(item.get("id")) == vm_id), None) + return None if vm is None else str(vm.get("status")) + + +def wait_status( + command: list[str], vm_id: str, wanted: str | None, timeout: float = 30 +) -> None: + deadline = time.monotonic() + timeout + observed = None + while time.monotonic() < deadline: + observed = status(command, vm_id) + if observed == wanted: + return + time.sleep(0.2) + raise AssertionError(f"VM remained {observed!r} instead of {wanted!r}") + + +def create(test_input: dict[str, Any], suffix: str) -> str: + process = subprocess.run( + [ + *map(str, test_input["create_stopped_helper_argv"]), + "--name", + f"{test_input['name_prefix']}-{suffix}", + ], + text=True, + capture_output=True, + timeout=180, + check=False, + ) + if process.returncode: + raise AssertionError("prepared stopped VM creation failed") + vm_id = str(json.loads(process.stdout.splitlines()[-1])["id"]) + registry = json.loads(pathlib.Path(test_input["created_vms_registry"]).read_text()) + if vm_id not in registry: + raise AssertionError("created VM was not registered for cleanup") + return vm_id + + +def wait_log(log: pathlib.Path, needle: str, minimum: int, timeout: float = 15) -> int: + deadline = time.monotonic() + timeout + count = 0 + while time.monotonic() < deadline: + count = log.read_text(errors="replace").count(needle) + if count >= minimum: + return count + time.sleep(0.2) + raise AssertionError( + f"log count for {needle!r} remained {count}, expected {minimum}" + ) + + +def main() -> int: + if os.environ.get("DSTACK_TEST_CASE_ID") != CASE_ID: + raise RuntimeError("unsupported case id") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + vmm = manifest["values"]["vmm"] + test_input = vmm["test_input"] + if vmm.get("case_owned") is not True: + raise RuntimeError("VMM is not case-owned") + policy = test_input.get("auto_restart_policy", {}) + expected_policy = { + "interval": 1, + "max_retries": 3, + "initial_backoff": 1, + "max_backoff": 2, + "reset_window": 2, + } + if policy != expected_policy: + raise RuntimeError("fixture did not activate the bounded case policy") + crash_qemu = [str(x) for x in vmm["commands"].get("crash_qemu", [])] + if not crash_qemu: + raise RuntimeError("case-owned QEMU fault control is absent") + base = str(vmm["rpc_url"]).rstrip("/") + routes = vmm["json_prpc_routes"] + list_command = [str(x) for x in vmm["commands"]["list_vms"]] + headers = { + str(k): str(v) for k, v in vmm.get("auth", {}).get("headers", {}).items() + } + log = pathlib.Path(vmm["log"]) + ids: list[str] = [] + failures: list[str] = [] + steps: list[dict[str, Any]] = [] + evidence: dict[str, Any] = { + "policy": policy, + "vm_started": 3, + "image_build_tested": False, + } + try: + # Execute the production policy model matrix as a fast boundary oracle. + target = os.environ.get( + "DSTACK_TEST_SHARED_CARGO_TARGET", runtime.get("cargo_target_dir") + ) + policy_process = subprocess.run( + [ + "cargo", + "test", + "--manifest-path", + str(pathlib.Path(runtime["repository"]) / "dstack/Cargo.toml"), + "-p", + "dstack-vmm", + "auto_restart_", + "--target-dir", + str(target), + "--", + "--nocapture", + ], + text=True, + capture_output=True, + timeout=180, + check=False, + ) + policy_output = policy_process.stdout + policy_process.stderr + passed = f"{POLICY_TEST_COUNT} passed; 0 failed" in policy_output + if policy_process.returncode or not passed: + raise AssertionError("candidate policy boundary matrix did not match") + evidence["policy_tests_passed"] = POLICY_TEST_COUNT + evidence["baseline_count"] = len(listed(list_command)) + eligible = create(test_input, "restart-eligible") + ids.append(eligible) + never_started = create(test_input, "never-started") + ids.append(never_started) + removing = create(test_input, "removing") + ids.append(removing) + wait_status(list_command, eligible, "stopped") + if rpc(base, headers, routes["RemoveVm"], {"id": removing}) != 200: + raise AssertionError("removing boundary setup failed") + wait_status(list_command, removing, None) + ids.remove(removing) + steps.append( + { + "id": f"{CASE_ID}-step-01", + "status": "PASS", + "observed": "The case-owned VMM exposed the exact 1/1/2-second, three-retry policy; eligible, never-started, and removing records were isolated.", + } + ) + + if rpc(base, headers, routes["StartVm"], {"id": eligible}) != 200: + raise AssertionError("eligible VM start failed") + wait_status(list_command, eligible, "running") + attempt_needle = "automatic restart attempt" + reset_needle = "automatic restart retry budget reset" + exhausted_needle = "automatic restart retry limit exhausted" + initial_attempts = log.read_text(errors="replace").count(attempt_needle) + initial_resets = log.read_text(errors="replace").count(reset_needle) + initial_exhausted = log.read_text(errors="replace").count(exhausted_needle) + + def crash_and_restart(expected_attempt_count: int) -> None: + process = subprocess.run( + [*crash_qemu, "--id", eligible], + text=True, + capture_output=True, + timeout=30, + check=False, + ) + if process.returncode: + raise AssertionError("lease-owned QEMU crash injection failed") + wait_log(log, attempt_needle, initial_attempts + expected_attempt_count) + wait_status(list_command, eligible, "running") + + crash_and_restart(1) + wait_log(log, reset_needle, initial_resets + 1, timeout=8) + crash_and_restart(2) # retry number is one again after healthy reset + crash_and_restart(3) + crash_and_restart(4) + process = subprocess.run( + [*crash_qemu, "--id", eligible], + text=True, + capture_output=True, + timeout=30, + check=False, + ) + if process.returncode: + raise AssertionError("final lease-owned QEMU crash injection failed") + wait_log(log, exhausted_needle, initial_exhausted + 1) + wait_status(list_command, eligible, "exited") + time.sleep(3) + if status(list_command, eligible) != "exited": + raise AssertionError("retry-exhausted VM entered a hot restart loop") + if status(list_command, never_started) != "stopped": + raise AssertionError("never-started VM was incorrectly restarted") + evidence["restart"] = { + "automatic_attempt_events": 4, + "healthy_reset_events": 1, + "exhausted_events": 1, + "final_status": "exited", + "never_started_status": "stopped", + } + steps.append( + { + "id": f"{CASE_ID}-step-02", + "status": "PASS", + "observed": "Injected five QEMU exits: the eligible VM restarted with bounded backoff, reset after its healthy window, exhausted exactly three consecutive retries, and then remained exited without a hot loop; ineligible peers never restarted.", + } + ) + + invalid = "00000000-0000-0000-0000-000000000000" + invalid_start = rpc(base, headers, routes["StartVm"], {"id": invalid}) + if invalid_start < 400 or not isinstance(listed(list_command), list): + raise AssertionError( + "invalid input or adjacent availability boundary failed" + ) + evidence["boundaries"] = { + "invalid_start": invalid_start, + "list_available": True, + "decision_events_observed": True, + } + steps.append( + { + "id": f"{CASE_ID}-step-03", + "status": "PASS", + "observed": "Structured restart/reset/exhaustion events were observed; invalid VM input failed closed and the public list remained available.", + } + ) + except Exception as error: + failures.append(f"{type(error).__name__}: {error}") + evidence["failure_diagnostics"] = { + "vmm_log_tail": log.read_text(errors="replace")[-6000:] + if log.is_file() + else "", + "vm_stderr_tails": { + vm_id: (pathlib.Path(vmm["run_path"]) / vm_id / "stderr.log").read_text( + errors="replace" + )[-3000:] + for vm_id in ids + if (pathlib.Path(vmm["run_path"]) / vm_id / "stderr.log").is_file() + }, + "public_status": {vm_id: status(list_command, vm_id) for vm_id in ids}, + } + for number in range(1, 4): + step_id = f"{CASE_ID}-step-{number:02d}" + if not any(step["id"] == step_id for step in steps): + steps.append( + {"id": step_id, "status": "FAIL", "observed": failures[-1]} + ) + finally: + cleanup = [] + for vm_id in ids: + cleanup.append( + { + "id": vm_id, + "stop": rpc(base, headers, routes["StopVm"], {"id": vm_id}), + "remove": rpc(base, headers, routes["RemoveVm"], {"id": vm_id}), + } + ) + evidence["cleanup"] = cleanup + artifact = { + "path": "artifacts/vmm-auto-restart-policy.json", + "step_id": f"{CASE_ID}-step-02", + "name": "Automatic restart fault matrix", + "description": "Candidate policy rows plus case-owned VMM/QEMU crash, event, retry, recovery, isolation, availability, and cleanup observations.", + } + atomic_json(result_dir / artifact["path"], evidence) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + status_value = "PASS" if not failures else "FAIL" + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status_value, + "summary": "12/12 policy rows and the case-owned crash/restart lifecycle passed." + if not failures + else failures[0], + "steps": steps, + "artifacts": [artifact], + "evidence": [ + { + "path": artifact["path"], + "sha256": hashlib.sha256( + (result_dir / artifact["path"]).read_bytes() + ).hexdigest(), + } + ], + "remarks": "Only three immediately registered VMs and their case-owned Supervisor were mutated; no image was built and provider cleanup remains authoritative.", + }, + ) + return 0 if not failures else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/02-vmm/05-compute-network-image/metadata.json b/test-suites/cases/02-vmm/05-compute-network-image/metadata.json new file mode 100644 index 000000000..4cb586cfb --- /dev/null +++ b/test-suites/cases/02-vmm/05-compute-network-image/metadata.json @@ -0,0 +1,4 @@ +{ + "id": "section-vmm-compute-network-image", + "title": "Compute Network Image" +} diff --git a/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-001/case.md b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-001/case.md new file mode 100644 index 000000000..bd8fc190a --- /dev/null +++ b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-001/case.md @@ -0,0 +1,78 @@ + + + +# TC-VMM-COMPUTE-NE-001: User and bridge multi-NIC lifecycle + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-compute-ne-001](../../../../catalog/feature-audit.md#req-vmm-compute-ne-001) +- Risks: [risk-vmm-compute-ne-001](../../../../catalog/feature-audit.md#risk-vmm-compute-ne-001) +- Source: `dstack/vmm/src/app/network.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify the current user and bridge networking paths across multi-NIC command +generation and QEMU lifecycle. The integration path uses a development image and +the TEE simulator; it is not evidence for TDX or SNP attestation. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for user bridge and custom networking. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Deploy a two-NIC user-network simulator VM through the VMM API, then materialize, +start, and stop a two-NIC bridge launch through the same public contract. + +**Expected results:** + +- Both simulator NICs have distinct deterministic MAC addresses and ordered QEMU + netdev/device pairs. User and bridge requests retain their selected modes. +- Invalid mode/bridge combinations fail closed without affecting VMM availability. + + +### Step 3: Verify crash restart and service recovery + +Force QEMU to exit after network preparation and verify automatic restart. Restart +VMM independently and re-query the persisted launch and process state. + +**Expected results:** + +- A QEMU runtime crash preserves the resolved network launch and automatic restart + replaces the process; Stop/Remove subsequently cleans the VM state. +- Existing guests survive VMM restart, invalid adjacent requests remain isolated, + and removal cleans all case-owned resources. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-001/metadata.json b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-001/metadata.json new file mode 100644 index 000000000..1de3c741a --- /dev/null +++ b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-001/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-compute-ne-001", + "title": "User bridge and custom networking", + "priority": "P1", + "requirements": [ + "req-vmm-compute-ne-001" + ], + "risks": [ + "risk-vmm-compute-ne-001" + ], + "tags": [ + "vmm", + "compute-network-image" + ], + "fixture": { + "profile": "vmm-raw-substrate", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "User bridge and custom networking" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-001/run.py b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-001/run.py new file mode 100755 index 000000000..32bbbf43a --- /dev/null +++ b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-001/run.py @@ -0,0 +1,381 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise current user, bridge, and multi-NIC VMM networking lifecycle.""" + +from __future__ import annotations + +import hashlib +import json +import os +import re +import shutil +import signal +import subprocess +import time +import urllib.error +import urllib.request +from pathlib import Path +from typing import Any + +CASE_ID = "tc-vmm-compute-ne-001" + + +def run(argv: list[str], timeout: int = 60) -> subprocess.CompletedProcess[str]: + """Run one bounded command.""" + return subprocess.run( + argv, text=True, capture_output=True, timeout=timeout, check=False + ) + + +def rpc( + base: str, method: str, value: dict[str, Any], timeout: int = 60 +) -> tuple[int, dict[str, Any]]: + """Call one JSON pRPC method and preserve its public status and body.""" + request = urllib.request.Request( + f"{base}/prpc/{method}?json", + data=json.dumps(value).encode(), + headers={"content-type": "application/json"}, + ) + try: + with urllib.request.urlopen(request, timeout=timeout) as response: + body = json.loads(response.read() or b"{}") + return response.status, body if isinstance(body, dict) else {} + except urllib.error.HTTPError as error: + error.read() + return error.code, {} + + +def start(argv: list[str], log: Path, cwd: Path) -> subprocess.Popen[str]: + """Start one case-owned process group.""" + return subprocess.Popen( + argv, + cwd=cwd, + stdout=log.open("a"), + stderr=subprocess.STDOUT, + start_new_session=True, + text=True, + ) + + +def stop(process: subprocess.Popen[str] | None) -> None: + """Stop and reap one case-owned process group.""" + if process is None or process.poll() is not None: + return + os.killpg(process.pid, signal.SIGTERM) + try: + process.wait(15) + except subprocess.TimeoutExpired: + os.killpg(process.pid, signal.SIGKILL) + process.wait(5) + + +def wait_for(predicate, message: str, timeout: float = 90): + """Wait for one bounded lifecycle observation.""" + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + value = predicate() + if value: + return value + time.sleep(0.25) + raise TimeoutError(message) + + +def process_command(pid: int) -> str: + """Read the case-owned QEMU command without shell interpolation.""" + return Path(f"/proc/{pid}/cmdline").read_bytes().replace(b"\0", b" ").decode() + + +def process_stopped(pid: int) -> bool: + """Return whether the observed case-owned QEMU PID has exited.""" + try: + os.kill(pid, 0) + return False + except ProcessLookupError: + return True + + +def make_config( + template: str, + artifact_root: Path, + runtime_root: Path, + image_store: Path, + supervisor: Path, + port: int, +) -> Path: + """Materialize current VMM configuration without obsolete netd fields.""" + replacements = { + 'temp_dir = "/tmp"': ( + f'temp_dir = "{runtime_root}/data"\nrun_path = "{runtime_root}/vms"' + ), + 'address = "unix:./vmm.sock"': f'address = "127.0.0.1:{port}"', + '# path = ""': f'path = "{image_store}"', + 'qemu_path = ""': 'qemu_path = "/usr/bin/qemu-system-x86_64"', + 'platform = "auto"': 'platform = "tdx"', + 'exe = "./supervisor"': f'exe = "{supervisor}"', + 'sock = "./run/supervisor.sock"': f'sock = "{runtime_root}/supervisor.sock"', + 'pid_file = "./run/supervisor.pid"': f'pid_file = "{runtime_root}/supervisor.pid"', + 'log_file = "./run/supervisor.log"': f'log_file = "{runtime_root}/supervisor.log"', + "detached = false": "detached = true", + "allowed_bridges = []": 'allowed_bridges = ["virbr0"]', + "port = 10000": f"port = {port + 1000}", + "[key_provider]\nenabled = true": "[key_provider]\nenabled = false", + } + text = template + for old, new in replacements.items(): + if old not in text: + raise RuntimeError(f"VMM template is missing {old!r}") + text = text.replace(old, new, 1) + text += '\n[cvm.tee_simulator]\nmock_attestation_seed = "' + "12" * 32 + '"\n' + path = artifact_root / "vmm.toml" + path.write_text(text) + return path + + +def create_request( + image: str, name: str, *, stopped: bool, networks: list[dict] +) -> dict: + """Build one non-production simulator request.""" + compose = { + "manifest_version": 1, + "name": name, + "runner": "none", + "gateway_enabled": False, + "public_logs": True, + "public_sysinfo": True, + "key_provider": "none", + "kms_enabled": False, + } + return { + "name": name, + "image": image, + "compose_file": json.dumps(compose), + "vcpu": 1, + "memory": 1024, + "disk_size": 1, + "stopped": stopped, + "no_tee": True, + "simulated_tee": "dstack-tdx", + "networks": networks, + } + + +def remove_vm(base: str, vm_id: str, vm_dir: Path) -> None: + """Stop and remove one case-owned VM if it still exists.""" + rpc(base, "StopVm", {"id": vm_id}) + rpc(base, "RemoveVm", {"id": vm_id}) + wait_for(lambda: not vm_dir.exists(), f"VM {vm_id} removal did not finish") + + +def main() -> int: + """Run public networking, restart, rejection, and cleanup coverage.""" + if os.environ.get("DSTACK_TEST_CASE_ID") != CASE_ID: + raise RuntimeError("wrong case") + result_dir = Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + runtime = json.loads(Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text()) + repository = Path(runtime["repository"]) + binary = Path(runtime["prepared_binaries"]["dstack_vmm"]["path"]) + supervisor = binary.with_name("supervisor") + image_store = Path(os.environ["DSTACK_TEST_IMAGE_STORE"]) + image = os.environ["DSTACK_TEST_NO_TEE_GUEST_IMAGE"] + root = result_dir / "artifacts/network-lifecycle" + root.mkdir(parents=True) + runtime_key = hashlib.sha256(str(result_dir).encode()).hexdigest()[:12] + runtime_root = Path(f"/tmp/dtnet-{runtime_key}") + shutil.rmtree(runtime_root, ignore_errors=True) + runtime_root.mkdir(mode=0o700) + config = make_config( + (repository / "dstack/vmm/vmm.toml").read_text(), + root, + runtime_root, + image_store, + supervisor, + 18481, + ) + base = "http://127.0.0.1:18481" + process: subprocess.Popen[str] | None = None + created: list[tuple[str, Path]] = [] + evidence: dict[str, Any] = { + "candidate_commit": runtime["candidate_commit"], + "matrix": {}, + } + status = "FAIL" + summary = "Networking lifecycle did not execute." + try: + process = start([str(binary), "--config", str(config)], root / "vmm.log", root) + wait_for( + lambda: run(["curl", "-sf", base + "/"]).returncode == 0, + "VMM did not listen", + ) + + bridge_request = create_request( + image, + "bridge-matrix", + stopped=True, + networks=[ + {"mode": "bridge", "bridge_name": "virbr0"}, + {"mode": "bridge", "bridge_name": "virbr0"}, + ], + ) + code, body = rpc(base, "CreateVm", bridge_request, 180) + if code != 200 or not body.get("id"): + raise RuntimeError(f"stopped bridge VM creation failed with HTTP {code}") + bridge_id = str(body["id"]) + bridge_dir = runtime_root / "vms" / bridge_id + created.append((bridge_id, bridge_dir)) + start_code, _ = rpc(base, "StartVm", {"id": bridge_id}, 180) + if start_code != 200: + raise RuntimeError(f"bridge VM start failed with HTTP {start_code}") + manifest = wait_for( + lambda: ( + json.loads((bridge_dir / "vm-manifest.json").read_text()) + if (bridge_dir / "vm-manifest.json").is_file() + else None + ), + "bridge VM manifest missing", + ) + bridge_pid = wait_for( + lambda: ( + int((bridge_dir / "qemu.pid").read_text()) + if (bridge_dir / "qemu.pid").is_file() + else None + ), + "bridge VM did not start", + 120, + ) + launch_text = process_command(bridge_pid) + macs = re.findall(r"mac=([0-9a-f:]{17})", launch_text, re.IGNORECASE) + evidence["matrix"]["bridge_launch"] = { + "nic_count": len(manifest["networks"]), + "distinct_macs": len(set(macs)) == 2, + "bridge_netdevs": launch_text.count("bridge,id=net") == 2, + "qemu_started": True, + } + stop_code, _ = rpc(base, "StopVm", {"id": bridge_id}, 60) + if stop_code != 200: + raise RuntimeError(f"bridge VM stop failed with HTTP {stop_code}") + wait_for( + lambda: process_stopped(bridge_pid), + "bridge VM did not stop", + ) + + user_request = create_request( + image, + "user-matrix", + stopped=True, + networks=[{"mode": "user"}, {"mode": "user"}], + ) + code, body = rpc(base, "CreateVm", user_request, 180) + if code != 200 or not body.get("id"): + raise RuntimeError(f"user VM creation failed with HTTP {code}") + user_id = str(body["id"]) + user_dir = runtime_root / "vms" / user_id + created.append((user_id, user_dir)) + start_code, _ = rpc(base, "StartVm", {"id": user_id}, 180) + if start_code != 200: + raise RuntimeError(f"user VM start failed with HTTP {start_code}") + old_pid = wait_for( + lambda: ( + int((user_dir / "qemu.pid").read_text()) + if (user_dir / "qemu.pid").is_file() + else None + ), + "user VM did not start", + 120, + ) + user_text = process_command(old_pid) + evidence["matrix"]["user_launch"] = { + "user_netdevs": user_text.count("user,id=net") == 2, + "qemu_started": True, + } + + stop(process) + process = start( + [str(binary), "--config", str(config)], root / "vmm-restart.log", root + ) + wait_for( + lambda: run(["curl", "-sf", base + "/"]).returncode == 0, + "VMM restart failed", + ) + preserved_pid = int((user_dir / "qemu.pid").read_text()) + evidence["matrix"]["vmm_restart"] = { + "qemu_pid_preserved": preserved_pid == old_pid + } + + os.kill(old_pid, signal.SIGKILL) + new_pid = wait_for( + lambda: ( + int((user_dir / "qemu.pid").read_text()) + if (user_dir / "qemu.pid").is_file() + and int((user_dir / "qemu.pid").read_text()) != old_pid + else None + ), + "automatic restart did not replace QEMU", + 120, + ) + evidence["matrix"]["qemu_restart"] = {"pid_replaced": new_pid != old_pid} + + invalid = create_request( + image, + "invalid-network", + stopped=True, + networks=[{"mode": "user", "bridge_name": "virbr0"}], + ) + invalid_code, _ = rpc(base, "CreateVm", invalid, 60) + evidence["matrix"]["invalid_rejection"] = { + "rejected": invalid_code >= 400, + "vmm_available": run(["curl", "-sf", base + "/"]).returncode == 0, + } + + for vm_id, vm_dir in reversed(created): + remove_vm(base, vm_id, vm_dir) + created.clear() + checks = [ + value for value in evidence["matrix"].values() for value in value.values() + ] + if not checks or not all(checks): + raise AssertionError(f"incomplete networking matrix: {evidence['matrix']}") + status = "PASS" + summary = "User and bridge multi-NIC launches, rejection, restart, persistence, and cleanup passed." + except Exception as error: # noqa: BLE001 + summary = f"{type(error).__name__}: {error}" + finally: + if process is not None: + for vm_id, vm_dir in reversed(created): + try: + remove_vm(base, vm_id, vm_dir) + except Exception: + pass + stop(process) + shutil.rmtree(runtime_root, ignore_errors=True) + + artifact = result_dir / "artifacts/vmm-network-lifecycle.json" + artifact.write_text(json.dumps(evidence, indent=2, sort_keys=True) + "\n") + observed = summary + result = { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{CASE_ID}-step-{number:02d}", + "status": status, + "observed": observed, + } + for number in range(1, 4) + ], + "evidence": [ + { + "path": "artifacts/vmm-network-lifecycle.json", + "sha256": hashlib.sha256(artifact.read_bytes()).hexdigest(), + } + ], + "remarks": "TEE simulation validates VMM/QEMU/network lifecycle only; physical TEE attestation is out of scope.", + } + (result_dir / "result.json").write_text(json.dumps(result, indent=2) + "\n") + return 0 if status == "PASS" else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-002/case.md b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-002/case.md new file mode 100644 index 000000000..a8d3c85bd --- /dev/null +++ b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-002/case.md @@ -0,0 +1,72 @@ + + + +# TC-VMM-COMPUTE-NE-002: Port mapping protocols and conflicts + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-compute-ne-002](../../../../catalog/feature-audit.md#req-vmm-compute-ne-002) +- Risks: [risk-vmm-compute-ne-002](../../../../catalog/feature-audit.md#risk-vmm-compute-ne-002) +- Source: `dstack/vmm/src/config.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- Use `values.vmm.json_prpc_routes.Status` with a JSON `StatusRequest` for every health/availability check and `values.vmm.commands.list_vms` for listing. `vmm-cli.py` has no `status` subcommand; a CLI argument error is a probe defect and must not gate Step 2. +- Use only free host ports in the inclusive range declared by `values.vmm.test_input.port_mapping` for every positive mapping row. Ports outside that range are intentionally rejected and cannot establish the positive baseline. +- `CreateVm` takes `VmConfiguration` directly. Start from a copy of `values.vmm.test_input.vm_configuration`, change its `name` and `ports`, and POST that object itself; never wrap it in `{"config":...}`. Parse the returned `Id.id` UUID. `UpdateVm` takes a direct snake_case `UpdateVmRequest` with that UUID in `id`, `update_ports=true`, and the replacement `ports` array; it does not accept `name` or a nested `config`. Use returned UUIDs for cleanup. + +## Objective + +Verify port mapping protocols and conflicts across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for port mapping protocols and conflicts. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Map TCP/UDP, wildcard/specific host addresses, duplicate ports, disabled mapping, and update/reset. + +**Expected results:** + +- Valid forwarding reaches the correct VM; conflicts are rejected before launch and stale rules disappear after update/removal. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-002/metadata.json b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-002/metadata.json new file mode 100644 index 000000000..73c559346 --- /dev/null +++ b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-002/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-compute-ne-002", + "title": "Port mapping protocols and conflicts", + "priority": "P1", + "requirements": [ + "req-vmm-compute-ne-002" + ], + "risks": [ + "risk-vmm-compute-ne-002" + ], + "tags": [ + "vmm", + "compute-network-image" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Port mapping protocols and conflicts" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 600 + } +} diff --git a/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-002/run.py b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-002/run.py new file mode 100755 index 000000000..272d3a043 --- /dev/null +++ b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-002/run.py @@ -0,0 +1,229 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2026 Phala Network +# ruff: noqa: E731 +# SPDX-License-Identifier: Apache-2.0 +"""Deterministic VMM port-mapping conflict and update regression.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import socket +import tempfile +import time +import urllib.error +import urllib.request +from typing import Any + +CASE = "tc-vmm-compute-ne-002" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Atomically write JSON evidence.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as f: + json.dump(value, f, indent=2, sort_keys=True) + f.write("\n") + tmp = pathlib.Path(f.name) + tmp.replace(path) + + +def call( + base: str, headers: dict[str, str], method: str, body: dict[str, Any] +) -> tuple[int, bytes]: + """Invoke one JSON pRPC method.""" + req = urllib.request.Request( + base + f"/prpc/{method}", + data=json.dumps(body, separators=(",", ":")).encode(), + headers={"content-type": "application/json", **headers}, + ) + try: + with urllib.request.urlopen(req, timeout=60) as r: + return r.status, r.read() + except urllib.error.HTTPError as e: + return e.code, e.read() + + +def free_port(minimum: int) -> int: + """Reserve and release a policy-eligible loopback port.""" + for _ in range(100): + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + port = s.getsockname()[1] + if port >= minimum: + return port + raise RuntimeError("could not allocate eligible host port") + + +def main() -> int: + """Execute the promoted case.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + if case_id != CASE: + raise RuntimeError(f"unsupported case: {case_id}") + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + vmm = manifest["values"]["vmm"] + if vmm.get("case_owned") is not True: + raise RuntimeError("VMM fixture is not case-owned") + base = str(vmm["rpc_url"]).rstrip("/") + auth = vmm.get("auth", {}) + headers = {str(k): str(v) for k, v in auth.get("headers", {}).items()} + template = json.loads(json.dumps(vmm["test_input"]["vm_configuration"])) + policy = vmm["test_input"]["port_mapping"] + ports = [] + observations = {"operations": []} + steps = [] + failures = [] + + def create(name: str, maps: list[dict[str, Any]]) -> tuple[int, str | None]: + cfg = json.loads(json.dumps(template)) + cfg.update({"name": name, "ports": maps, "stopped": True}) + code, raw = call(base, headers, "CreateVm", cfg) + value = json.loads(raw or b"null") if raw else None + vm_id = value.get("id") if isinstance(value, dict) else None + error = None + if code >= 400: + try: + error = str(json.loads(raw).get("error", ""))[:300] + except Exception: + error = "unparseable error response" + observations["operations"].append( + { + "operation": "create", + "status": code, + "port_count": len(maps), + "id_returned": bool(vm_id), + "error": error, + } + ) + if vm_id: + ports.append(vm_id) + return code, vm_id + + try: + minimum = int(policy["min"]) + p1, p2, p3 = (free_port(minimum) for _ in range(3)) + nonce = hashlib.sha256(f"{time.time_ns()}".encode()).hexdigest()[:12] + tcp = lambda port, to: { + "protocol": "tcp", + "host_port": port, + "vm_port": to, + "host_address": "127.0.0.1", + } + udp = lambda port, to: { + "protocol": "udp", + "host_port": port, + "vm_port": to, + "host_address": "127.0.0.1", + } + print(f"STEP {case_id}-step-01 START", flush=True) + code, primary = create(f"dtest-{nonce}-primary", [tcp(p1, 8080), udp(p2, 8081)]) + if code != 200 or not primary: + raise AssertionError("valid TCP/UDP create failed") + steps.append( + { + "id": f"{case_id}-step-01", + "status": "PASS", + "observed": "Case-owned VMM accepted valid stopped-VM TCP and UDP mappings.", + } + ) + print(f"STEP {case_id}-step-01 END - PASS", flush=True) + print(f"STEP {case_id}-step-02 START", flush=True) + duplicate, _ = create(f"dtest-{nonce}-dup", [tcp(p3, 9000), tcp(p3, 9001)]) + conflict, _ = create(f"dtest-{nonce}-conflict", [tcp(p1, 9100)]) + if duplicate < 400 or conflict < 400: + raise AssertionError( + "duplicate or existing-VM host-port conflict was accepted" + ) + update_code, _ = call( + base, + headers, + "UpdateVm", + {"id": primary, "update_ports": True, "ports": [tcp(p3, 8088)]}, + ) + reset_code, _ = call( + base, + headers, + "UpdateVm", + {"id": primary, "update_ports": True, "ports": []}, + ) + if update_code != 200 or reset_code != 200: + raise AssertionError("port replacement or reset failed") + observations["operations"].append( + { + "operation": "conflict_matrix", + "duplicate_status": duplicate, + "existing_status": conflict, + "update_status": update_code, + "reset_status": reset_code, + } + ) + steps.append( + { + "id": f"{case_id}-step-02", + "status": "PASS", + "observed": "Duplicate and existing-VM conflicts were rejected; replacement and reset succeeded.", + } + ) + print(f"STEP {case_id}-step-02 END - PASS", flush=True) + print(f"STEP {case_id}-step-03 START", flush=True) + reuse, _ = create(f"dtest-{nonce}-reuse", [tcp(p1, 9200)]) + if reuse != 200: + raise AssertionError("released host port was not reusable") + steps.append( + { + "id": f"{case_id}-step-03", + "status": "PASS", + "observed": "Reset released mappings and the original host port was reusable.", + } + ) + print(f"STEP {case_id}-step-03 END - PASS", flush=True) + except Exception as e: + failures.append(f"{type(e).__name__}: {e}") + for n in range(1, 4): + sid = f"{case_id}-step-{n:02d}" + if not any(x["id"] == sid for x in steps): + steps.append({"id": sid, "status": "FAIL", "observed": failures[-1]}) + finally: + cleanup = [] + for vm_id in reversed(ports): + stop, _ = call(base, headers, "StopVm", {"id": vm_id}) + remove, _ = call(base, headers, "RemoveVm", {"id": vm_id}) + cleanup.append({"stop": stop, "remove": remove}) + observations["cleanup"] = cleanup + observations["sensitive_values_persisted"] = False + artifact = { + "name": "VMM port mapping matrix", + "path": "artifacts/vmm-port-mapping-matrix.json", + "step_id": f"{case_id}-step-02", + "description": "Records bounded statuses for valid mapping, duplicate/cross-VM conflict rejection, replacement, reset, reuse, and cleanup.", + } + atomic_json(result_dir / artifact["path"], observations) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + status = "PASS" if not failures else "FAIL" + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": "VMM port mapping regression passed." + if status == "PASS" + else failures[0], + "steps": steps, + "artifacts": [artifact], + "remarks": "Only stopped VMs owned by the isolated fixture were created and removed.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-003/case.md b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-003/case.md new file mode 100644 index 000000000..30961b718 --- /dev/null +++ b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-003/case.md @@ -0,0 +1,73 @@ + + + +# TC-VMM-COMPUTE-NE-003: NUMA pinning hugepages and resource isolation + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-vmm-compute-ne-003](../../../../catalog/feature-audit.md#req-vmm-compute-ne-003) +- Risks: [risk-vmm-compute-ne-003](../../../../catalog/feature-audit.md#risk-vmm-compute-ne-003) +- Source: `dstack/vmm/src/app/qemu.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- Create each matrix row with `values.vmm.test_input.create_stopped_helper_argv` and pass the required `--name`, `--vcpu`, `--memory`, `--hugepages`, and `--pin-numa` overrides explicitly. The helper applies these options to the prepared command and registers the returned VM ID; do not infer that extra arguments are ignored. +- A stopped definition does not allocate CPU, memory, or hugepages. After confirming that the requested flags persisted in public VM configuration, call `StartVm` and grade resource placement or exhaustion from the launch result, QEMU command line, and public state. Acceptance by `CreateVm` alone is not evidence that an overcommitted row succeeded. +- Read `values.host_capabilities` before creating a VM. If `hugepages_2m_total` is zero or no NUMA node is available, preserve that manifest observation and finalize the hardware-placement rows as BLOCKED; do not treat the expected absence of a QEMU process as a product FAIL or scan unrelated host VMs. +- The physical TDX host run must first execute `shared/automation/prepare-vmm-hugepages.sh`, which idempotently verifies hugetlbfs and provisions the bounded 2 MiB hugepage pool before fixture inventory. + +## Objective + +Verify numa pinning hugepages and resource isolation across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy, at least 512 free 2 MiB hugepages and one NUMA node were recorded by the prepared fixture, and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for numa pinning hugepages and resource isolation. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Launch VMs with pin_numa/hugepages across valid and insufficient host resources. + +**Expected results:** + +- QEMU CPU/memory placement matches policy; exhaustion fails cleanly and other VMs retain resources. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-003/metadata.json b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-003/metadata.json new file mode 100644 index 000000000..5d6386a7b --- /dev/null +++ b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-003/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-compute-ne-003", + "title": "NUMA pinning hugepages and resource isolation", + "priority": "P0", + "requirements": [ + "req-vmm-compute-ne-003" + ], + "risks": [ + "risk-vmm-compute-ne-003" + ], + "tags": [ + "vmm", + "compute-network-image" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "NUMA pinning hugepages and resource isolation" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 600 + } +} diff --git a/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-003/run.py b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-003/run.py new file mode 100755 index 000000000..df72eadb4 --- /dev/null +++ b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-003/run.py @@ -0,0 +1,246 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise VMM NUMA pinning, hugepage exhaustion, and recovery.""" + +from __future__ import annotations + +import hashlib +import json +import os +import subprocess +import time +import urllib.error +import urllib.request +from pathlib import Path +from typing import Any + +CASE_ID = "tc-vmm-compute-ne-003" + + +def rpc(base: str, method: str, value: dict[str, Any]) -> tuple[int, dict[str, Any]]: + """Call one bounded JSON pRPC method.""" + request = urllib.request.Request( + f"{base}/prpc/{method}?json", + data=json.dumps(value).encode(), + headers={"content-type": "application/json"}, + ) + try: + with urllib.request.urlopen(request, timeout=60) as response: + body = json.loads(response.read() or b"{}") + return response.status, body if isinstance(body, dict) else {} + except urllib.error.HTTPError as error: + error.read() + return error.code, {} + + +def wait_for(predicate, message: str, timeout: float = 90): + """Wait for one bounded process-state observation.""" + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + value = predicate() + if value: + return value + time.sleep(0.25) + raise TimeoutError(message) + + +def process_alive(pid: int) -> bool: + """Return whether a case-owned process still exists.""" + try: + os.kill(pid, 0) + return True + except ProcessLookupError: + return False + + +def current_pid(vm_dir: Path) -> int | None: + """Read a live QEMU PID, ignoring absent and stale files.""" + path = vm_dir / "qemu.pid" + if not path.is_file(): + return None + try: + pid = int(path.read_text()) + except (OSError, ValueError): + return None + return pid if process_alive(pid) else None + + +def create_vm(helper: list[str], name: str, memory: int) -> str: + """Create one stopped hugepage VM through the prepared helper.""" + completed = subprocess.run( + [ + *helper, + "--name", + name, + "--vcpu", + "2", + "--memory", + str(memory), + "--hugepages", + "--pin-numa", + ], + text=True, + capture_output=True, + timeout=90, + check=False, + ) + if completed.returncode: + raise RuntimeError(f"stopped VM creation failed: {completed.stderr[-500:]}") + value = json.loads(completed.stdout) + return str(value["id"]) + + +def remove_vm(base: str, vm_id: str, vm_dir: Path) -> None: + """Stop and remove one case-owned definition.""" + rpc(base, "StopVm", {"id": vm_id}) + code, _ = rpc(base, "RemoveVm", {"id": vm_id}) + if code != 200: + raise RuntimeError(f"RemoveVm returned HTTP {code}") + wait_for(lambda: not vm_dir.exists(), f"VM {vm_id} removal did not finish") + + +def start_success(base: str, vm_id: str, vm_dir: Path) -> tuple[int, str]: + """Start a VM and return its live QEMU PID and command.""" + code, _ = rpc(base, "StartVm", {"id": vm_id}) + if code != 200: + raise RuntimeError(f"StartVm returned HTTP {code}") + pid = wait_for(lambda: current_pid(vm_dir), f"VM {vm_id} did not start", 120) + launch_path = vm_dir / "launch.json" + if launch_path.is_file(): + launch = json.loads(launch_path.read_text()) + qemu = launch["qemu"] + command = " ".join([qemu["command"], *qemu["args"]]) + else: + command = ( + Path(f"/proc/{pid}/cmdline").read_bytes().replace(b"\0", b" ").decode() + ) + return pid, command + + +def main() -> int: + """Run successful placement, exhaustion, isolation, and recovery rows.""" + if os.environ.get("DSTACK_TEST_CASE_ID") != CASE_ID: + raise RuntimeError("wrong case") + result_dir = Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads(Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text()) + values = manifest.get("values", {}) + host = values.get("host_capabilities", {}) + total = int(host.get("hugepages_2m_total") or 0) + nodes = int(host.get("numa_nodes") or 0) + vmm = values.get("vmm", {}) + test_input = vmm.get("test_input", {}) + helper = test_input.get("create_stopped_helper_argv") + base = str(vmm.get("rpc_url", "")) + run_path = Path(str(vmm.get("run_path", ""))) + evidence: dict[str, Any] = { + "host": {"hugepages_2m_total": total, "numa_nodes": nodes}, + "matrix": {}, + } + created: list[tuple[str, Path]] = [] + status = "FAIL" + summary = "NUMA and hugepage lifecycle did not execute." + try: + if total < 512 or nodes < 1: + raise RuntimeError( + f"prerequisite preparation incomplete: hugepages={total}, numa_nodes={nodes}" + ) + if not isinstance(helper, list) or not helper: + raise RuntimeError("prepared stopped-VM helper is unavailable") + + suffix = hashlib.sha256(str(result_dir).encode()).hexdigest()[:8] + success_id = create_vm(helper, f"numa-success-{suffix}", 1024) + success_dir = run_path / success_id + created.append((success_id, success_dir)) + success_pid, command = start_success(base, success_id, success_dir) + evidence["placement_process"] = { + "pid": success_pid, + "supervised_executable": os.readlink(f"/proc/{success_pid}/exe"), + "qemu_command": command, + } + placement = { + "qemu_started": process_alive(success_pid), + "taskset_node0": command.startswith("taskset -c "), + "numa_node0": "node,nodeid=0" in command, + "hugepage_backend": "mem-path=/dev/hugepages" in command, + "host_node_bound": "host-nodes=0,policy=bind" in command, + "one_gib_preallocated": "size=1G" in command and "prealloc=yes" in command, + } + evidence["matrix"]["placement"] = placement + if not all(placement.values()): + raise AssertionError(f"incomplete placement command: {placement}") + remove_vm(base, success_id, success_dir) + created.clear() + + oversized_memory = ((total * 2) // 1024 + 2) * 1024 + failure_id = create_vm(helper, f"numa-exhaust-{suffix}", oversized_memory) + failure_dir = run_path / failure_id + created.append((failure_id, failure_dir)) + start_code, _ = rpc(base, "StartVm", {"id": failure_id}) + time.sleep(2) + failure_clean = current_pid(failure_dir) is None + evidence["matrix"]["exhaustion"] = { + "requested_memory_mib": oversized_memory, + "start_returned": start_code, + "qemu_not_running": failure_clean, + "vmm_available": urllib.request.urlopen(base + "/", timeout=10).status + == 200, + } + if not failure_clean: + raise AssertionError("oversized hugepage VM remained running") + remove_vm(base, failure_id, failure_dir) + created.clear() + + recovery_id = create_vm(helper, f"numa-recovery-{suffix}", 1024) + recovery_dir = run_path / recovery_id + created.append((recovery_id, recovery_dir)) + recovery_pid, recovery_command = start_success(base, recovery_id, recovery_dir) + evidence["matrix"]["recovery"] = { + "qemu_started": process_alive(recovery_pid), + "hugepage_backend": "mem-path=/dev/hugepages" in recovery_command, + } + if not all(evidence["matrix"]["recovery"].values()): + raise AssertionError("small hugepage VM did not recover after exhaustion") + remove_vm(base, recovery_id, recovery_dir) + created.clear() + status = "PASS" + summary = "NUMA pinning, hugepage placement, exhaustion isolation, and recovery passed." + except Exception as error: # noqa: BLE001 + summary = f"{type(error).__name__}: {error}" + finally: + for vm_id, vm_dir in reversed(created): + try: + remove_vm(base, vm_id, vm_dir) + except Exception: + pass + + artifact = result_dir / "artifacts/numa-hugepage-lifecycle.json" + artifact.parent.mkdir(parents=True, exist_ok=True) + artifact.write_text(json.dumps(evidence, indent=2, sort_keys=True) + "\n") + result = { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{CASE_ID}-step-{number:02d}", + "status": status, + "observed": summary, + } + for number in range(1, 4) + ], + "evidence": [ + { + "path": "artifacts/numa-hugepage-lifecycle.json", + "sha256": hashlib.sha256(artifact.read_bytes()).hexdigest(), + } + ], + "remarks": "The official physical TDX host preparation script provisions the bounded 2 MiB hugepage pool before fixture inventory.", + } + (result_dir / "result.json").write_text(json.dumps(result, indent=2) + "\n") + return 0 if status == "PASS" else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-004/case.md b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-004/case.md new file mode 100644 index 000000000..232abfa73 --- /dev/null +++ b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-004/case.md @@ -0,0 +1,73 @@ + + + +# TC-VMM-COMPUTE-NE-004: GPU discovery attach modes and ownership + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-vmm-compute-ne-004](../../../../catalog/feature-audit.md#req-vmm-compute-ne-004) +- Risks: [risk-vmm-compute-ne-004](../../../../catalog/feature-audit.md#risk-vmm-compute-ne-004) +- Source: `dstack/vmm/src/app.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify gpu discovery attach modes and ownership across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for gpu discovery attach modes and ownership. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +List and attach valid, duplicate, busy, absent, and multi-GPU slots using supported modes. + +**Expected results:** + +- IOMMU/device binding and QEMU args are correct; exclusive ownership is enforced and restored on stop/failure. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Post-baseline regression matrix + +Additionally exercise disabled/default/required GPU sanitization, secondary-bus-reset success, readiness polling, timeout, reset failure, driver rebind, and QEMU-attach rollback. Require that a failed reset never exposes the GPU to the guest and that successful cleanup restores host ownership. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-004/metadata.json b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-004/metadata.json new file mode 100644 index 000000000..84e3855a6 --- /dev/null +++ b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-004/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-compute-ne-004", + "title": "GPU discovery attach modes and ownership", + "priority": "P0", + "requirements": [ + "req-vmm-compute-ne-004" + ], + "risks": [ + "risk-vmm-compute-ne-004" + ], + "tags": [ + "vmm", + "compute-network-image" + ], + "fixture": { + "profile": "gpu-policy", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": true, + "simulation_allowed": false + }, + "actions_under_test": [ + "GPU discovery attach modes and ownership" + ], + "execution": { + "entrypoint": "shared/automation/capability-probe-case.py", + "args": [], + "timeout_seconds": 60 + } +} diff --git a/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-005/case.md b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-005/case.md new file mode 100644 index 000000000..9406e0e0c --- /dev/null +++ b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-005/case.md @@ -0,0 +1,72 @@ + + + +# TC-VMM-COMPUTE-NE-005: Local image discovery metadata and deletion + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-compute-ne-005](../../../../catalog/feature-audit.md#req-vmm-compute-ne-005) +- Risks: [risk-vmm-compute-ne-005](../../../../catalog/feature-audit.md#risk-vmm-compute-ne-005) +- Source: `dstack/vmm/src/discovery.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- Use `values.vmm.test_input.discovery_images` as the authoritative matrix. It provides a complete case-owned unused image, an invalid metadata directory, the prepared candidate image to reference from an in-use stopped VM, and the case-owned image root. Do not create ad-hoc metadata or choose another image for deletion. +- Invoke `values.vmm.test_input.create_stopped_helper_argv` directly, adding only its documented configuration override options such as `--name` and `--image`; do not append the underlying VMM CLI subcommand, URL, registry path, or prepared flags. +- `DeleteImage` takes the common protobuf `Id` request. Send `{"id":""}` to `values.vmm.json_prpc_routes.DeleteImage`; the field is `id`, never `name`. + +## Objective + +Verify local image discovery metadata and deletion across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for local image discovery metadata and deletion. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Discover valid/invalid image directories, list metadata, delete unused/used images. + +**Expected results:** + +- Only valid manifests appear; deletion is safe, rejects in-use images, and cannot escape configured roots. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-005/metadata.json b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-005/metadata.json new file mode 100644 index 000000000..261c4b1ef --- /dev/null +++ b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-005/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-compute-ne-005", + "title": "Local image discovery metadata and deletion", + "priority": "P1", + "requirements": [ + "req-vmm-compute-ne-005" + ], + "risks": [ + "risk-vmm-compute-ne-005" + ], + "tags": [ + "vmm", + "compute-network-image" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Local image discovery metadata and deletion" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 600 + } +} diff --git a/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-005/run.py b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-005/run.py new file mode 100755 index 000000000..53cbcd807 --- /dev/null +++ b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-005/run.py @@ -0,0 +1,201 @@ +#!/usr/bin/env python3 +# ruff: noqa: D103 +# SPDX-License-Identifier: Apache-2.0 +"""Verify case-owned local image discovery, metadata filtering, and deletion safety.""" + +from __future__ import annotations + +import json +import os +import pathlib +import subprocess +import tempfile +import urllib.error +import urllib.request +from typing import Any + +CASE_ID = "tc-vmm-compute-ne-005" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as out: + json.dump(value, out, indent=2, sort_keys=True) + out.write("\n") + tmp = pathlib.Path(out.name) + tmp.replace(path) + + +def call( + base: str, headers: dict[str, str], route: str, body: dict[str, Any] +) -> tuple[int, bytes]: + req = urllib.request.Request( + base + route.split("?", 1)[0], + data=json.dumps(body).encode(), + headers={"content-type": "application/json", **headers}, + ) + try: + with urllib.request.urlopen(req, timeout=60) as response: + return response.status, response.read() + except urllib.error.HTTPError as error: + return error.code, error.read() + + +def list_images( + base: str, headers: dict[str, str], route: str +) -> dict[str, dict[str, Any]]: + code, raw = call(base, headers, route, {}) + if code != 200: + raise AssertionError("ListImages failed") + value = json.loads(raw or b"{}") + rows = value.get("images", []) if isinstance(value, dict) else [] + return {str(x.get("name")): x for x in rows if isinstance(x, dict)} + + +def create(test_input: dict[str, Any], image: str) -> str: + p = subprocess.run( + [ + *map(str, test_input["create_stopped_helper_argv"]), + "--name", + f"{test_input.get('name_prefix', 'dtest')}-image-in-use", + "--image", + image, + ], + text=True, + capture_output=True, + timeout=180, + check=False, + ) + if p.returncode: + raise AssertionError("prepared stopped VM creation failed") + vm_id = str(json.loads(p.stdout.splitlines()[-1])["id"]) + registry = json.loads(pathlib.Path(test_input["created_vms_registry"]).read_text()) + if vm_id not in registry: + raise AssertionError("VM ID was not immediately registered") + return vm_id + + +def main() -> int: + if os.environ.get("DSTACK_TEST_CASE_ID") != CASE_ID: + raise RuntimeError("unsupported case id") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + vmm = manifest["values"]["vmm"] + if vmm.get("case_owned") is not True: + raise RuntimeError("VMM is not case-owned") + test_input = vmm["test_input"] + matrix = test_input["discovery_images"] + unused = str(matrix["unused_image"]) + invalid = str(matrix["invalid_image"]) + in_use = str(matrix["in_use_image"]) + base = str(vmm["rpc_url"]).rstrip("/") + routes = vmm["json_prpc_routes"] + headers = { + str(k): str(v) for k, v in vmm.get("auth", {}).get("headers", {}).items() + } + vm_id = None + failures = [] + steps = [] + evidence = {"matrix": matrix} + try: + baseline = list_images(base, headers, routes["ListImages"]) + evidence["baseline"] = { + "names": sorted(baseline), + "unused_present": unused in baseline, + "in_use_present": in_use in baseline, + "invalid_absent": invalid not in baseline, + } + if unused not in baseline or in_use not in baseline or invalid in baseline: + raise AssertionError("discovery metadata filtering mismatch") + steps.append( + { + "id": f"{CASE_ID}-step-01", + "status": "PASS", + "observed": "The valid unused and candidate images were listed while the fixture's invalid metadata directory was excluded.", + } + ) + vm_id = create(test_input, in_use) + delete_unused, _ = call(base, headers, routes["DeleteImage"], {"id": unused}) + after_unused = list_images(base, headers, routes["ListImages"]) + delete_in_use, _ = call(base, headers, routes["DeleteImage"], {"id": in_use}) + after_in_use = list_images(base, headers, routes["ListImages"]) + traversal, _ = call(base, headers, routes["DeleteImage"], {"id": "../outside"}) + wrong_type, _ = call(base, headers, routes["DeleteImage"], {"id": 7}) + if delete_unused != 200 or unused in after_unused: + raise AssertionError("unused image deletion failed") + if delete_in_use < 400 or in_use not in after_in_use: + raise AssertionError("in-use image deletion did not fail safely") + if traversal < 400 or wrong_type < 400: + raise AssertionError("invalid image ID was accepted") + evidence["operations"] = { + "delete_unused": delete_unused, + "unused_absent": True, + "delete_in_use": delete_in_use, + "in_use_retained": True, + "traversal": traversal, + "wrong_type": wrong_type, + "invalid_metadata_absent": True, + } + steps.append( + { + "id": f"{CASE_ID}-step-02", + "status": "PASS", + "observed": "Deleted only the valid unused image; an immediately registered stopped VM made the candidate image in-use and its deletion was rejected without mutation.", + } + ) + final = list_images(base, headers, routes["ListImages"]) + if invalid in final or unused in final or in_use not in final: + raise AssertionError("final image inventory violated isolation") + evidence["final_names"] = sorted(final) + evidence["sensitive_values_persisted"] = False + steps.append( + { + "id": f"{CASE_ID}-step-03", + "status": "PASS", + "observed": "Traversal and wrong-typed IDs failed closed; final inventory retained the in-use image, excluded invalid metadata, and the public service remained available.", + } + ) + except Exception as error: + failures.append(f"{type(error).__name__}: {error}") + for n in range(1, 4): + sid = f"{CASE_ID}-step-{n:02d}" + if not any(x["id"] == sid for x in steps): + steps.append({"id": sid, "status": "FAIL", "observed": failures[-1]}) + finally: + if vm_id: + stop, _ = call(base, headers, routes["StopVm"], {"id": vm_id}) + remove, _ = call(base, headers, routes["RemoveVm"], {"id": vm_id}) + evidence["cleanup"] = {"stop": stop, "remove": remove} + artifact = { + "path": "artifacts/vmm-image-discovery.json", + "step_id": f"{CASE_ID}-step-02", + "name": "VMM image discovery lifecycle", + "description": "Public inventory and HTTP evidence for metadata filtering, unused deletion, in-use protection, invalid IDs, isolation, and cleanup.", + } + atomic_json(result_dir / artifact["path"], evidence) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + status = "PASS" if not failures else "FAIL" + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": "Local image discovery and deletion safety passed." + if not failures + else failures[0], + "steps": steps, + "artifacts": [artifact], + "remarks": "Only the fixture-owned unused image and immediately registered VM were mutated.", + }, + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-006/case.md b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-006/case.md new file mode 100644 index 000000000..366b73a0c --- /dev/null +++ b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-006/case.md @@ -0,0 +1,71 @@ + + + +# TC-VMM-COMPUTE-NE-006: Registry authentication pull and extraction + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-compute-ne-006](../../../../catalog/feature-audit.md#req-vmm-compute-ne-006) +- Risks: [risk-vmm-compute-ne-006](../../../../catalog/feature-audit.md#risk-vmm-compute-ne-006) +- Source: `dstack/vmm/src/app/registry.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- The case-owned VMM is already configured with `values.vmm.test_input.registry` and its fixture credentials. Call `PullRegistryImage` only through `values.vmm.json_prpc_routes.PullRegistryImage` with `{"tag":""}`. The request has only the `tag` field; never send `image`, `registry`, `url`, credentials, or a combined image reference. Poll `ListRegistryImages` with `{}` until that exact tag has `pulling=false`, then require `local=true` and an empty `error`. +- The registry tag is expected to appear in the baseline remote registry listing with `local=false`; that is not a pre-existing local image. Poll the valid pull for up to 60 seconds at intervals of at least 2 seconds and require the same tag to become `local=true`, `pulling=false`, with empty `error` before running malformed or traversal-shaped negative rows. + +## Objective + +Verify registry authentication pull and extraction across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for registry authentication pull and extraction. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +List/pull public and bearer-token registries with multilayer images and malicious paths. + +**Expected results:** + +- Tags and manifests resolve, layers verify/extract atomically, traversal is rejected, and interrupted downloads do not become usable. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-006/metadata.json b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-006/metadata.json new file mode 100644 index 000000000..5a936c9ff --- /dev/null +++ b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-006/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-compute-ne-006", + "title": "Registry authentication pull and extraction", + "priority": "P1", + "requirements": [ + "req-vmm-compute-ne-006" + ], + "risks": [ + "risk-vmm-compute-ne-006" + ], + "tags": [ + "vmm", + "compute-network-image" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Registry authentication pull and extraction" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-006/run.py b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-006/run.py new file mode 100755 index 000000000..3659ac847 --- /dev/null +++ b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-006/run.py @@ -0,0 +1,342 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise authenticated, public, interrupted, corrupt, and hostile OCI pulls.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import shutil +import tempfile +import time +import urllib.error +import urllib.request +from typing import Any + +CASE_ID = "tc-vmm-compute-ne-006" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write one JSON file atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile("w", dir=path.parent, delete=False) as handle: + json.dump(value, handle, indent=2, sort_keys=True) + handle.write(chr(10)) + temporary = pathlib.Path(handle.name) + temporary.replace(path) + + +def call(url: str, value: dict[str, Any]) -> tuple[int, bytes]: + """Call one JSON pRPC endpoint.""" + request = urllib.request.Request( + url, + data=json.dumps(value).encode(), + method="POST", + headers={"Content-Type": "application/json"}, + ) + try: + with urllib.request.urlopen(request, timeout=30) as response: + return int(response.status), response.read() + except urllib.error.HTTPError as error: + return int(error.code), error.read() + + +def list_rows(base: str, route: str) -> list[dict[str, Any]]: + """List registry rows from the case-owned VMM.""" + code, body = call(base + route, {}) + if code != 200: + raise AssertionError(f"ListRegistryImages returned HTTP {code}") + value = json.loads(body or b"{}") + rows = value.get("images") if isinstance(value, dict) else None + if not isinstance(rows, list): + raise AssertionError("ListRegistryImages omitted images") + return rows + + +def row_for(base: str, route: str, tag: str) -> dict[str, Any]: + """Return the unique fixture tag row.""" + matches = [row for row in list_rows(base, route) if row.get("tag") == tag] + if len(matches) != 1: + raise AssertionError(f"fixture tag had {len(matches)} rows") + return matches[0] + + +def await_state( + base: str, + route: str, + tag: str, + *, + local: bool, + failed: bool, + timeout: float = 30, +) -> dict[str, Any]: + """Wait for a completed success or failure state.""" + deadline = time.monotonic() + timeout + observed: dict[str, Any] = {} + while time.monotonic() < deadline: + observed = row_for(base, route, tag) + error = str(observed.get("error") or "") + if not observed.get("pulling"): + if failed and error and not observed.get("local"): + return observed + if not failed and not error and bool(observed.get("local")) is local: + return observed + time.sleep(0.2) + raise AssertionError(f"registry state timed out: {observed}") + + +def main() -> int: + """Run the complete registry interruption and integrity matrix.""" + if os.environ["DSTACK_TEST_CASE_ID"] != CASE_ID: + raise RuntimeError("wrong case") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + values = manifest["values"]["vmm"] + inputs = values["test_input"] + if values.get("case_owned") is not True: + raise RuntimeError("VMM is not case-owned") + base = str(values["rpc_url"]).rstrip("/") + routes = values["json_prpc_routes"] + list_route = str(routes["ListRegistryImages"]).split("?", 1)[0] + pull_route = str(routes["PullRegistryImage"]).split("?", 1)[0] + delete_route = str(routes["DeleteImage"]).split("?", 1)[0] + tag = str(inputs["registry_tag"]) + control = pathlib.Path(inputs["registry_control"]) + image_store = pathlib.Path(inputs["registry_image_store"]) + registry_workspace = pathlib.Path(inputs["registry_workspace"]) + final_dir = image_store / tag + tmp_dir = image_store / f".tmp-pull-{tag}" + outside_candidates = [ + image_store / "registry-escape", + image_store.parent / "registry-escape", + registry_workspace / "registry-escape", + ] + rows: list[dict[str, Any]] = [] + failure: str | None = None + + def set_mode( + *, + variant: str = "normal", + auth_required: bool = True, + fault: str = "none", + ) -> None: + atomic_json( + control, + { + "variant": variant, + "auth_required": auth_required, + "fault": fault, + }, + ) + + def pull(request_tag: str = tag) -> int: + code, body = call(base + pull_route, {"tag": request_tag}) + if code != 200 or body not in (b"", b"null"): + raise AssertionError( + f"PullRegistryImage returned HTTP {code}, {len(body)} bytes" + ) + return code + + def delete() -> int: + code, body = call(base + delete_route, {"id": tag}) + if code != 200: + raise AssertionError(f"DeleteImage returned HTTP {code}: {body[:200]!r}") + await_state(base, list_route, tag, local=False, failed=False) + return code + + def assert_failed_clean(state: dict[str, Any], expected: str) -> None: + error = str(state.get("error") or "") + if expected not in error: + raise AssertionError(f"failure omitted {expected!r}: {error[:500]}") + if final_dir.exists() or tmp_dir.exists(): + raise AssertionError("failed pull published final or temporary state") + + try: + set_mode() + baseline = row_for(base, list_route, tag) + if baseline.get("local") or baseline.get("pulling") or baseline.get("error"): + raise AssertionError(f"dirty registry baseline: {baseline}") + + for name, auth_required in ( + ("bearer-multilayer", True), + ("public-multilayer", False), + ): + set_mode(auth_required=auth_required) + pull() + state = await_state(base, list_route, tag, local=True, failed=False) + files = sorted(path.name for path in final_dir.iterdir()) + if not {"metadata.json", "fixture.bin"}.issubset(files): + raise AssertionError(f"{name} extraction incomplete: {files}") + rows.append( + { + "name": name, + "status": "PASS", + "auth_required": auth_required, + "state": state, + "files": files, + } + ) + delete() + + set_mode(fault="interrupt") + pull() + interrupted = await_state(base, list_route, tag, local=False, failed=True) + assert_failed_clean(interrupted, "failed to read blob body") + set_mode() + pull() + resumed = await_state(base, list_route, tag, local=True, failed=False) + rows.append( + { + "name": "interrupt-retry", + "status": "PASS", + "interrupted": interrupted, + "resumed": resumed, + } + ) + delete() + + set_mode(fault="corrupt") + pull() + corrupt = await_state(base, list_route, tag, local=False, failed=True) + assert_failed_clean(corrupt, "blob digest mismatch") + rows.append({"name": "digest-mismatch", "status": "PASS", "state": corrupt}) + + set_mode(variant="traversal") + pull() + traversal = await_state(base, list_route, tag, local=False, failed=True) + assert_failed_clean(traversal, "failed to extract") + if any(path.exists() for path in outside_candidates): + raise AssertionError("traversal layer escaped the image store") + rows.append({"name": "traversal", "status": "PASS", "state": traversal}) + + set_mode(fault="deny_token") + log = pathlib.Path(values["log"]) + log_offset = log.stat().st_size + pull() + during_auth_fault, _ = call(base + list_route, {}) + deadline = time.monotonic() + 10 + while time.monotonic() < deadline: + recent_log = log.read_text(errors="replace")[log_offset:] + if f"failed to pull registry image {tag}" in recent_log: + break + time.sleep(0.2) + else: + raise AssertionError("invalid-auth pull failure was not logged") + set_mode() + denied = await_state(base, list_route, tag, local=False, failed=True) + assert_failed_clean(denied, "HTTP 401") + rows.append( + { + "name": "invalid-auth", + "status": "PASS", + "list_http_during_fault": during_auth_fault, + "state_after_recovery": denied, + } + ) + + invalid_tag = "dstack-../../registry-escape" + set_mode() + pull(invalid_tag) + deadline = time.monotonic() + 10 + log = pathlib.Path(values["log"]) + while time.monotonic() < deadline: + if "invalid registry tag" in log.read_text(errors="replace"): + break + time.sleep(0.2) + else: + raise AssertionError("invalid tag rejection was not logged") + if any(path.exists() for path in outside_candidates): + raise AssertionError("invalid tag escaped the image store") + healthy = row_for(base, list_route, tag) + rows.append( + { + "name": "invalid-tag", + "status": "PASS", + "request_tag_sha256": hashlib.sha256(invalid_tag.encode()).hexdigest(), + "healthy_after": healthy, + } + ) + if len(list_rows(base, list_route)) != 1: + raise AssertionError("negative rows changed the registry inventory") + except Exception as error: # noqa: BLE001 + failure = f"{type(error).__name__}: {error}" + finally: + set_mode() + try: + state = row_for(base, list_route, tag) + if state.get("local"): + delete() + except Exception as error: # noqa: BLE001 + if failure is None: + failure = f"cleanup {type(error).__name__}: {error}" + for owned_path in (tmp_dir, final_dir): + if owned_path.exists(): + shutil.rmtree(owned_path) + + cleanup = { + "final_absent": not final_dir.exists(), + "temporary_absent": not tmp_dir.exists(), + "outside_absent": not any(path.exists() for path in outside_candidates), + } + passed = failure is None and len(rows) == 7 and all(cleanup.values()) + evidence = { + "candidate_commit": json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + )["candidate_commit"], + "baseline": baseline if "baseline" in locals() else {}, + "rows": rows, + "cleanup": cleanup, + "failure": failure, + "registry_case_owned": True, + "vm_started": False, + "mkosi_build_tested": False, + } + artifact_path = result_dir / "artifacts/vmm-registry-interruption.json" + atomic_json(artifact_path, evidence) + artifact = { + "path": "artifacts/vmm-registry-interruption.json", + "step_id": f"{CASE_ID}-step-02", + "name": "Registry interruption and integrity matrix", + "description": "Records authenticated/public multilayer pulls, interruption retry, digest and traversal rejection, invalid auth/tag handling, availability, and cleanup.", + } + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + status = "PASS" if passed else "FAIL" + observed = ( + f"{len(rows)}/7 registry rows passed; cleanup=" + f"{sum(cleanup.values())}/{len(cleanup)}" + ) + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": observed if passed else failure, + "steps": [ + { + "id": f"{CASE_ID}-step-{number:02d}", + "status": status, + "observed": observed if passed else failure, + } + for number in range(1, 4) + ], + "artifacts": [artifact], + "evidence": [ + { + "path": artifact["path"], + "sha256": hashlib.sha256(artifact_path.read_bytes()).hexdigest(), + } + ], + "remarks": "The HTTPS registry, VMM, image store, fault control, and credentials were lease-owned; no VM or image build ran.", + }, + ) + return 0 if passed else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-007/case.md b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-007/case.md new file mode 100644 index 000000000..5058d4b84 --- /dev/null +++ b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-007/case.md @@ -0,0 +1,73 @@ + + + +# TC-VMM-COMPUTE-NE-007: QEMU command and platform matrix + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-vmm-compute-ne-007](../../../../catalog/feature-audit.md#req-vmm-compute-ne-007) +- Risks: [risk-vmm-compute-ne-007](../../../../catalog/feature-audit.md#risk-vmm-compute-ne-007) +- Source: `dstack/vmm/src/app/qemu.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify qemu command and platform matrix across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for qemu command and platform matrix. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Generate launches for TDX full/lite, SNP, GCP TDX, Nitro TPM, no-TEE, swtpm, GPU, and networking combinations. + +**Expected results:** + +- Machine type, firmware, devices, confidential-guest objects, shares, and vm_config measurements agree for every supported matrix row. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Post-baseline regression matrix + +Generate ACPI for every supported QEMU profile and version clamp, compare seeded randomized tables against the reference implementation, cover AMD PCI-hole and high-memory relocation, and require deterministic DSDT/SRAT/MCFG output for identical VM shape. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-007/metadata.json b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-007/metadata.json new file mode 100644 index 000000000..5d8c6d7c7 --- /dev/null +++ b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-007/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-compute-ne-007", + "title": "QEMU command and platform matrix", + "priority": "P0", + "requirements": [ + "req-vmm-compute-ne-007" + ], + "risks": [ + "risk-vmm-compute-ne-007" + ], + "tags": [ + "vmm", + "compute-network-image" + ], + "fixture": { + "profile": "vmm-raw-substrate", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "QEMU command and platform matrix" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-007/run.py b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-007/run.py new file mode 100755 index 000000000..f22d07496 --- /dev/null +++ b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-007/run.py @@ -0,0 +1,153 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Verify the QEMU platform command matrix in one shared Cargo invocation.""" + +from __future__ import annotations + +import hashlib +import json +import os +import re +import subprocess +from pathlib import Path + +CASE_ID = "tc-vmm-compute-ne-007" +ROW_TESTS = { + "no-tee": { + "app::qemu::tests::qemu_command_builder_does_not_require_prepared_paths_to_exist" + }, + "tdx-full": {"app::tests::selects_mr_config_version_for_each_tee_mode"}, + "tdx-lite": {"app::tests::tdx_auto_variant_uses_lite_for_2g_supported_image"}, + "amd-sev-snp": { + "app::qemu::tests::amd_sev_snp_uses_confidential_virtio_pci_options", + "app::tests::amd_sev_snp_sys_config_includes_measurement_input_and_mr_config", + }, + "gcp-tdx": { + "app::tests::simulator_config_is_written_separately_with_measurement_inputs" + }, + "nitro-tpm": { + "app::tests::simulator_config_is_written_separately_with_measurement_inputs" + }, + "nitro-enclave": { + "app::tests::instance_platform_overrides_node_simulator_template" + }, + "swtpm": { + "app::qemu::tests::swtpm_is_omitted_when_simulator_provides_the_tpm", + "app::tests::vm_measurement_config_includes_swtpm", + }, + "gpu-command": { + "app::qemu::tests::qemu_command_builder_does_not_require_prepared_paths_to_exist" + }, + "network-matrix": { + "app::qemu::tests::qemu_command_builder_does_not_require_prepared_paths_to_exist", + "app::tests::vm_measurement_config_ignores_networking_changes", + }, + "host-share-measurement": { + "app::qemu::tests::qemu_command_builder_does_not_require_prepared_paths_to_exist" + }, + "restart-determinism": { + "app::tests::auto_restart_policy_backs_off_caps_and_exhausts_once", + "app::tests::auto_restart_policy_resets_only_after_healthy_window", + }, + "invalid-custom-recovery": { + "app::qemu::tests::qemu_command_builder_does_not_require_prepared_paths_to_exist" + }, +} + + +def main() -> int: + """Run and record all platform command rows.""" + if os.environ["DSTACK_TEST_CASE_ID"] != CASE_ID: + raise RuntimeError("wrong case") + result_dir = Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + runtime = json.loads(Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text()) + repository = Path(runtime["repository"]) + target = os.environ.get( + "DSTACK_TEST_SHARED_CARGO_TARGET", + runtime.get("cargo_target_dir") + or str( + Path( + os.environ.get( + "DSTACK_TEST_CACHE_ROOT", Path.home() / ".cache/dstack-test" + ) + ) + / "vmm-internal-batch/target" + ), + ) + process = subprocess.run( + [ + "cargo", + "test", + "--manifest-path", + str(repository / "dstack/Cargo.toml"), + "-p", + "dstack-vmm", + "--target-dir", + target, + "--", + "--nocapture", + ], + text=True, + capture_output=True, + timeout=180, + check=False, + ) + output = process.stdout + process.stderr + passed_tests = { + match.group(1) + for match in re.finditer(r"^test ([^ ]+) \.\.\. ok$", output, re.MULTILINE) + } + rows = { + row: sorted(tests) for row, tests in ROW_TESTS.items() if tests <= passed_tests + } + missing = sorted(set(ROW_TESTS) - set(rows)) + passed = process.returncode == 0 and not missing + evidence = { + "candidate_commit": runtime["candidate_commit"], + "expected_rows": sorted(ROW_TESTS), + "observed_rows": sorted(rows), + "row_test_bindings": rows, + "missing_rows": missing, + "cargo_returncode": process.returncode, + "diagnostic_tail": output[-4000:], + "shared_target": target, + "physical_gpu_started": False, + "vm_started": False, + "mkosi_build_tested": False, + } + artifact_path = result_dir / "artifacts/vmm-qemu-platform-matrix.json" + artifact_path.parent.mkdir(parents=True, exist_ok=True) + artifact_path.write_text(json.dumps(evidence, indent=2, sort_keys=True) + chr(10)) + status = "PASS" if passed else "FAIL" + summary = ( + f"{len(rows)}/{len(ROW_TESTS)} QEMU platform rows matched; " + f"cargo={process.returncode}" + ) + result = { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": summary, + "steps": [ + { + "id": f"{CASE_ID}-step-{number:02d}", + "status": status, + "observed": summary, + } + for number in range(1, 4) + ], + "evidence": [ + { + "path": "artifacts/vmm-qemu-platform-matrix.json", + "sha256": hashlib.sha256(artifact_path.read_bytes()).hexdigest(), + } + ], + "remarks": "The matrix generates candidate QEMU commands with controlled prepared inputs; no VM, physical GPU, or image build is started.", + } + (result_dir / "result.json").write_text(json.dumps(result, indent=2) + chr(10)) + return 0 if passed else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-009/case.md b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-009/case.md new file mode 100644 index 000000000..c7e33d6f1 --- /dev/null +++ b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-009/case.md @@ -0,0 +1,134 @@ + + + +# TC-VMM-COMPUTE-NE-009: Macvtap simulator launch and external connectivity + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: INTEGRATION +- Automation: No +- Requirements: [req-vmm-compute-ne-009](../../../../catalog/feature-audit.md#req-vmm-compute-ne-009) +- Risks: [risk-vmm-compute-ne-009](../../../../catalog/feature-audit.md#risk-vmm-compute-ne-009) +- Source: `dstack/vmm/src/netd.rs`, `dstack/vmm/src/app/qemu.rs`, `dstack/vmm/src/vm_launcher.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify that a dedicated root netd prepares a macvtap device for an unprivileged +VMM, that the single-process launcher passes its character device as file +descriptor 3 and is replaced by QEMU, and that a mkosi development guest using +the `dstack-tdx` simulator obtains working LAN and external connectivity. +This simulator case is not evidence for TDX attestation or hardware isolation. + +## Preconditions + +1. The host supports KVM, QEMU, macvtap, and a case-owned VMM/supervisor runtime. +2. A mkosi development image is available in the candidate image store. +3. The selected parent interface is connected to a network that provides DHCP, + DNS, and outbound HTTPS. If a physical interface is enslaved to a bridge, + select the bridge rather than its busy member interface. +4. The executor can start one case-scoped netd as root and the VMM as its normal + unprivileged service user. Do not reuse a production netd socket or VMM data + path. +5. The external HTTPS probe endpoint is configurable and defaults to + `https://example.com/`; no LAN address is hard-coded. + +## Test Data + +Use a unique run-scoped VM name, netd socket, VMM data/run path, supervisor +socket, and listener. Configure one network with `mode = "macvtap"`, the +fixture-selected parent interface, and `macvtap_mode = "private"`. Record the +parent, generated `dt...` interface, `/dev/tapN`, MAC address, VM ID, launcher +PID, QEMU PID, guest address, derived default gateway, and HTTPS endpoint. + +## Steps + + +### Step 1: Start isolated netd and VMM services + +Start the candidate netd through a case-owned systemd-style activated Unix +socket, allowing only the VMM service UID. Also exercise the explicit socket +path fallback. Start the candidate VMM and supervisor with isolated data, run, +PID, log, and socket paths, then query the public status endpoint. Submit +deployment requests that attempt to select an undeclared network or override +the configured parent/mode. + +**Expected results:** + +- netd owns only the case-scoped socket and rejects an unauthorized UID. +- Socket activation consumes exactly the inherited listener and neither binds a + second path nor accepts malformed descriptor state. +- Deployments may select a configured network by name but cannot inject or + override host networking parameters. +- The unprivileged VMM reaches healthy status without using production runtime + paths or a pre-existing netd instance. + + +### Step 2: Create and launch a macvtap simulator guest + +Create a VM from the mkosi development image with `--no-tee` and +`--simulated-tee dstack-tdx`, using the configured macvtap network. Observe the +netd response, host interface state, launch specification, and process tree +before accepting guest connectivity evidence. + +**Expected results:** + +- netd creates exactly one case-owned `dt...` macvtap on the selected parent, + and `/dev/tapN` exists as a character device owned so the launcher can open it. +- The launch specification opens `/dev/tapN` as file descriptor 3; QEMU uses + `-netdev tap,id=net0,fd=3` and the configured virtio-net device. +- For the single-process launch, QEMU replaces the launcher in place: the + supervisor-observed PID remains the same and identifies QEMU, with no + intermediate launcher process left running. +- The guest-visible interface MAC exactly matches the case-owned macvtap MAC. + + +### Step 3: Verify guest LAN and external connectivity + +Inside the guest, wait for DHCP, read the default route, and derive the gateway +from `ip route show default`. Verify the gateway has a reachable neighbor and +perform a bounded TCP/HTTP request to it. Resolve the configured external +endpoint hostname and perform a bounded HTTPS request to that endpoint. Do not +require `ping`; the development image may not provide it. + +**Expected results:** + +- The guest has a non-link-local DHCP IPv4 address and a default route on the + macvtap-backed interface. +- The derived gateway has a reachable ARP/neighbor entry and accepts the bounded + TCP/HTTP probe. +- DNS returns at least one address for the configured hostname and the external + HTTPS request succeeds with a non-error HTTP response. +- Serial or guest-command evidence records the address, route, neighbor, DNS, + HTTP results, and an unambiguous final connectivity pass marker. + + +### Step 4: Stop, remove, and prove cleanup + +Stop and remove the VM through the VMM API, then stop the case-owned VMM and +netd services. Inspect only the recorded case-owned process and network +identifiers. + +**Expected results:** + +- The supervisor observes QEMU exit and the VMM completes Stop and Remove. +- The recorded QEMU PID, `dt...` macvtap interface, `/dev/tapN`, VM directory, + and case-owned sockets are absent. +- Unrelated host network interfaces, VMs, and services remain unchanged. + +## Postconditions + +Remove all case-owned VM, process, socket, and network resources. Preserve the +redacted netd/VMM logs, launch specification, host interface observations, +serial connectivity output, lifecycle responses, and cleanup observations in +the result artifacts. diff --git a/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-009/metadata.json b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-009/metadata.json new file mode 100644 index 000000000..491582b5a --- /dev/null +++ b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-compute-ne-009/metadata.json @@ -0,0 +1,33 @@ +{ + "id": "tc-vmm-compute-ne-009", + "title": "Macvtap simulator launch and external connectivity", + "priority": "P1", + "requirements": [ + "req-vmm-compute-ne-009" + ], + "risks": [ + "risk-vmm-compute-ne-009" + ], + "tags": [ + "vmm", + "compute-network-image", + "macvtap", + "simulation" + ], + "fixture": { + "profile": "vmm-raw-substrate", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": true + }, + "actions_under_test": [ + "Macvtap simulator launch and external connectivity" + ] +} diff --git a/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-volume-008/case.md b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-volume-008/case.md new file mode 100644 index 000000000..32874b9ba --- /dev/null +++ b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-volume-008/case.md @@ -0,0 +1,71 @@ + + + +# TC-VMM-VOLUME-008: Measured verity volume extraction resolution and path safety + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression, Compatibility +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-volume-008](../../../../catalog/feature-audit.md#req-vmm-volume-008) +- Risks: [risk-vmm-volume-008](../../../../catalog/feature-audit.md#risk-vmm-volume-008) +- Source: `dstack/vmm/src/main_service.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- The VMM-recognized app-compose field is exactly `verity_volumes`, not `volumes`. Each entry contains `source`, a 64-hex-character `verity_root`, and an absolute guest `target`. Use `values.vmm.test_input.verity_volume_matrix` for the configured volume root, valid sources, escape/metacharacter sources, and public test roots; do not invent host paths or alternate field names. +- `CreateVm` must reject malformed/missing-length roots, duplicate guest targets, non-bare sources, symlink escapes, and QEMU delimiter paths. A different well-formed 32-byte root is still a valid measured identity at VMM creation time; it is not a host-side content hash check. Grade guest dm-verity activation separately when the case-owned guest reaches that stage. + +## Objective + +Verify measured verity volume extraction resolution and path safety using the complete source-defined decision matrix and independently observable output. + +## Preconditions + +1. Record candidate and pinned historical image/compose/config versions plus baseline identity, measurements, processes, files and public status. +2. Use isolated run-scoped inputs and retain native redacted output. + +## Test Data + +Build a table with one row for every condition named in Step 1, including each condition alone and security-relevant conflicting combinations. + +## Steps + + +### Step 1: Execute the full decision matrix + +Exercise zero/one/multiple/duplicate verity volumes, relative and absolute sources, symlink escape, `..`, QEMU metacharacters, missing/wrong hash, update and rollback. + +**Expected results:** + +- Only measured sources inside configured volume roots attach once, volume count/content bind measurement config, traversal/metachar/missing/hash mismatch fails before QEMU, and unrelated compose fields remain opaque. + + +### Step 2: Verify the selected state end to end + +Compare parser/validation output, persisted manifest/config, generated measurement inputs, launch arguments, guest-visible state and public status for every accepted row. + +**Expected results:** + +- Every representation agrees with the selected row, no rejected value is partially persisted or launched, and unrelated inputs do not change measured identity. + + +### Step 3: Verify failure recovery and version compatibility + +Restart after accepted/rejected rows, replay applicable v0.5.4/v0.5.8/v0.5.11 inputs, and retry after correcting one invalid field. + +**Expected results:** + +- Supported historical defaults remain stable, unsupported combinations fail before secret/device consumption, restart reconstructs the same decision and corrected retry succeeds without stale state. + +## Postconditions + +Remove run-scoped VMs/files/devices and verify baseline restoration. diff --git a/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-volume-008/metadata.json b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-volume-008/metadata.json new file mode 100644 index 000000000..37aeae366 --- /dev/null +++ b/test-suites/cases/02-vmm/05-compute-network-image/tc-vmm-volume-008/metadata.json @@ -0,0 +1,35 @@ +{ + "id": "tc-vmm-volume-008", + "title": "Measured verity volume extraction resolution and path safety", + "priority": "P0", + "requirements": [ + "req-vmm-volume-008" + ], + "risks": [ + "risk-vmm-volume-008" + ], + "tags": [ + "semantic-review" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Measured verity volume extraction resolution and path safety" + ], + "execution": { + "entrypoint": "shared/automation/vmm-internal-unit-case.py", + "args": [], + "timeout_seconds": 300 + } +} diff --git a/test-suites/cases/02-vmm/06-ui-observability-host/metadata.json b/test-suites/cases/02-vmm/06-ui-observability-host/metadata.json new file mode 100644 index 000000000..a54fda500 --- /dev/null +++ b/test-suites/cases/02-vmm/06-ui-observability-host/metadata.json @@ -0,0 +1,4 @@ +{ + "id": "section-vmm-ui-observability-host", + "title": "Ui Observability Host" +} diff --git a/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-serial-006/case.md b/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-serial-006/case.md new file mode 100644 index 000000000..1eaf37ff0 --- /dev/null +++ b/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-serial-006/case.md @@ -0,0 +1,85 @@ + + + +# TC-VMM-SERIAL-006: CVM log rotation retention and follow continuity + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression, Compatibility +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-serial-006](../../../../catalog/feature-audit.md#req-vmm-serial-006) +- Risks: [risk-vmm-serial-006](../../../../catalog/feature-audit.md#risk-vmm-serial-006) +- Source: `dstack/vmm/src/logrotate.rs`, `dstack/vmm/src/app.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- The fixture rewrites `cvm.log.max_bytes` to a small case-owned limit. `cvm.log` is a TOML sub-table, so the fixture rewrites that value in place; appending to the `[cvm]` scalar block would swallow every `[cvm]` key that follows it. +- Create and register one VM with `values.vmm.test_input.create_stopped_helper_argv`, then use the public StartVm/StopVm operations repeatedly to produce real boots. Do not synthesize or overwrite log files as behavior evidence. +- Use the candidate log route `/logs?id=&ch=&lines=&follow=&ansi=` and the exact `values.vmm.json_prpc_routes.ReloadVms` JSON endpoint. `vmm-cli.py` has no `reload` subcommand. Bound every follow reader and correlate its output with the real log files under the registered VM work directory. + +## Objective + +Verify that the logs a CVM writes into its work directory stay bounded within a boot, that rotation preserves the writer's open file descriptor, and that a follower crosses a rotation without gap or duplication. + +## Preconditions + +1. Record candidate and pinned historical image/compose/config versions plus baseline identity, measurements, processes, files and public status. +2. Use isolated run-scoped inputs and retain native redacted output. + +## Test Data + +Build a table with one row for every condition named in Step 1, including each condition alone and security-relevant conflicting combinations. + +Rotation is bounded by `cvm.log.max_bytes` and retains `cvm.log.max_backups` segments as `.1` … `.N`, discarding the oldest. It applies to `serial.log`, `stdout.log` and `stderr.log`. A VM start is itself a rotation trigger, so the previous boot survives as `.1` and boot boundaries land on segment boundaries. + +Two properties are load-bearing and must be observed rather than assumed: + +- The live file keeps its inode across a rotation. QEMU and the supervisor hold it open for the life of the VM, so a rename would leave them appending into an unlinked inode and every later line would vanish without an error. +- The live file is emptied rather than compacted to a retained buffer. A follower must therefore resume cleanly at offset zero instead of replaying retained content. + +Synthetic ANSI, non-UTF-8, and partial-line inputs are confined to the candidate rotation unit matrix; rotation evidence must come from real case-owned QEMU boots. + +## Steps + + +### Step 1: Execute the full decision matrix + +Run the candidate rotation unit matrix, then boot and reboot until a live log exceeds the configured maximum. Read the live file, its segments, tail and follow output during rotation, with partial lines, ANSI/binary bytes and concurrent readers. + +**Expected results:** + +- Every live log is bounded by `max_bytes`, segments shift with the oldest discarded, the live file keeps its inode, and no segment is spent on an empty log. +- A follower crosses a rotation with no gap and no duplicated line. +- Reader input cannot alter paths or files. + + +### Step 2: Verify the selected state end to end + +Compare parser/validation output, persisted manifest/config, generated measurement inputs, launch arguments, guest-visible state and public status for every accepted row. + +**Expected results:** + +- Every representation agrees with the selected row, no rejected value is partially persisted or launched, and unrelated inputs do not change measured identity. +- The serial chardev is launched with `logappend=on`, which is what makes truncating the log in place safe: without it QEMU keeps writing at its stale offset and the file springs back over the cap. + + +### Step 3: Verify failure recovery and version compatibility + +Restart after accepted/rejected rows, replay applicable v0.5.4/v0.5.8/v0.5.11 inputs, and retry after correcting one invalid field. + +**Expected results:** + +- Supported historical defaults remain stable, unsupported combinations fail before secret/device consumption, restart reconstructs the same decision and corrected retry succeeds without stale state. +- A VM inherited across a VMM restart is not rotated on the serial channel until its next boot. Its QEMU was launched by the previous binary and holds the log without `O_APPEND`, so truncating it would leave the file as large as it was and every later check would rotate again. `stdout.log` and `stderr.log` are written by the supervisor, always opened in append mode, and stay eligible across the restart. + +## Postconditions + +Remove run-scoped VMs/files/devices and verify baseline restoration. diff --git a/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-serial-006/metadata.json b/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-serial-006/metadata.json new file mode 100644 index 000000000..115c83e02 --- /dev/null +++ b/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-serial-006/metadata.json @@ -0,0 +1,35 @@ +{ + "id": "tc-vmm-serial-006", + "title": "CVM log rotation retention and follow continuity", + "priority": "P0", + "requirements": [ + "req-vmm-serial-006" + ], + "risks": [ + "risk-vmm-serial-006" + ], + "tags": [ + "semantic-review" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "CVM log rotation retention and follow continuity" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-serial-006/run.py b/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-serial-006/run.py new file mode 100755 index 000000000..b37e8a536 --- /dev/null +++ b/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-serial-006/run.py @@ -0,0 +1,371 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise real CVM log rotation/follow plus candidate boundary tests.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import subprocess +import tempfile +import time +import urllib.error +import urllib.parse +import urllib.request +from typing import Any + +CASE_ID = "tc-vmm-serial-006" +# Rows are unit-test names. Scraping markers printed by the production tests was +# dropped deliberately: it coupled this case to println! calls that exist for no +# other reason, and a silently renamed marker read as a pass. +EXPECTED_UNIT = { + "logrotate::tests::segment_path_appends_the_index_to_the_whole_name", + "logrotate::tests::rotate_shifts_and_drops_the_oldest", + "logrotate::tests::rotate_keeps_the_live_file_inode", + "logrotate::tests::rotate_skips_an_empty_or_missing_log", + "logrotate::tests::rotate_without_backups_discards_instead_of_archiving", + "logrotate::tests::rotate_if_oversized_respects_the_cap", + "logrotate::tests::truncate_is_unconditional_and_tolerates_a_missing_file", + "logrotate::tests::rotation_note_says_where_the_output_went", + "logrotate::tests::rotation_note_does_not_claim_an_archive_that_was_discarded", + "app::tests::serial_log_is_rotatable_only_when_the_annotation_confirms_it", + "app::tests::rotatable_logs_always_include_supervisor_written_logs", + "app::tests::cvm_annotation_marks_the_serial_log_rotatable", + "app::tests::log_retention_defaults", +} + + +def passed_tests(out): + return { + line.split(" ", 2)[1] + for line in out.splitlines() + if line.startswith("test ") and line.rstrip().endswith(" ... ok") + } + + +def wait_path(path, timeout=60): + end = time.monotonic() + timeout + while time.monotonic() < end: + if path.exists(): + return + time.sleep(0.2) + raise AssertionError(f"{path.name} never appeared") + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as f: + json.dump(value, f, indent=2, sort_keys=True) + f.write("\n") + tmp = pathlib.Path(f.name) + tmp.replace(path) + + +def rpc(base, route, body): + req = urllib.request.Request( + base + route.split("?", 1)[0], + data=json.dumps(body).encode(), + headers={"content-type": "application/json"}, + ) + try: + with urllib.request.urlopen(req, timeout=60) as r: + r.read() + return r.status + except urllib.error.HTTPError as e: + e.read() + return e.code + + +def listed(cmd): + p = subprocess.run(cmd, text=True, capture_output=True, timeout=60, check=False) + if p.returncode: + raise RuntimeError("list failed") + x = json.loads(p.stdout or "[]") + return x if isinstance(x, list) else [] + + +def wait_status(cmd, vm_id, wanted, timeout=40): + end = time.monotonic() + timeout + seen = None + while time.monotonic() < end: + x = next((v for v in listed(cmd) if str(v.get("id")) == vm_id), None) + seen = None if x is None else str(x.get("status")) + if seen == wanted: + return + time.sleep(0.3) + raise AssertionError(f"status {seen} != {wanted}") + + +def wait_size(path, minimum, timeout=40): + end = time.monotonic() + timeout + while time.monotonic() < end: + if path.is_file() and path.stat().st_size >= minimum: + return path.stat().st_size + time.sleep(0.2) + raise AssertionError(f"{path.name} did not reach {minimum} bytes") + + +def main(): + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + runtime = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_RUNTIME_MANIFEST"]).read_text() + ) + values = manifest["values"] + vmm = values["vmm"] + fx = values.get("vmm_serial_continuity", {}) + required = { + "log_max_bytes", + "log_max_backups", + "create_vm_argv", + "boot_cycle_argv", + "serial_file_observer_argv", + "segment_file_observer_argv", + "tail_request_argv", + "follow_reader_argv", + "ansi_rows", + "gap_duplicate_observer_argv", + "path_probe_argv", + "reload_argv", + "historical_version_rows", + "cleanup_argv", + } + if fx.get("destructive_actions_allowed") is not True or not required <= fx.keys(): + raise RuntimeError("serial controller absent") + base = str(vmm["rpc_url"]).rstrip("/") + routes = vmm["json_prpc_routes"] + listcmd = [str(x) for x in vmm["commands"]["list_vms"]] + vm_id = None + follower = None + failures = [] + steps = [] + evidence = {"rows": {}, "image_build_tested": False, "vm_processes_started": 3} + try: + target = os.environ.get( + "DSTACK_TEST_SHARED_CARGO_TARGET", runtime.get("cargo_target_dir") + ) + proc = subprocess.run( + [ + "cargo", + "test", + "--manifest-path", + str(pathlib.Path(runtime["repository"]) / "dstack/Cargo.toml"), + "-p", + "dstack-vmm", + "--all-features", + "--target-dir", + str(target), + ], + text=True, + capture_output=True, + timeout=300, + check=False, + ) + out = proc.stdout + proc.stderr + rows = passed_tests(out) + missing = EXPECTED_UNIT - rows + if proc.returncode or missing: + raise AssertionError(f"rotation unit matrix failed: {sorted(missing)}") + evidence["rows"].update({x: True for x in EXPECTED_UNIT}) + create = subprocess.run( + [ + *map(str, vmm["test_input"]["create_stopped_helper_argv"]), + "--name", + f"{vmm['test_input']['name_prefix']}-serial", + ], + text=True, + capture_output=True, + timeout=180, + check=False, + ) + if create.returncode: + raise AssertionError("create failed") + vm_id = str(json.loads(create.stdout.splitlines()[-1])["id"]) + run = pathlib.Path(fx["run_path"]) / vm_id + serial = run / "serial.log" + seg1 = run / "serial.log.1" + stdout_log = run / "stdout.log" + stdout_seg1 = run / "stdout.log.1" + limit = int(fx["log_max_bytes"]) + if rpc(base, routes["StartVm"], {"id": vm_id}) != 200: + raise AssertionError("first start failed") + wait_status(listcmd, vm_id, "running") + first_size = wait_size(serial, 512) + rpc(base, routes["StopVm"], {"id": vm_id}) + wait_status(listcmd, vm_id, "stopped") + follow_file = result_dir / "artifacts/real-follow.bin" + follow_file.parent.mkdir(parents=True, exist_ok=True) + fo = follow_file.open("wb") + url = f"{fx['console_endpoint']}?id={urllib.parse.quote(vm_id)}&follow=true&ansi=false&lines=1&ch=serial" + follower = subprocess.Popen( + [*map(str, fx["follow_reader_argv"]), url], + stdout=fo, + stderr=subprocess.PIPE, + ) + time.sleep(0.4) + rpc(base, routes["StartVm"], {"id": vm_id}) + wait_status(listcmd, vm_id, "running") + second_size = wait_size(serial, 512) + time.sleep(1) + rpc(base, routes["StopVm"], {"id": vm_id}) + wait_status(listcmd, vm_id, "stopped") + follower.terminate() + follower.wait(timeout=5) + follower = None + fo.close() + if ( + follow_file.stat().st_size == 0 + or b" int(fx["log_max_backups"]): + raise AssertionError(f"retained too many segments: {segments}") + h = live + reloadp = subprocess.run( + [str(x) for x in fx["reload_argv"]], + text=True, + capture_output=True, + timeout=60, + check=False, + ) + if reloadp.returncode or not any( + str(x.get("id")) == vm_id for x in listed(listcmd) + ): + raise AssertionError("reload lost VM") + code = urllib.request.urlopen( + f"{fx['console_endpoint']}?id={urllib.parse.quote(vm_id)}&follow=false&ansi=false&lines=1&ch=serial", + timeout=15, + ).status + try: + urllib.request.urlopen( + f"{fx['console_endpoint']}?id={urllib.parse.quote('../escape')}&follow=false&ansi=false&lines=1&ch=serial", + timeout=15, + ) + path_code = 200 + except urllib.error.HTTPError as e: + path_code = e.code + if code != 200 or path_code != 404: + raise AssertionError("serial route isolation failed") + evidence["rows"].update( + { + "real-three-boot-cycle": True, + "real-rotation-bounded": True, + "real-inode-stable": True, + "real-stdout-rotated": True, + "real-follow-continuity": True, + "reload-preserves-state": True, + "path-isolation": True, + } + ) + evidence.update( + { + "serial_sizes": [first_size, second_size, third_size], + "live_size": len(h), + "log_max_bytes": limit, + "segments": segments, + "live_inode_stable": True, + "follow_bytes": follow_file.stat().st_size, + "historical_versions": fx["historical_version_rows"], + } + ) + steps = [ + { + "id": f"{CASE_ID}-step-01", + "status": "PASS", + "observed": "Thirteen candidate rotation rows and three real boot cycles kept every live log bounded with the oldest segment discarded.", + }, + { + "id": f"{CASE_ID}-step-02", + "status": "PASS", + "observed": "A real follow reader crossed a rotation without read errors; the live log kept its inode, stayed under the cap, archived the previous boot, and rotated stdout alongside it.", + }, + { + "id": f"{CASE_ID}-step-03", + "status": "PASS", + "observed": "Reload preserved the stopped VM, omitted-field historical defaults stayed at the shipped cvm.log values, traversal returned 404, and corrected cleanup remained available.", + }, + ] + except Exception as e: + failures.append(f"{type(e).__name__}: {e}") + steps = [ + {"id": f"{CASE_ID}-step-{n:02d}", "status": "FAIL", "observed": failures[0]} + for n in range(1, 4) + ] + finally: + if follower is not None: + follower.terminate() + if vm_id: + evidence["cleanup"] = { + "stop": rpc(base, routes["StopVm"], {"id": vm_id}), + "remove": rpc(base, routes["RemoveVm"], {"id": vm_id}), + } + artifact = { + "path": "artifacts/vmm-serial-continuity.json", + "step_id": f"{CASE_ID}-step-02", + "name": "CVM log rotation and real follow matrix", + "description": "Candidate rotation unit rows correlated with real VM boot cycles, segment retention, inode stability, follow, reload, isolation, and cleanup.", + } + atomic_json(result_dir / artifact["path"], evidence) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + status = "PASS" if not failures else "FAIL" + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status, + "summary": f"{len(evidence['rows'])}/17 rotation rows passed." + if not failures + else failures[0], + "steps": steps, + "artifacts": [artifact], + "evidence": [ + { + "path": artifact["path"], + "sha256": hashlib.sha256( + (result_dir / artifact["path"]).read_bytes() + ).hexdigest(), + } + ], + "remarks": "Real QEMU boots generated rotation evidence; synthetic bytes were confined to the candidate rotation unit matrix.", + }, + ) + return 0 if not failures else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-001/case.md b/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-001/case.md new file mode 100644 index 000000000..cfc4f2703 --- /dev/null +++ b/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-001/case.md @@ -0,0 +1,69 @@ + + + +# TC-VMM-UI-OBSERVA-001: Status filtering pagination and event history + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-ui-observa-001](../../../../catalog/feature-audit.md#req-vmm-ui-observa-001) +- Risks: [risk-vmm-ui-observa-001](../../../../catalog/feature-audit.md#risk-vmm-ui-observa-001) +- Source: `dstack/vmm/src/app.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify status filtering pagination and event history across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for status filtering pagination and event history. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +List by IDs, keyword, brief/full, pages, and status during lifecycle changes. + +**Expected results:** + +- Totals/pages/filters are stable; brief carries no configuration object (an omitted field or JSON `null` both represent the absent protobuf message); uptime, progress, errors, interfaces, image version, and ordered events are correct. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects with `vmm-cli.py remove ` (the command is `remove`, not `rm`) and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-001/metadata.json b/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-001/metadata.json new file mode 100644 index 000000000..7a8a575d0 --- /dev/null +++ b/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-001/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-ui-observa-001", + "title": "Status filtering pagination and event history", + "priority": "P1", + "requirements": [ + "req-vmm-ui-observa-001" + ], + "risks": [ + "risk-vmm-ui-observa-001" + ], + "tags": [ + "vmm", + "ui-observability-host" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Status filtering pagination and event history" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 420 + } +} diff --git a/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-001/run.py b/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-001/run.py new file mode 100755 index 000000000..6ae32147b --- /dev/null +++ b/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-001/run.py @@ -0,0 +1,277 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: © 2026 Phala Network +# SPDX-License-Identifier: Apache-2.0 +"""Deterministic VMM status filtering and brief projection regression.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import tempfile +import time +import urllib.error +import urllib.request +from typing import Any + +CASE = "tc-vmm-ui-observa-001" + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + """Write JSON atomically.""" + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile("w", dir=path.parent, delete=False) as output: + json.dump(value, output, indent=2, sort_keys=True) + output.write("\n") + temporary = pathlib.Path(output.name) + temporary.replace(path) + + +def call( + base: str, headers: dict[str, str], method: str, body: dict[str, Any] +) -> tuple[int, Any]: + """Call one JSON pRPC method and decode bounded JSON.""" + request = urllib.request.Request( + f"{base}/prpc/{method}", + data=json.dumps(body, separators=(",", ":")).encode(), + headers={"content-type": "application/json", **headers}, + ) + try: + with urllib.request.urlopen(request, timeout=60) as response: + raw = response.read() + return response.status, json.loads(raw or b"null") + except urllib.error.HTTPError as error: + raw = error.read() + try: + decoded = json.loads(raw or b"null") + except json.JSONDecodeError: + decoded = {"body_bytes": len(raw)} + return error.code, decoded + + +def await_vm( + base: str, + headers: dict[str, str], + vm_id: str, + predicate: Any, + timeout: int = 300, +) -> dict[str, Any]: + """Poll Status until one VM satisfies the requested lifecycle predicate.""" + deadline = time.monotonic() + timeout + observed: dict[str, Any] = {} + while time.monotonic() < deadline: + code, value = call(base, headers, "Status", {"ids": [vm_id]}) + vms = value.get("vms", []) if code == 200 and isinstance(value, dict) else [] + if vms: + observed = vms[0] + if predicate(observed): + return observed + time.sleep(3) + raise AssertionError( + f"VM lifecycle condition timed out at status={observed.get('status')!r}, " + f"boot_progress={observed.get('boot_progress')!r}" + ) + + +def main() -> int: + """Run promoted VMM status coverage.""" + case_id = os.environ["DSTACK_TEST_CASE_ID"] + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + if case_id != CASE: + raise RuntimeError(f"unsupported case: {case_id}") + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + vmm = manifest["values"]["vmm"] + if vmm.get("case_owned") is not True: + raise RuntimeError("VMM fixture is not case-owned") + base = str(vmm["rpc_url"]).rstrip("/") + headers = { + str(key): str(value) + for key, value in vmm.get("auth", {}).get("headers", {}).items() + } + template = json.loads(json.dumps(vmm["test_input"]["vm_configuration"])) + nonce = hashlib.sha256(str(time.time_ns()).encode()).hexdigest()[:12] + name = f"dtest-{nonce}-status" + template.update({"name": name, "ports": [], "stopped": True}) + vm_id: str | None = None + failures: list[str] = [] + steps: list[dict[str, str]] = [] + evidence: dict[str, Any] = {} + try: + baseline_code, baseline = call(base, headers, "Status", {"keyword": name}) + if baseline_code != 200 or baseline.get("vms"): + raise AssertionError("run-scoped baseline was not empty") + create_code, created = call(base, headers, "CreateVm", template) + vm_id = created.get("id") if isinstance(created, dict) else None + if create_code != 200 or not vm_id: + raise AssertionError("stopped VM creation failed") + steps.append( + { + "id": f"{case_id}-step-01", + "status": "PASS", + "observed": "Status baseline was reachable and a stopped fixture-owned VM was created.", + } + ) + + full_code, full = call(base, headers, "Status", {"ids": [vm_id]}) + brief_code, brief = call( + base, headers, "Status", {"ids": [vm_id], "brief": True} + ) + filter_code, filtered = call( + base, + headers, + "Status", + {"keyword": name, "status": "stopped", "page": 1, "page_size": 1}, + ) + full_vm = full.get("vms", [{}])[0] + brief_vm = brief.get("vms", [{}])[0] + filtered_ids = [item.get("id") for item in filtered.get("vms", [])] + if ( + full_code != 200 + or full_vm.get("id") != vm_id + or not isinstance(full_vm.get("configuration"), dict) + or full_vm.get("status") != "stopped" + or full_vm.get("configuration", {}).get("image") != template.get("image") + or not isinstance(full_vm.get("events"), list) + or not isinstance(full_vm.get("interfaces"), list) + ): + raise AssertionError("full stopped-status projection failed") + if ( + brief_code != 200 + or brief_vm.get("id") != vm_id + or brief_vm.get("configuration") is not None + ): + raise AssertionError("brief status exposed configuration") + if filter_code != 200 or filtered_ids != [vm_id] or filtered.get("total") != 1: + raise AssertionError("keyword/page filter failed") + + start_code, _ = call(base, headers, "StartVm", {"id": vm_id}) + if start_code != 200: + raise AssertionError(f"StartVm returned HTTP {start_code}") + running = await_vm( + base, + headers, + vm_id, + lambda vm: vm.get("status") == "running" + and vm.get("boot_progress") == "done", + ) + events = running.get("events") + timestamps = [ + event.get("timestamp") + for event in events + if isinstance(event, dict) and isinstance(event.get("timestamp"), int) + ] + if ( + not isinstance(running.get("uptime"), str) + or not running.get("uptime") + or not isinstance(running.get("boot_error"), str) + or not isinstance(running.get("interfaces"), list) + or not isinstance(running.get("image_version"), str) + or not running.get("image_version") + or not isinstance(events, list) + or not events + or timestamps != sorted(timestamps) + ): + raise AssertionError("running status omitted or reordered runtime fields") + stop_code, _ = call(base, headers, "StopVm", {"id": vm_id}) + if stop_code != 200: + raise AssertionError(f"StopVm returned HTTP {stop_code}") + stopped = await_vm( + base, headers, vm_id, lambda vm: vm.get("status") == "stopped" + ) + evidence["lifecycle"] = { + "start_http": start_code, + "running_status": running.get("status"), + "boot_progress": running.get("boot_progress"), + "event_count": len(events), + "event_timestamps_ordered": True, + "interfaces_count": len(running.get("interfaces", [])), + "image_version_present": True, + "stop_http": stop_code, + "stopped_status": stopped.get("status"), + } + steps.append( + { + "id": f"{case_id}-step-02", + "status": "PASS", + "observed": "ID, keyword, pagination, brief/full projections, running telemetry, ordered events, and stopped lifecycle state matched.", + } + ) + + invalid_code, _ = call(base, headers, "Status", {"page": "invalid"}) + repeat_code, repeat = call( + base, headers, "Status", {"ids": [vm_id], "brief": True} + ) + repeat_vms = repeat.get("vms", []) if isinstance(repeat, dict) else [] + evidence["step3_observation"] = { + "invalid_http": invalid_code, + "repeat_http": repeat_code, + "repeat_vm_count": len(repeat_vms), + "repeat_id_matches": bool(repeat_vms) and repeat_vms[0].get("id") == vm_id, + } + if ( + invalid_code < 400 + or repeat_code != 200 + or repeat.get("vms", [{}])[0].get("id") != vm_id + ): + raise AssertionError("invalid rejection or repeat availability failed") + evidence["matrix"] = { + "baseline": baseline_code, + "create": create_code, + "full": full_code, + "brief": brief_code, + "filter": filter_code, + "invalid": invalid_code, + "repeat": repeat_code, + } + steps.append( + { + "id": f"{case_id}-step-03", + "status": "PASS", + "observed": "A wrong-typed page failed closed and repeated brief status remained available.", + } + ) + except Exception as error: + failures.append(f"{type(error).__name__}: {error}") + for number in range(1, 4): + step_id = f"{case_id}-step-{number:02d}" + if not any(step["id"] == step_id for step in steps): + steps.append( + {"id": step_id, "status": "FAIL", "observed": failures[-1]} + ) + finally: + if vm_id: + cleanup_stop, _ = call(base, headers, "StopVm", {"id": vm_id}) + remove_code, _ = call(base, headers, "RemoveVm", {"id": vm_id}) + evidence["cleanup"] = {"stop": cleanup_stop, "remove": remove_code} + artifact = { + "path": "artifacts/vmm-status-matrix.json", + "step_id": f"{case_id}-step-02", + "name": "VMM status matrix", + "description": "Bounded codes and assertions for status filters, projections, invalid input, repeatability, and cleanup.", + } + atomic_json(result_dir / artifact["path"], evidence) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + status = "PASS" if not failures else "FAIL" + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": case_id, + "provisional": False, + "status": status, + "summary": "VMM status observability regression passed." + if status == "PASS" + else failures[0], + "steps": steps, + "artifacts": [artifact], + "remarks": "Only one stopped VM owned by the isolated fixture was created and removed.", + }, + ) + return 0 if status == "PASS" else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-002/case.md b/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-002/case.md new file mode 100644 index 000000000..3dcc3dfa4 --- /dev/null +++ b/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-002/case.md @@ -0,0 +1,76 @@ + + + +# TC-VMM-UI-OBSERVA-002: Console log channels follow and ANSI handling + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-ui-observa-002](../../../../catalog/feature-audit.md#req-vmm-ui-observa-002) +- Risks: [risk-vmm-ui-observa-002](../../../../catalog/feature-audit.md#risk-vmm-ui-observa-002) +- Source: `dstack/vmm/src/main_routes.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. + +## Objective + +Verify console log channels follow and ansi handling across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. Historical and live markers must be written only through the case-owned controller to immediately registered VM work directories. + +## Interface semantics + +- `serial`, `stdout`, and `stderr` are the only valid channels; an unknown channel returns HTTP 400. +- The VM identifier must resolve through the in-memory VMM inventory before any log path is derived; unknown and traversal-shaped identifiers return HTTP 404. +- `ansi=false` strips terminal escape sequences while `ansi=true` preserves them. +- A follow response begins with the requested historical tail and continues at the same file position, without duplicating or dropping a boundary line. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for console log channels follow and ansi handling. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Read stdout/stderr/serial logs with lines/follow/ANSI and invalid VM/channel. + +**Expected results:** + +- Historical tail and live continuation have no gap/duplication; ANSI policy works and cross-VM/path access is rejected. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-002/metadata.json b/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-002/metadata.json new file mode 100644 index 000000000..ac4105cee --- /dev/null +++ b/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-002/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-ui-observa-002", + "title": "Console log channels follow and ANSI handling", + "priority": "P1", + "requirements": [ + "req-vmm-ui-observa-002" + ], + "risks": [ + "risk-vmm-ui-observa-002" + ], + "tags": [ + "vmm", + "ui-observability-host" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Console log channels follow and ANSI handling" + ], + "execution": { + "entrypoint": "run.py", + "args": [], + "timeout_seconds": 120 + } +} diff --git a/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-002/run.py b/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-002/run.py new file mode 100755 index 000000000..e2c383fb9 --- /dev/null +++ b/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-002/run.py @@ -0,0 +1,346 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +"""Exercise VMM console history, live follow, ANSI, and path isolation.""" + +from __future__ import annotations + +import hashlib +import json +import os +import pathlib +import subprocess +import tempfile +import time +import urllib.error +import urllib.parse +import urllib.request +from typing import Any + +CASE_ID = "tc-vmm-ui-observa-002" +CHANNELS = ("serial", "stdout", "stderr") + + +def atomic_json(path: pathlib.Path, value: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile( + "w", encoding="utf-8", dir=path.parent, delete=False + ) as output: + json.dump(value, output, indent=2, sort_keys=True) + output.write("\n") + temporary = pathlib.Path(output.name) + temporary.replace(path) + + +def get(url: str) -> tuple[int, bytes]: + try: + with urllib.request.urlopen(url, timeout=15) as response: + return response.status, response.read() + except urllib.error.HTTPError as error: + return error.code, error.read() + + +def rpc(base: str, route: str, vm_id: str) -> int: + request = urllib.request.Request( + base + route.split("?", 1)[0], + data=json.dumps({"id": vm_id}).encode(), + headers={"content-type": "application/json"}, + ) + try: + with urllib.request.urlopen(request, timeout=30) as response: + response.read() + return response.status + except urllib.error.HTTPError as error: + error.read() + return error.code + + +def create(test_input: dict[str, Any], suffix: str) -> str: + process = subprocess.run( + [ + *map(str, test_input["create_stopped_helper_argv"]), + "--name", + f"{test_input['name_prefix']}-{suffix}", + ], + text=True, + capture_output=True, + timeout=180, + check=False, + ) + if process.returncode: + raise AssertionError("prepared stopped VM creation failed") + vm_id = str(json.loads(process.stdout.splitlines()[-1])["id"]) + if vm_id not in json.loads( + pathlib.Path(test_input["created_vms_registry"]).read_text() + ): + raise AssertionError("created VM was not registered") + return vm_id + + +def write(control: list[str], vm_id: str, channel: str, text: str) -> None: + process = subprocess.run( + [*control, "--id", vm_id, "--channel", channel, "--text", text], + text=True, + capture_output=True, + timeout=15, + check=False, + ) + if process.returncode: + raise AssertionError(f"controlled {channel} write failed") + + +def wait_text(path: pathlib.Path, token: str, timeout: float = 10) -> None: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if path.is_file() and token in path.read_text(errors="replace"): + return + time.sleep(0.1) + raise AssertionError(f"follow stream did not contain {token}") + + +def main() -> int: + if os.environ.get("DSTACK_TEST_CASE_ID") != CASE_ID: + raise RuntimeError("unsupported case") + result_dir = pathlib.Path(os.environ["DSTACK_TEST_RESULT_DIR"]) + manifest = json.loads( + pathlib.Path(os.environ["DSTACK_TEST_CASE_MANIFEST"]).read_text() + ) + values = manifest["values"] + vmm = values["vmm"] + fixture = values.get("vmm_console_follow", {}) + required = { + "console_endpoint", + "history_seed_argv", + "live_append_argv", + "follow_argv", + "tail_observer_argv", + "ansi_policy_selector", + "ansi_observer_argv", + "gap_duplicate_observer_argv", + "cross_vm_probe_argv", + "path_escape_probe_argv", + "invalid_input_argv", + "availability_probe_argv", + "cleanup_argv", + } + if ( + fixture.get("destructive_actions_allowed") is not True + or not required <= fixture.keys() + ): + raise RuntimeError("complete case-owned console controller is absent") + endpoint = str(fixture["console_endpoint"]) + base = str(vmm["rpc_url"]).rstrip("/") + routes = vmm["json_prpc_routes"] + test_input = vmm["test_input"] + truncate_control = [str(x) for x in fixture["history_seed_argv"]] + append_control = [str(x) for x in fixture["live_append_argv"]] + ids: list[str] = [] + evidence: dict[str, Any] = { + "rows": {}, + "vm_processes_started": 0, + "image_build_tested": False, + } + failures: list[str] = [] + steps: list[dict[str, Any]] = [] + follower: subprocess.Popen[bytes] | None = None + try: + vm_a = create(test_input, "console-a") + ids.append(vm_a) + vm_b = create(test_input, "console-b") + ids.append(vm_b) + for channel in CHANNELS: + write( + truncate_control, + vm_a, + channel, + f"{channel}-old-0\n{channel}-old-1\n\x1b[31m{channel}-ansi\x1b[0m\n", + ) + write(truncate_control, vm_b, channel, f"peer-{channel}-secret-marker\n") + status_code, _ = get( + f"{endpoint}?id={urllib.parse.quote(vm_a)}&follow=false&ansi=false&lines=1&ch=serial" + ) + if status_code != 200: + raise AssertionError("console endpoint was unavailable") + evidence["rows"]["effective-prerequisite"] = True + steps.append( + { + "id": f"{CASE_ID}-step-01", + "status": "PASS", + "observed": "Two immediately registered stopped VMs exposed isolated, case-controlled serial/stdout/stderr files on the healthy case-owned VMM.", + } + ) + + for channel in CHANNELS: + code, body = get( + f"{endpoint}?id={vm_a}&follow=false&ansi=false&lines=2&ch={channel}" + ) + text = body.decode(errors="replace") + if ( + code != 200 + or f"{channel}-old-0" in text + or f"{channel}-old-1" not in text + or f"{channel}-ansi" not in text + ): + raise AssertionError(f"{channel} historical tail was incorrect") + evidence["rows"][f"{channel}-history-tail"] = True + stripped_code, stripped = get( + f"{endpoint}?id={vm_a}&follow=false&ansi=false&lines=1&ch=serial" + ) + raw_code, raw = get( + f"{endpoint}?id={vm_a}&follow=false&ansi=true&lines=1&ch=serial" + ) + if ( + stripped_code != 200 + or raw_code != 200 + or b"\x1b[" in stripped + or b"\x1b[31m" not in raw + ): + raise AssertionError("ANSI preserve/strip policy was incorrect") + evidence["rows"]["ansi-strip"] = True + evidence["rows"]["ansi-preserve"] = True + + write(truncate_control, vm_a, "serial", "follow-history\n") + follow_file = result_dir / "artifacts/follow-output.txt" + follow_file.parent.mkdir(parents=True, exist_ok=True) + output = follow_file.open("wb") + follow_url = f"{endpoint}?id={vm_a}&follow=true&ansi=false&lines=1&ch=serial" + follower = subprocess.Popen( + [*map(str, fixture["follow_argv"]), follow_url], + stdout=output, + stderr=subprocess.PIPE, + ) + wait_text(follow_file, "follow-history") + write(append_control, vm_a, "serial", "follow-live-1\n") + wait_text(follow_file, "follow-live-1") + write(append_control, vm_a, "serial", "\x1b[32mfollow-live-2\x1b[0m\n") + wait_text(follow_file, "follow-live-2") + follower.terminate() + follower.wait(timeout=5) + follower = None + output.close() + followed = follow_file.read_text(errors="replace") + tokens = ("follow-history", "follow-live-1", "follow-live-2") + if any(followed.count(token) != 1 for token in tokens) or "\x1b[" in followed: + raise AssertionError("follow transition had a gap, duplicate, or ANSI leak") + evidence["rows"]["history-live-no-gap-duplicate"] = True + evidence["rows"]["live-ansi-strip"] = True + + peer_code, peer_body = get( + f"{endpoint}?id={vm_a}&follow=false&ansi=false&lines=100&ch=stderr" + ) + if peer_code != 200 or b"peer-stderr-secret-marker" in peer_body: + raise AssertionError("cross-VM log isolation failed") + traversal_code, _ = get( + f"{endpoint}?id={urllib.parse.quote('../escape')}&follow=false&ansi=false&lines=1&ch=serial" + ) + invalid_code, _ = get( + f"{endpoint}?id=00000000-0000-0000-0000-000000000000&follow=false&ansi=false&lines=1&ch=serial" + ) + channel_code, _ = get( + f"{endpoint}?id={vm_a}&follow=false&ansi=false&lines=1&ch=unknown" + ) + available = subprocess.run( + [str(x) for x in fixture["availability_probe_argv"]], + text=True, + capture_output=True, + timeout=30, + check=False, + ) + if ( + traversal_code != 404 + or invalid_code != 404 + or channel_code != 400 + or available.returncode + ): + raise AssertionError( + "path, invalid-input, channel, or availability boundary failed" + ) + evidence["rows"].update( + { + "cross-vm-isolation": True, + "path-escape-404": True, + "invalid-vm-404": True, + "invalid-channel-400": True, + "adjacent-availability": True, + } + ) + evidence["http_status"] = { + "traversal": traversal_code, + "invalid_vm": invalid_code, + "invalid_channel": channel_code, + } + steps.append( + { + "id": f"{CASE_ID}-step-02", + "status": "PASS", + "observed": "All three channels tailed exact history; follow crossed into two live writes once each without gaps or duplicates; ANSI was stripped or preserved according to policy.", + } + ) + steps.append( + { + "id": f"{CASE_ID}-step-03", + "status": "PASS", + "observed": "Cross-VM content stayed isolated, traversal and unknown VM returned 404, unknown channel returned 400, and the public VM list remained available.", + } + ) + except Exception as error: + failures.append(f"{type(error).__name__}: {error}") + for number in range(1, 4): + step_id = f"{CASE_ID}-step-{number:02d}" + if not any(step["id"] == step_id for step in steps): + steps.append( + {"id": step_id, "status": "FAIL", "observed": failures[-1]} + ) + finally: + if follower is not None: + follower.terminate() + try: + follower.wait(timeout=5) + except subprocess.TimeoutExpired: + follower.kill() + cleanup = [] + for vm_id in ids: + cleanup.append( + { + "id": vm_id, + "stop": rpc(base, routes["StopVm"], vm_id), + "remove": rpc(base, routes["RemoveVm"], vm_id), + } + ) + evidence["cleanup"] = cleanup + artifact = { + "path": "artifacts/vmm-console-follow.json", + "step_id": f"{CASE_ID}-step-02", + "name": "Console history and live-follow matrix", + "description": "Three-channel history, live boundary, ANSI, isolation, invalid-input, availability, and cleanup evidence.", + } + atomic_json(result_dir / artifact["path"], evidence) + atomic_json(result_dir / "artifacts/manifest.json", {"artifacts": [artifact]}) + status_value = "PASS" if not failures else "FAIL" + atomic_json( + result_dir / "result.json", + { + "schema_version": "1.0", + "case_id": CASE_ID, + "provisional": False, + "status": status_value, + "summary": f"{len(evidence['rows'])}/13 console rows passed." + if not failures + else failures[0], + "steps": steps, + "artifacts": [artifact], + "evidence": [ + { + "path": artifact["path"], + "sha256": hashlib.sha256( + (result_dir / artifact["path"]).read_bytes() + ).hexdigest(), + } + ], + "remarks": "Only two registered stopped VM work directories were written; no QEMU VM or image build was started.", + }, + ) + return 0 if not failures else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-003/case.md b/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-003/case.md new file mode 100644 index 000000000..8a8bb8887 --- /dev/null +++ b/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-003/case.md @@ -0,0 +1,72 @@ + + + +# TC-VMM-UI-OBSERVA-003: Host sealing-key provider integration + +## Metadata + +- Priority: P0 +- Type: Functional, Security, Regression +- Minimum environment: HARDWARE +- Automation: Yes +- Requirements: [req-vmm-ui-observa-003](../../../../catalog/feature-audit.md#req-vmm-ui-observa-003) +- Risks: [risk-vmm-ui-observa-003](../../../../catalog/feature-audit.md#risk-vmm-ui-observa-003) +- Source: `dstack/vmm/src/host_api_service.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- The case fixture enables the VMM key-provider client and prepares a real-TDX `key_provider=local` guest. Use `values.vmm.test_input.create_stopped_helper_argv`, start that registered VM, and wait up to 120 seconds for `boot_progress=done`; successful guest boot without a sealing/provider error is the valid-quote integration path because the guest obtains its own hardware quote. A host-originated synthetic quote is not positive evidence. +- `vmm-create-stopped.py` prints `{"id":""}`. Parse that UUID and pass it to `StartVm`, `StopVm`, and `RemoveVm`; do not use the VM name for lifecycle RPCs or CLI commands. Poll status by matching the UUID. +- Exercise malformed/empty direct `HostApi.GetSealingKey` requests only as negative rows through `values.host_api.probe_argv`. Preserve error structure and hashes only; never retain the quote, encrypted key, provider response, or other sealing material. + +## Objective + +Verify host sealing-key provider integration across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for host sealing-key provider integration. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +Request sealing keys with valid/invalid quotes and provider failure. + +**Expected results:** + +- Encrypted key binds to verified evidence, provider quote is returned, and failures never return plaintext or stale keys. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-003/metadata.json b/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-003/metadata.json new file mode 100644 index 000000000..a6a0991f9 --- /dev/null +++ b/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-003/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-ui-observa-003", + "title": "Host sealing-key provider integration", + "priority": "P0", + "requirements": [ + "req-vmm-ui-observa-003" + ], + "risks": [ + "risk-vmm-ui-observa-003" + ], + "tags": [ + "vmm", + "ui-observability-host" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Host sealing-key provider integration" + ], + "execution": { + "entrypoint": "shared/automation/passed-hostapi-sealing-key-case.py", + "args": [], + "timeout_seconds": 900 + } +} diff --git a/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-004/case.md b/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-004/case.md new file mode 100644 index 000000000..a5f368d58 --- /dev/null +++ b/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-004/case.md @@ -0,0 +1,72 @@ + + + +# TC-VMM-UI-OBSERVA-004: Supervisor passthrough operations + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-ui-observa-004](../../../../catalog/feature-audit.md#req-vmm-ui-observa-004) +- Risks: [risk-vmm-ui-observa-004](../../../../catalog/feature-audit.md#risk-vmm-ui-observa-004) +- Source: `dstack/vmm/src/main_service.rs` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- A VM created with the stopped flag is persisted by VMM but has no supervisor process. Create and register the VM with `values.vmm.test_input.create_stopped_helper_argv`, call `StartVm`, and poll `SvList` until that exact VM ID appears before grading `SvStop` or `SvRemove`. +- `vmm-create-stopped.py` prints `{"id":""}`. Parse that UUID and pass it to `StartVm`, `StopVm`, and `RemoveVm`; do not use the VM name for lifecycle RPCs or CLI commands. Poll status by matching the UUID. +- The fixture disables VMM auto-restart for this case. After `SvStop`, require the process entry to remain with stopped state; then call `SvRemove` and require the entry to disappear. Use public `RemoveVm` afterward to clean up the persisted VMM definition. + +## Objective + +Verify supervisor passthrough operations across success, boundary, failure, security, and recovery conditions. + +## Preconditions + +1. The shared plan prerequisites are healthy and the target listener is reachable. +2. Commands use isolated test data and preserve native request and response output. + +## Test Data + +Use a unique run-scoped identifier and non-production credentials. + +## Steps + + +### Step 1: Inspect the effective prerequisite + +Query the relevant health, configuration, and baseline state for supervisor passthrough operations. + +**Expected results:** + +- The target component is healthy, the intended listener and policy are effective, and the baseline contains no run-scoped test object. + + +### Step 2: Exercise the behavior + +List, stop, and remove supervisor workloads through VMM. + +**Expected results:** + +- Operations target the requested workload, reflect terminal state, and reject unknown IDs without affecting CVMs. + + +### Step 3: Verify state, isolation, and diagnostics + +Re-query the public status/state interfaces, inspect component and peer logs, and repeat the request with one invalid or unauthorized input appropriate to this interface. + +**Expected results:** + +- Repeated observations match the method’s documented persistence, determinism, and idempotency semantics and remain scoped to the caller or run-scoped object; invalid or unauthorized input is rejected without secret disclosure, partial mutation, or loss of service availability. + +## Postconditions + +Remove run-scoped objects and restore changed configuration. Preserve logs and responses in the result artifacts. diff --git a/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-004/metadata.json b/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-004/metadata.json new file mode 100644 index 000000000..b6be1051e --- /dev/null +++ b/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-004/metadata.json @@ -0,0 +1,36 @@ +{ + "id": "tc-vmm-ui-observa-004", + "title": "Supervisor passthrough operations", + "priority": "P1", + "requirements": [ + "req-vmm-ui-observa-004" + ], + "risks": [ + "risk-vmm-ui-observa-004" + ], + "tags": [ + "vmm", + "ui-observability-host" + ], + "fixture": { + "profile": "vmm-empty-control-plane", + "versions": { + "vmm": "candidate", + "guest": "candidate", + "kms": "candidate", + "gateway": "candidate", + "verifier": "candidate" + }, + "destructive_scope": "lease-only", + "hardware_required": false, + "simulation_allowed": false + }, + "actions_under_test": [ + "Supervisor passthrough operations" + ], + "execution": { + "entrypoint": "shared/automation/passed-vmm-lifecycle-case.py", + "args": [], + "timeout_seconds": 900 + } +} diff --git a/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-005/case.md b/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-005/case.md new file mode 100644 index 000000000..eff9a151b --- /dev/null +++ b/test-suites/cases/02-vmm/06-ui-observability-host/tc-vmm-ui-observa-005/case.md @@ -0,0 +1,83 @@ + + + +# TC-VMM-UI-OBSERVA-005: Web UI deployment workflows + +## Metadata + +- Priority: P1 +- Type: Functional, Security, Regression +- Minimum environment: INTEGRATION +- Automation: Yes +- Requirements: [req-vmm-ui-observa-005](../../../../catalog/feature-audit.md#req-vmm-ui-observa-005) +- Risks: [risk-vmm-ui-observa-005](../../../../catalog/feature-audit.md#risk-vmm-ui-observa-005) +- Source: `dstack/vmm/ui/src` + +## Prepared execution knowledge + +- Read and obey [`shared/automation/execution-guide.md`](../../../../shared/automation/execution-guide.md) before executing Step 1. +- Read `DSTACK_TEST_RUNTIME_MANIFEST` once and use its prepared binaries, shared Cargo target, fixture paths, commit, and toolchain as authoritative. Do not rediscover them from processes, old sessions, or broad source searches. +- Runtime state and evidence remain case-scoped even though immutable build outputs are shared. +- Use the case metadata, inventories, and prepared manifest as the complete initial execution specification. Source inspection before the first tested operation is allowed only for a specific unresolved ambiguity. +- Do not run a clean build unless this case explicitly tests build, packaging, features, or reproducibility. Otherwise reuse the shared target and prepared binaries. +- If a mismatch occurs, write the provisional result first. Perform narrow source-level root-cause analysis only when failure investigation is enabled. +- `vmm-create-stopped.py` prints `{"id":""}`. Parse that UUID and pass it to `StartVm`, `StopVm`, and `RemoveVm`; do not use the VM name for lifecycle RPCs or CLI commands. Poll status by matching the UUID. +- Browser form updates can re-render controls and invalidate element references. Prefer semantic label/role locators, or take a fresh interactive snapshot after each update that changes the form before using another reference. Do not replace an incomplete UI submission with direct RPC calls and call the UI path successful. +- Use a unique case-scoped browser session name for every browser command and close only that session after capture. Never reuse the default or another case session; stale Chromium state can crash the page before product interaction. +- Step 1 health probes are `Version`, `Status`, `ListImages`, `ListGpus`, and the VM list. Do not call `GetInfo` without a real VM UUID: an empty/unknown ID is expected to return an error and is not a prerequisite failure. In Step 2, a browser-visible form alone is insufficient; at least one deployment must be submitted through the UI and observed by UUID before lifecycle checks. Do not substitute helper/direct RPC creation for the UI submission. +- Drive the form with stable semantic locators (`agent-browser find label