feat: add volcano deploy to deploy a whole project in one run

[subnetmarco]

Summary

Third of three stacked PRs; based on PR2. Adds one command that deploys a whole project.

Before, a new Next.js project printed ten commands to run by hand:

volcano start
npm run local:env
volcano variables deploy -f volcano/volcano.local.env
npm run build:functions
volcano functions deploy --all
volcano durable deploy --all
volcano config deploy
volcano config deploy -f volcano/auth.local.yaml
volcano migrations deploy --all -d app
npm run local:frontend

After:

volcano start
volcano deploy --with-migrations

• volcano deploy (local) and volcano cloud deploy run, in order: variables, migrations (only with --with-migrations), functions, durable functions, the frontends listed in .volcano/source-layout.json, and the config manifests. Then they wait for every deployment, print each result and the frontend URLs, and exit non-zero if one failed. --no-wait skips the wait.
• extra inputs come only from flags: --env-file, --no-env-file (for a project whose manifest owns its variables), repeatable -c/--config, -d/--database, --no-frontends
• the env file, manifests, and source layout are read and checked before anything changes; function sources and frontend directories are packaged in their own steps. It refuses:
  • an env file and a manifest variables section together, since applying that section deletes the env file's variables. The refusal points at --no-env-file
  • two manifests that declare the same variables, shared_variables, or frontend_shared_variables section, for the same reason
  • --env-file together with --no-env-file
  • in the cloud, an env file holding the local server's fixed keys or a loopback VOLCANO_API_URL/NEXT_PUBLIC_VOLCANO_API_URL. The starters' volcano/volcano.env holds exactly those, and a cloud frontend would compile the localhost URL in.
  • -d without --with-migrations, which would otherwise skip migrations silently
  • a manifest after the first that declares kind: durable or a frontend's variable_scope/variables. The code steps read those from the first manifest only, and a function created as the wrong kind can't change.
  • -c manifests that leave out volcano/volcano-config.yaml while it declares durable functions the first manifest doesn't, which would create them as standard functions
  • a cloud project that deploys code from Git, instead of failing with 409 after variables and migrations ran. The refusal prints variables deploy -f <env file> and config deploy for the parts a deploy can still handle.
• a failed step names the steps that already ran and prints the commands that finish the rest
• the wait follows a newer deployment of the same resource, both when the tracked one is superseded and when it goes active with a redeploy queued behind it. So a config step that redeploys (for example, on a region change) doesn't end the run early with a false success. A superseded deployment with no newer one in the feed is shown without the success tick.
• each frontend is built with the variable_scope/variables selection from the primary manifest, because NEXT_PUBLIC_* values are compiled into the build
• volcano init prints the one command; for the cloud, volcano cloud deploy --env-file volcano/cloud.env, with a line saying that file holds the cloud project's variables. The base starter's README says the same. The Next.js starters ship .volcano/source-layout.json, a scoped frontend variable selection, and volcano/*.env in the root .gitignore.
• the notes starter's frontend selects NEXT_PUBLIC_VOLCANO_API_URL, NEXT_PUBLIC_VOLCANO_ANON_KEY, and NEXT_PUBLIC_VOLCANO_DATABASE. Frontend builds drop every VOLCANO_* name, so selecting VOLCANO_ANON_KEY built a browser bundle with no key; the cloud E2E caught it. Local builds don't drop the prefix, which is why local mode passed.
• fix: the notes starter's migration is split into one statement per file. Volcano's Postgres proxy refuses multi-statement queries, so volcano migrations deploy was already failing on it; the new local-mode E2E caught it. docs/databases.md and the starter's migrations README now state the rule.
• docs/deploy.md is new; the README quickstart, docs/README.md, and the functions, durable, variables, frontends, and configuration pages point at it
• .volcano/source-layout.json is checked with the same path rules as the Git deploy parser, so a layout the CLI accepts is one Git accepts

Behavior changes outside the new command:

• frontend deploys accept an app_root without its own package.json when a parent package.json builds it with next build <dir>, the layout volcano init nextjs creates. Previously --app-root web on that layout was refused.
• starter .gitignore files list volcano/*.env; the base volcano/.gitignore ignores *.env, so cloud.env stays out of Git too
• frontend uploads always leave out volcano/*.env and volcano.env, like .env*.local, and volcano deploy also leaves out the env file it deployed, wherever it is. A project scaffolded before the .gitignore line, deploying a frontend from the project root, would otherwise upload its service key.
• the base volcano/volcano.env (and .example) carries NEXT_PUBLIC_VOLCANO_API_URL, NEXT_PUBLIC_VOLCANO_ANON_KEY, and NEXT_PUBLIC_VOLCANO_DATABASE with the local values, so every new project has the names a Next.js frontend reads

Open questions:

• a deploy-wide --dry-run is not in v1. volcano cloud config deploy --dry-run still previews config.
• a frontend deploy can only set the all or scoped selection. For an entry with variable_scope: shared, or a missing field, the run warns: the build uses the frontend's current selection, the config step applies the new one, and the next deploy rebuilds with it.

Coordinated changes, all on branch feat/volcano-deploy:

• Hosting: https://github.com/Kong/volcano-hosting/pull/1797 (CLI E2E smoke and docs/public)
• Skills: docs(platform): deploy whole projects with volcano deploy volcano-skills#58

Verification

• make lint
• go test ./...
• unit tests stop the run at each step in turn and check the exact resume hint. They also cover each refusal, local mode accepting the local values, and the wait following a superseded deployment to its failed successor and an active one to a redeploy queued behind it. Further cases cover the shared-selection warning, --no-env-file, and the packager leaving out the Volcano env files and the deployed env file.
• make localmode-e2e against kong/volcano:local-nightly (the smoke takes about 5.5 minutes, inside the 35-minute limit). It covers:
  • a whole-project deploy with a migration, an RPC function, an HTTP function routed under the frontend, a durable function, a frontend rendering a scoped NEXT_PUBLIC_* variable, and a manifest with an auth section. The test invokes the function, reaches the HTTP function through the frontend route, completes a durable execution, and reads the auth setting back with config pull.
  • a re-run with a changed variable, which checks local deploys don't collide with 409, and that the function, the route, and the page see the new value
  • --no-frontends deploying no frontend, --no-wait returning while the build runs, and a broken build exiting non-zero with the frontend's name and its logs command
  • the two-variable-sources refusal, then --no-env-file deploying the manifest's variables
  • init nextjs --example notes followed by deploy --with-migrations, which serves the app with the anon key compiled in and the service key absent
• TestAPIE2ECloudDeploy (new) passed against staging through the hosting repo's cloud-e2e-local, run in a Linux container as in CI. It runs the notes starter's cloud steps and checks that the anon key is compiled into the page and the service key is not a project variable. As a signed-up user it inserts and reads back a note, which works only if the migrations ran. It then re-runs with a durable function, a routed HTTP function, and an auth setting, and checks the route, a durable execution, and the setting through config pull. The test's new first check, that a bare volcano cloud deploy refuses the starter's volcano/volcano.env, was added after that run. It fails before any request, and the hosting CLI E2E lane runs the same refusal.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[subnetmarco]

The overall shape works well. Every input is checked before anything changes, the resume hints are precise, and the E2E coverage in both modes is thorough. The touched packages' unit tests pass locally for me. I'd fix one issue before merging; the rest are smaller.

Fix before merging: the cloud deploy uses the local env file by default. volcano init and base/volcano/README.md both print volcano cloud deploy with no --env-file. That deploys volcano/volcano.env, which holds the local server's values, to the cloud project. This PR also adds NEXT_PUBLIC_VOLCANO_API_URL=http://localhost:8000 to that file, so the cloud frontend gets compiled against localhost. Details are inline on init.go.

Smaller issues (details inline):

• A superseded deployment counts as success. The config step can supersede deployments this run just started, and the run then exits 0 without waiting for the deployments that actually serve.
• Only the first manifest decides which functions are durable. With -c auth.yaml -c volcano-config.yaml, durable functions get created as standard functions. A function's kind can't change after creation.
• -d without --with-migrations is silently ignored.
• Frontend uploads rely on the root .gitignore to keep volcano/*.env out. That works for new starters. A project scaffolded before this release has no such line, so once it adds a .volcano/source-layout.json with "workspace": "", its volcano/volcano.env (including the service key) goes into the frontend's source archive. Since the packager already excludes .env*.local unconditionally, I'd exclude volcano/*.env the same way rather than depend on the user's .gitignore.
• docs/deploy.md should say which env file a cloud deploy should use. That part depends on how the first issue gets resolved.

[subnetmarco]

Addressed all of these; details are in the inline replies. The two without a thread:

• Frontend uploads now always leave out volcano/*.env and volcano.env, next to .env*.local, whatever the project's .gitignore says. TestPackageDirectoryLeavesOutVolcanoEnvFiles covers it, and volcano.env.example is still uploaded.
• docs/deploy.md has a Variables subsection: local values in volcano/volcano.env, cloud values in volcano/cloud.env passed with --env-file, and the refusal. It also documents the -d and later-manifest refusals, and --no-frontends for function-only changes. volcano cloud deploy --help carries the env-file note too.

[swkeever]

Requesting changes for one issue: when -c leaves out volcano/volcano-config.yaml, durable functions declared there are created as standard functions, and a function's kind can't change afterward (inline on plan.go:95).

The rest looks solid: step order is right, refusals happen before anything changes, both modes have E2E, and the strings hosting#1797 asserts and the commands skills#58 teaches all match what this ships. Other findings are inline; none block on their own.

One without a line to anchor: an --env-file outside volcano/ (e.g. a root cloud.env) still lands in frontend uploads that archive the project root. The packager could also exclude the file this run deploys.

[swkeever]

Both blocking issues are fixed. -c volcano/auth.yaml is now refused when volcano/volcano-config.yaml declares durable functions the first manifest doesn't (checkDefaultManifest). The Git refusal prints variables deploy -f <env file> with the file this run already checked for local values. The other earlier items check out too: two manifests declaring the same variable section are refused, --no-env-file ends the rename loop, the wait follows a redeploy queued behind an active deployment, and the deployed env file stays out of frontend uploads.

Build, lint, and the touched packages' unit tests pass locally.

One gap remains in the resume hint (inline on plan.go:547). It needs a first manifest other than the default and a failed functions step, so I'm not blocking on it. It ends in the same unchangeable kind, though.