Skip to content

feat: replace Profile/Compose/K8s model with flat Task scheduler (Phase 1) - #9

Open
czprz wants to merge 16 commits into
mainfrom
feat/task-executor-core
Open

czprz wants to merge 16 commits into
mainfrom
feat/task-executor-core

Conversation

@czprz

@czprz czprz commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Summary

Replaces the Profile/Service/Dep + Docker Compose/Kubernetes rendering architecture with the flat Task-based manifest and scheduler described in docs/design/task-executor-refactor.md (Phase 1 of 3).

Motivation

The old model's strong coupling to Compose/k8s schema versions made adding capability feel like "a new feature every time," and the design doc (reviewed and refined across several prior review cycles — see design/task-executor-refactor-doc branch, PR #8) proposes a simpler, Make-like flat task graph with run:/deps:/ready: instead. This PR builds that core.

Changes

  • internal/config: new Manifest/Task/ReadyCheck schema — task names are top-level devx.yaml keys directly (no profiles:/services:/deps: wrapper). includes: resolves and namespaces another file's tasks (infra:db), with per-file vars: scoping. New Validate/ValidateTools/ValidateSetup.
  • internal/graph: generalized to map[string]config.Task; added Closure() for partial up/run <name> invocations.
  • internal/task (new package — the architectural core): ${VAR} substitution + run:-text inference (ports/names/detached-ness), process and container drivers, os:/sources:/generates: inference helpers, a .devx/tasks.json state registry so status/logs/exec/down can reconstruct what's running across separate CLI invocations, and a levelized parallel scheduler (Up/Run) with --dry-run/--keep-going/--group.
  • internal/doctor: dropped k8s/registry checks, added requires: binary checks, rewired port-collision checking to the new Task model.
  • CLI: up/down/run/status/logs/exec/validate/doctor/tasks/new/init rewritten for the Task model. devx <name> is shorthand for devx run <name>. render/lock/providers/export now print a clear "not supported" error (no compose/k8s artifact to render anymore).
  • devx down correctness fix: the state registry now records each detached process task's process-group PID, and down signals it directly (SIGTERM, then SIGKILL after stopTimeout) — previously a no-op for anything that wasn't a container.
  • Deleted: internal/compose, internal/k8s, internal/providers, internal/lock, and internal/ai/export.go (GenerateDeployment depended on the deleted Profile type; its only caller was already a stub).
  • Docs: docs/manifest.md and schemas/devx.schema.json rewritten for the Task schema; examples/basic/devx.yaml rewritten (db/cache/migrate/api/worker tasks).

Known gaps, flagged explicitly (not silently dropped)

  • The design doc assumed internal/process.Supervisor already existed — it didn't, anywhere in the codebase. internal/task's process/container drivers were built from scratch.
  • The old container: auto|true|false convenience (auto-dispatch a run into the devcontainer) has no Task-model equivalent yet.
  • --watch is 1s mtime polling, not fsnotify; run: executes via the system shell, not an embedded shell interpreter — both intentionally deferred per the design doc's phasing.
  • Cobra CLI migration and the workspace/agents feature (parallel per-branch workspaces + auto-started AI coding agent) are out of scope for this PR — tracked as later phases.

Testing

  • Unit tests added / updated — internal/config, internal/graph, internal/task (new: resolve_test.go, scheduler_test.go), internal/doctor all pass.
  • Integration tests added / updated — not added in this PR; tests/integration/ still targets the old schema and will need a follow-up rewrite.
  • Tested manually — steps:
    • go build ./..., go vet ./..., go test ./..., golangci-lint run ./... all pass clean.
    • Ran the full lifecycle against real Docker on examples/basic/devx.yaml: devx validate (clean), devx up -d (db/cache/migrate/api/worker all start and become ready), devx status (shows all 4 containers running), curl localhost:8080/ (API responds), devx down (all 4 containers stop and are removed).

Checklist

  • go build ./... passes
  • go test ./... passes
  • Docs updated (docs/manifest.md, schemas/devx.schema.json, examples/basic/devx.yaml)
  • CHANGELOG.md updated under [Unreleased]

czprz and others added 3 commits October 28, 2026 15:06
…se 1)

Implements the task-executor-refactor design doc's core: devx.yaml is now
a flat map of task names to task definitions (run/deps/ready/detached/...)
instead of profiles/services/deps rendered to Docker Compose or k8s
manifests. run: executes directly via the system shell — nothing is
rendered to an intermediate format.

New packages:
- internal/config: Manifest/Task/ReadyCheck schema, includes: resolution
  with namespacing and per-file ScopeVars, Validate/ValidateTools/ValidateSetup.
- internal/graph: generalized topological sort + Closure() over
  map[string]config.Task (was *config.Profile-specific).
- internal/task: the new execution engine — ${VAR} substitution and
  run:-text inference (resolve.go), process/container drivers (driver.go),
  os:/sources:/generates: inference helpers, a .devx/tasks.json state
  registry so status/logs/exec/down can reconstruct running tasks across
  separate CLI invocations, and a levelized parallel scheduler (Up/Run)
  with --dry-run/--keep-going/--group support.

CLI: up/down/run/status/logs/exec/validate/doctor/tasks/new/init rewritten
for the Task model; devx <name> is shorthand for devx run <name>; render/
lock/providers/export are stubbed with a clear "not supported" error,
since there's no compose/k8s artifact to render anymore.

devx down now actually stops plain-process detached tasks: the state
registry records each task's process-group PID, and down sends
SIGTERM/SIGKILL to it directly (previously a no-op for anything that
wasn't a container).

Deleted: internal/compose, internal/k8s, internal/providers, internal/lock,
internal/ai/export.go (GenerateDeployment depended on the deleted Profile
type and its only caller was already a stub).

Docs: docs/manifest.md and schemas/devx.schema.json rewritten for the Task
schema; examples/basic/devx.yaml rewritten (db/cache/migrate/api/worker
tasks) and verified end-to-end against real Docker (up -d, status, down).

Known gaps for a later phase (not silently dropped — flagging explicitly):
- internal/process.Supervisor, which the design doc assumed already
  existed, did not exist anywhere in the codebase; internal/task's
  process/container drivers were built from scratch instead.
- The old container: auto|true|false convenience (auto-dispatch a run to
  the devcontainer) has no Task-model equivalent yet.
- --watch is 1s mtime polling, not fsnotify; run: uses the system shell,
  not an embedded shell interpreter — both deferred per the design doc.
- Cobra CLI migration and the workspace/agents feature are out of scope
  for this change.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
In-depth review (code-review agent) of the task-executor-core PR found
four real bugs/gaps, fixed here:

- internal/task/resolve.go: port inference (`-p HOST:CONTAINER`) captured
  the container-internal port instead of the host-reachable one whenever
  they differ (e.g. `-p 8080:80` inferred port 80, not 8080) — only the
  common host==container case worked by accident. Rewrote portPattern to
  capture the whole mapping value and added hostPortFromMapping to pick
  the correct field out of `[ip:]hostPort:containerPort[/proto]`.

- internal/config/manifest.go: an included file's own deps: entries
  (referencing another task defined in that same included file) were
  never namespaced during Load's includes: merge, so two tasks in a
  reusable infra/devx.yaml that depend on each other broke with "unknown
  dependency" after merging. Added namespaceDeps to rewrite intra-include
  deps to their merged name, leaving already-qualified ("ns:task") or
  root-level dep references untouched.

- cmd/devx/cmd_logs.go: streamPrefixed's follow-mode loop had no
  sleep/backoff when a scanner hit EOF with no new data, busy-spinning a
  full CPU core for as long as `devx logs --follow` or attached `devx up`
  ran. Added the same 300ms poll interval internal/task's followReader
  already uses.

- cmd/devx/cmd_down.go: stopTimeout: was documented/validated but never
  read by devx down — a single hardcoded 10s applied to every task
  regardless of a task's own stopTimeout: override. Added stopTimeoutFor
  to resolve each task's own value (skipped under --force, which always
  means immediate SIGKILL).

Also flagged (not fixed — correctly scoped as a later-phase feature,
matching how agents: is already handled): restart: never|on-failure|
always is validated and persisted but devx has no supervisor loop that
watches a detached task for exit and restarts it. Documented as reserved
in docs/manifest.md and schemas/devx.schema.json rather than silently
implying it works.

Added regression tests: TestInfer_HostPortWhenDifferentFromContainerPort,
TestLoad_NamespacesIntraIncludeDeps. Re-verified the full examples/basic
lifecycle against real Docker (validate/up -d/status/curl/down) — the
api task's -p ${API_PORT}:80 mapping is exactly the port-inference bug
scenario and now resolves correctly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@czprz

czprz commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

In-depth review

Ran a thorough code review (concurrency, correctness, dependency-resolution logic, ${VAR} substitution precedence, YAML unmarshalling edge cases, schema/docs drift) over the full diff. Found and fixed 4 real issues:

# Severity Finding Fix
1 High -p HOST:CONTAINER port inference captured the container port, not the host-reachable one, whenever they differ (e.g. -p 8080:80 → inferred 80). Only host==container mappings worked by accident. Rewrote portPattern/added hostPortFromMapping to correctly parse [ip:]hostPort:containerPort[/proto]. Added regression test.
2 High An included file's own deps: between two of its own tasks were never namespaced during includes: merging — broke any reusable module with internal dependencies. Added namespaceDeps to rewrite intra-include deps to their merged name. Added regression test.
3 High devx logs --follow (and attached devx up) busy-spun a full CPU core with no backoff when a tailed file had no new data. Added the same 300ms poll interval already used by internal/task's followReader.
4 Medium stopTimeout: was documented and validated but never actually read by devx down — always used a hardcoded 10s. devx down now resolves each task's own stopTimeout: (skipped under --force, which is always immediate).

Also flagged, not fixed (correctly out of scope for this phase, now documented as such rather than silently implied as working): restart: never\|on-failure\|always is validated/persisted but devx has no supervisor loop watching a detached task for exit — a crashed task with restart: always just stays dead. Marked "reserved for a later phase" in docs/manifest.md/schemas/devx.schema.json, same treatment as agents:.

All fixes verified: go build/vet/test/lint clean, plus re-ran the full real-Docker lifecycle against examples/basic/devx.yaml (whose api task's -p ${API_PORT}:80 mapping is exactly finding #1's bug scenario) — validate → up -d → status → curl localhost:8080/ → down all behave correctly.

See commit da314a2 for details.

czprz and others added 13 commits October 28, 2026 15:35
- tests/integration/cli_test.go, devx_test.go: rewrite every manifest
  fixture from the old Profile/Compose schema (profiles:/services:/
  defaultProfile:/--no-telemetry/--volumes) to the flat Task model, and
  replace the now-permanently-erroring 'devx render compose' test with
  coverage asserting render/lock/providers/export all return the
  documented removal error.
- devx_test.go: add TestDevxDown_StopsBackgroundProcess, a real
  subprocess-level regression test for the PID-based process-group stop
  fix from the prior review; TestDevxTasksJSON and TestDevxRunShorthand
  for the Task-model-specific 'devx tasks --json' and 'devx <name>'
  shorthand behaviors that had no integration coverage; and make
  TestDevxDoctor tolerant of a missing podman runtime (optional,
  environment-dependent) instead of treating it as a hard failure.
- cmd/devx: add helpers_test.go, cmd_validate_test.go, cmd_run_test.go,
  cmd_down_test.go, main_test.go — the package previously had zero unit
  tests (the old main_test.go/cmd_validate_test.go were deleted during
  the schema migration and never replaced). Covers parseCLIVars,
  loadManifest/loadValidManifest, ensureDevxDir/ensureGitignore,
  runValidate + extendedWarnings (built-in name collisions, port
  collisions), runTasks (human/--json), runRun (unknown task, --dry-run,
  --watch validation, one-shot execution), and stopProcessGroup/
  stopTimeoutFor (devx down's PID-signal logic, using a real grouped
  subprocess).
- Use a hand-rolled chdirTemp(t) test helper instead of testing.T.Chdir,
  since golangci-lint's stdversion check flags T.Chdir (Go 1.24+) against
  this module's go 1.21 go.mod directive.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fixes:
- internal/task/driver.go: processHandle.Wait()/Status() used a
  single-value "done" channel that only the first reader could ever
  observe, so calling Wait() permanently starved Status() into
  reporting "running" forever. Replaced with a close-once "exited"
  channel + stored exitErr so both methods are safe to call any
  number of times in any order.

New tests:
- internal/task/driver_test.go: processDriver Start/Wait/Status/Stop/
  Exec, followReader busy-spin regression, tailFile, CheckReady/
  WaitReady (cmd/HTTP/TCP paths, timeouts, nil ready-check).
- internal/task/state_test.go: StateStore LoadState/Save/Load/Remove/
  Lookup/All, including malformed-JSON and snapshot-copy semantics.
- internal/task/inference_helpers_test.go: osMatches coverage
  (empty filter, GOOS match, macos/darwin alias, multi-entry).
- internal/config/manifest_test.go: Validate() gaps (version != 1,
  no tasks, ai: block provider/model requirements, restart: value
  and detached-only validation, scope: value validation).
- internal/config/tools_test.go: ValidateSetup missing name/run.
- internal/ui/ui_test.go: PrintTable column alignment.
- internal/util/maps_test.go: SortedKeys ordering/empty-map.

All packages verified with go build/vet/test, golangci-lint run
(0 issues), and the full tagged integration suite against Docker.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
internal/doctor: coverage 12.6% -> 47.4%
- checkPortConflicts, checkTaskRequires, checkTools (no-tools case),
  checkDevcontainerCLI, checkAI (not-configured/malformed/configured
  paths). Run/detectAllRuntimes/detectCompose remain untested as they
  depend on live docker/podman and are better suited to integration
  tests.

cmd/devx: coverage 11.4% -> 19.4%
- cmd_down.go: reverseDependencyOrder (dependents-before-dependencies,
  cycle detection)
- cmd_env.go: sourceLabel, isLikelySecret, maskSecret,
  loadDotEnvToMap, dotEnvPath
- cmd_clone.go: expandRepoArg, repoBaseName
- cmd_update.go: binaryAssetName, normaliseVersion
- cmd_auth.go: statusIcon
- cmd_dev.go: shortFeatureName, reCapture, mountSummary,
  commandString, truncate, stripJSONComments, lifecycleCommands,
  vscodeExtensions
- cmd_ci.go: stackFromDevcontainerFeatures, globExists, readJSONC,
  detectStack, generateCIWorkflow

Remaining 0%-coverage code is almost entirely thin wrappers around
external tools (docker/podman/devcontainer CLIs, AI provider APIs,
self-update network calls) better exercised via integration tests
than unit mocks.

Verified: go build/vet, go test ./... -race (all green), golangci-lint
run ./... (0 issues), and the full Docker-backed integration suite
(go test ./tests/integration/... -tags integration, DEVX_BIN set).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Bugs fixed:
- cmd_new.go: `devx new <name> --run ... --desc ...` (the command's own
  documented usage) silently failed with a usage error, because Go's
  flag package stops parsing at the first non-flag argument, so a task
  name given before its flags left them unparsed. Added
  extractPositional() to pull the bare task-name positional out of args
  regardless of its position relative to --run/--desc, so the name can
  appear before, after, or between flags.
- cmd_dev.go/cmd_ci.go: stripJSONComments (and the separate, now-removed
  duplicate regex in readJSONC) used `//[^\n]*` to strip JSONC comments,
  which also ate anything after "//" inside a JSON string value —
  including the "https://..." URL devx's own `devx init` writes into
  initializeCommand. This broke `devx dev status`/`doctor` JSON parsing
  for any devcontainer.json containing a URL. Rewrote as a string-aware
  stripper that tracks whether it's inside a JSON string literal before
  treating "//" as a comment start.

New test coverage (cmd/devx: 19.4% -> 27.4%):
- cmd_new_test.go: round-trips a new task into valid YAML, the
  bare-shorthand path, missing-manifest and bad-arg-count errors, and
  regression coverage for the flag-ordering bug above.
- cmd_init_test.go: writeInitManifest produces parseable YAML for every
  stack (+ai block), writeDevcontainer produces valid JSONC and never
  overwrites an existing file.
- cmd_status_test.go: runStatus --json/human output with no running
  tasks, missing-manifest error, devcontainerState/aiStatusLine
  not-configured paths.
- watch_test.go: snapshotMTimes glob matching, changed() added/updated/
  unchanged snapshot comparison.
- helpers_test.go: lookPathQuiet, selectContainerRuntime.

Verified: go build/vet, go test ./... -race (all green), golangci-lint
run ./... (0 issues), and the full Docker-backed integration suite
(DEVX_BIN set). Manually reproduced and confirmed the fix for the
`devx new` flag-ordering bug against the built binary.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ompts, down, dev files, logs, exec

Covers previously-untested pure/file-based logic: runRender/runLock/
runProviders/runExport error messages, completion script generation,
checkEnvKey masking, setupStatusIcon, the AI wizard's promptChoice/
promptString/promptYN/printYAMLSnippet/printAIUsage helpers, stopEntry/
stopContainerByName dispatch paths, requireDevcontainerCLI/devcontainerID/
printDevUsage, loadDevcontainerJSON/findBaseImages, runExec argument
validation, containerStatus, and cmd_logs.go's streamPrefixed/tailOne.

cmd/devx coverage: 27.4% -> 37.6%.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
New tests cover runInit's interactive wizard (stdin-fed, full round-trip
to devx.yaml + devcontainer.json), runEnv's variable listing/masking,
runAuth's check list, runClone/runDown/runUp/runSetup/runLogs argument
and missing-manifest validation, runCI (stdout vs --write), and
runDevStatus's devcontainer.json summary output.

Found and fixed a real bug: tailAll (devx logs with no task name and no
--follow) spawned one goroutine per running task to stream its log but
never waited for them before returning, so main() could exit before any
output was flushed — devx logs would often print nothing. Added a
sync.WaitGroup so non-follow mode waits for all goroutines to finish;
follow mode is unaffected (it already blocks forever on select{}).
Reproduced the drop deterministically over 5 repeated test runs before
the fix, and confirmed 10/10 passes after.

cmd/devx coverage: 37.6% -> 50.2%.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…runAIReset, runCILocal

Covers runDev's subcommand dispatch table (help/unknown/status default),
runDevBuild/runDevUp/runDevExec's devcontainer-CLI-missing error paths,
runDevOpen's no-devcontainer error path, runDevPS's docker ps output,
runAIStatus/runAIReset's not-configured and confirm/decline-reset flows
(stdin-fed), and runCILocal's act-not-installed instructions.

cmd/devx coverage: 50.2% -> 54.2%.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… chat, runAI dispatch

Covers wizardLocal/wizardRemote/wizardCloud/wizardPickModel/wizardTool
(stdin-fed, declining any real backend setup/tool launch), runAIModel's
list/pull/remove subcommands and usage/validation paths, runAIChat's
no-config error, runExternalCommand, and runAI's full dispatch including
the end-to-end first-time wizard flow via runAI(nil).

Note: this sandbox has a real Ollama server reachable via
host.docker.internal, so runAIModelList's test tolerates either a
detected-backend or no-backend outcome rather than asserting one.

cmd/devx coverage: 54.2% -> 63.2%.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…and signal handling

- devcontainerBuild/runDevcontainerCLI/runDevLogs/runDevShell: error paths
  when the devcontainer CLI or a running container isn't present.
- runAIModelPull/runAIModelRemove: MLX/remote-hint branches plus the
  ollama-not-installed error path.
- watchAndRerun: verifies the ctx.Done() case returns promptly instead of
  blocking on the 1s ticker, using an already-cancelled context.
- attachAndWaitForInterrupt: sends SIGINT to the test process itself to
  verify it unblocks and runs runDown instead of hanging forever.

All new tests stable across repeated -race runs (5x).

cmd/devx coverage: 63.2% -> 66.8%.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ad EF Core detection

These three packages had zero test coverage before this change:
- internal/ai (92.8%): Client.New/complete across openai/anthropic/
  azure-openai/ollama providers (via httptest servers), DetectFrameworks'
  per-language/framework marker detection, Detect()'s JSON-fence-stripping
  and error paths.
- internal/localai (77.4%): model suggestion lists, backend probing
  (checkURL/fetchMLXModel/probeRemote via httptest, ResolveBackend/
  IsAppleSilicon), SavedAIConfig load/save/delete round-trips,
  ToolsForProvider/buildToolCmd/aiderArgs/setenv, LaunchTool's
  backend-unreachable and unknown-tool error paths, DetectCloudProviders,
  buildContinueConfig/WriteContinueConfig for every provider branch, and
  Setup()'s unknown-backend/non-Apple-Silicon-MLX error paths (without
  ever invoking the real network-installing code paths).
- internal/plugins (100%): Discover()'s PATH-scanning, prefix matching,
  cross-directory deduplication, and unreadable-PATH-entry handling.

Bug fix: internal/ai/hints.go's EF Core detection passed the literal
string .csproj to hasContentMatch's exact-filename check instead of an
actual matched file name, so 'EntityFramework' content detection could
never fire for any real *.csproj file. Fixed to check content of the
actual matched .csproj/.fsproj files. Also removed the now-unused
anyMatch helper.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use fake shell-script binaries (via the Runtime.Binary field) to
deterministically exercise Up/Down/Exec/Logs/Status/ResolveImageDigest
and the underlying exec.Command plumbing, without requiring real
docker/podman compose behavior. Brings both packages from 0% to
~93%/~92.6% coverage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…Env, and doctor.Run

- internal/task/container_driver_test.go: covers containerDriver.Start,
  containerHandle Status/Stop/Logs/Exec/Wait/PID using fake docker/podman
  binaries injected via PATH.
- internal/task/scheduler_resolve_test.go: covers Scheduler.ResolveOne,
  ResolveAll, probeExisting (container + process paths), processAlive,
  sanitize, taskTimeout.
- internal/task/resolve_test.go: adds RedactedEnv coverage.
- internal/doctor/doctor_test.go: adds Run, detectAllRuntimes, detectCompose
  coverage.

internal/task: 65.8% -> 78.9%, internal/doctor: 47.4% -> 74.1%.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Change releasesAPI/releaseDL from consts to vars so tests can point
fetchLatestVersion at a local httptest.Server instead of the real
GitHub API. Covers fetchLatestVersion's success/malformed-JSON/
unreachable-server paths, and runUpdate's dev-build-skip,
already-on-latest, --check, and fetch-error branches.

selfUpdate and main() remain untested: selfUpdate replaces the
currently running binary via os.Executable(), which would corrupt
the test binary itself if exercised directly.

cmd/devx: 66.8% -> 68.3%.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant