From 185174d8e3e3a466f77bcc0b927d82ecba7e9d19 Mon Sep 17 00:00:00 2001 From: Nick Huo Date: Mon, 31 Aug 2026 17:37:35 -0700 Subject: [PATCH 01/14] Port the porting skill to the new layout The skill wrote agents/, workflow/ and config/ beside an untouched source tree. It now duplicates the source into .car/app, edits adapters in place inside that copy, and keeps every declaration in .car/config. Two rules came out of running it end to end on a real LangGraph project: M27 -- root the copy at the source's *import* root, not its repository root. A project whose modules import each other as top-level names from under src/ builds green, reports healthy, and answers `No agent loaded` on the first request when the copy is rooted one level too high. validate.py already caught this as V031; nobody had run it before building. M25 -- two agents may not share one entrypoint, because each agent's stub is written over its own entrypoint and the second would land on the first. V023 is rewritten around the same idea: the workflow must reach an agent through the module the build replaces with a stub, so a flat name or a package re-export now reports where the stub actually lands. V020 becomes the shared entrypoint check, V032 requires config/ beside app/, and the stale `stub_two_destinations` capability is gone -- a stub has one destination now. Also applied Anthropic's skill-authoring guidance: the invented `compatibility` frontmatter key is folded into the body, the description carries the terms that should trigger it (`ventis`, `.car`, `ventis build`), the workflow opens with a copyable checklist, the validator is stated as a loop with an exit condition rather than a one-shot, references over 100 lines gained a table of contents, and a concrete worked example is added under references/. --- .../skills/porting-to-canyonos-core/SKILL.md | 180 ++++++++--- .../references/example-port.md | 119 +++++++ .../references/packaging.md | 53 +++- .../references/runtime-contract.md | 63 ++-- .../references/troubleshooting.md | 13 +- .../porting-to-canyonos-core/validate.py | 291 +++++++++--------- 6 files changed, 497 insertions(+), 222 deletions(-) create mode 100644 .claude/skills/porting-to-canyonos-core/references/example-port.md diff --git a/.claude/skills/porting-to-canyonos-core/SKILL.md b/.claude/skills/porting-to-canyonos-core/SKILL.md index fe6dd49..1776b98 100644 --- a/.claude/skills/porting-to-canyonos-core/SKILL.md +++ b/.claude/skills/porting-to-canyonos-core/SKILL.md @@ -1,16 +1,38 @@ --- name: porting-to-canyonos-core -description: Ports existing LangChain, LangGraph, CrewAI, AutoGen, and hand-rolled Python agent projects to CanyonOS Core. Use when converting, migrating, adapting, packaging, building, or deploying an existing agent or multi-agent project onto CanyonOS Core. -compatibility: Requires Python, Docker, and the `ventis` compatibility CLI. Runtime identifiers remain `ventis`, `VENTIS_*`, and `ventis-*`. +description: Ports existing LangChain, LangGraph, CrewAI, AutoGen, and hand-rolled Python agent projects onto CanyonOS Core, whose CLI is `ventis` and whose artifacts live in a `.car` directory. Writes the `.car/config` manifest and declarations, duplicates the source into `.car/app`, writes adapters and the workflow, then validates, builds, deploys and probes. Use when converting, migrating, adapting, packaging, building or deploying an existing agent or multi-agent project onto CanyonOS Core or ventis, when running `ventis build` or `ventis deploy`, or when a `.car` port fails to build, load an agent, or answer a request. --- # Port an agent project to CanyonOS Core +Requires Python, Docker, and the `ventis` CLI. `validate.py` in this skill needs +Python 3 and `pyyaml`. + CanyonOS Core is the product name. Its compatibility executable and Python package remain `ventis`; environment variables and Docker resources retain the `VENTIS_*` and `ventis-*` prefixes. These are protocol identifiers, not branding strings. Do not rename them. +## Port checklist + +Copy this into your response and check items off as you go. Every step below +maps to one line here. + +``` +Port progress: +- [ ] 1. Copy the source into .car/app, rooted at its import root +- [ ] 2. Survey the copy and choose service boundaries +- [ ] 3. Write declarations, adapters, workflow, config +- [ ] 4. validate.py reports 0 errors +- [ ] 5. ventis build succeeds +- [ ] 6. Both images pass their probes +- [ ] 7. A real request returns through /status +- [ ] 8. Clean up; git status shows nothing outside .car +``` + +Do not skip step 4. Every failure mode it reports survives a green build and a +healthy replica, and then costs a deploy cycle to rediscover. + ## Load references only when needed - Read [references/packaging.md](references/packaging.md) when a source import @@ -24,37 +46,74 @@ strings. Do not rename them. failed build, image probe, deploy, or request. - Read [references/runtime-contract.md](references/runtime-contract.md) when a validator finding needs explanation or the runtime mechanism is unclear. +- Read [references/example-port.md](references/example-port.md) for one port end + to end -- the decisions, the files, and the evidence that closed it. -## Goal: thin scaffolding beside untouched source +## Goal: a self-contained `.car`, and a source tree that never learns about it + +The port lives entirely inside `.car/`, next to the application source and +never inside it: ```text -agents/.yaml one callable surface per service -agents/.py one thin adapter per service, when needed -workflow/_workflow.py HTTP entry point; calls deploy() -config/global_controller.yaml deployment manifest -config/policy.yaml optional access restriction -pyproject.toml conditional nested-import scaffolding - unchanged +.car/config/global_controller.yaml deployment manifest +.car/config/policy.yaml optional access restriction +.car/config/.yaml one callable surface per service +.car/app/ a copy of the application source +.car/app//.py the adapter, written where the code it wraps lives +.car/app//_workflow.py HTTP entry point; calls deploy() +.car/app/pyproject.toml conditional nested-import scaffolding +/ the developer's tree, untouched and unaware ``` +`.car` has exactly two authored directories: `config/`, which holds every +declaration Canyon owns, and `app/`, the copy that becomes `/app` in every +container. The container keeps the directory structure the application already +had. Write adapters into that copy, in the module the code they wrap already +lives in -- not into new `agents/` and `workflow/` directories. `ventis` +commands run from the application root and read `.car` below it. + +Nothing under `.car` points back out at the application source, and nothing in +the application source points at `.car`. Deleting `.car` returns the project to +exactly where it started; regenerating it touches no file the developer owns. + The file count follows the deployment. A multi-agent port has one yaml/adapter -pair per service that is worth deploying separately. If a source class already -satisfies the runtime contract, point its config entry at that file and do not -copy it into an adapter. +pair per service that is worth deploying separately. If a source class in the +copy already satisfies the runtime contract, point its config entry at that +file and do not write an adapter beside it. Everything the source already owns—prompts, tools, schemas, parsing, retries, -model clients, and node bodies—is imported. The port re-expresses only the -CanyonOS Core boundary and framework-owned orchestration. +model clients, and node bodies—is imported from where the copy keeps it. The +port re-expresses only the CanyonOS Core boundary and framework-owned +orchestration. + +## 1. Duplicate the source, then survey it + +Copy the application source into `.car/app/`, preserving its structure. Leave +out only what no container should carry: `.git/`, `.car/` itself, virtualenvs, +caches, build outputs, and `.env` files holding real credentials. + +**Root the copy at the source's import root, which is not always its repository +root.** `/app` is the copy, and without the editable-install capability it is +the only entry on `sys.path`. A source whose modules import each other as +`from tools import ...` while living under `src/` has `src/` as its import +root: copy the *contents* of `src/` to `.car/app/`, or every one of those +imports raises `ModuleNotFoundError` inside `_load_agent` and the first +request answers `No agent loaded`. Read the source's own imports, not its +directory names, to decide. Re-rooting is free here in a way it never was +before: `.car/app` is a copy Canyon owns, so nothing in the developer's tree +moves. -The port root is the existing repository root and the directory from which -`ventis build` runs. Write scaffolding there beside existing directories. If the -repository already uses `src/`, leave it in place and put `agents/`, `workflow/`, -and `config/` beside it. Never move or copy the repository into a new `src/` -directory, and never create an outer wrapper merely for the port. +```bash +mkdir -p .car/config +rsync -a --exclude '.git' --exclude '.car' --exclude '.venv' --exclude 'venv' \ + --exclude '__pycache__' --exclude '.env' / .car/app/ +``` -## 1. Survey before writing +Every edit from here on is inside `.car`. The application source outside it is +read-only for the rest of the port -- `git status` at the end shows `.car/` and +nothing else. -Identify: +Then survey the copy. Identify: 1. The source entry point and callable input/output. 2. Framework-owned control flow (`StateGraph`, `Crew`, `GroupChat`, routing, @@ -65,13 +124,16 @@ Identify: 5. Imports and declared runtime dependencies. 6. Model provider, credential names, streaming use, and optional `llm_proxy`. 7. Whether independent work fans out and benefits from separate replicas. -8. Whether source imports resolve from the project root that becomes `/app`. +8. Whether source imports resolve from `.car/app`, the root that becomes + `/app`. This is the check the validator turns into V031, and it is the one + most likely to survive a green build and a healthy replica. -Run the validator once now. Its header detects capabilities directly from the -importable runtime rather than external development metadata: +Run the validator now, and again after every change until it reports 0 errors. +Execute it; do not read it. Its header detects capabilities directly from the +importable runtime rather than from release history: ```bash -python /validate.py . +python /validate.py .car ``` If config or agent yaml is malformed, the validator defers to `ventis build`. @@ -96,6 +158,10 @@ Report any choice the source does not specify. ### Agent yaml +Declarations go in `.car/config/`, beside the manifest. The build reads every +`*.yaml` there and keeps the ones with a top-level `agent.name`, so the +manifest and `policy.yaml` drop out on their own. + Use one yaml per deployed service. Argument types are bare builtins only: `str`, `int`, `float`, `bool`, `dict`, or `list`. Every declared argument is required by the generated stub. `returns.type` is documentation; use `dict` or @@ -103,6 +169,12 @@ required by the generated stub. `returns.type` is documentation; use `dict` or ### Adapter +Write the adapter where the code it wraps already lives, and give the module a +name of its own -- two agents may not share one entrypoint, because each +agent's stub is written over its own entrypoint and the second would land on +the first. Prefer editing the copied module in place over adding a parallel +one; that is what the copy is for. + The entrypoint exposes a module-level class named exactly `agent.name`. It constructs with no arguments and its declared methods are synchronous. Read configuration from the environment in `__init__`. Bridge source coroutines @@ -115,13 +187,19 @@ provider and SDK. ### Workflow Expose `main(query: str)` and call `deploy(main, port=...)` at module scope. -Import generated stubs by yaml basename and agent class name: +Import each agent from its own `entrypoint`, exactly where the source copy +keeps it -- that is the one module the build replaces with a stub: ```python from deploy import deploy -from agents. import +from . import # the agent's entrypoint path ``` +Any other route to the class -- a flat name, a package re-export, a second +copy of the module -- reaches the real class and runs the agent in the workflow +process with none of the deployment behind it. That import needs no rewriting +when the source already imported the agent from there. + The deployment platform sends `{query: string}` to `/main`. Pack richer input inside `query`; any additional workflow parameter has a default. @@ -144,8 +222,9 @@ For each service, keep these names aligned: config entry name == yaml agent.name == entrypoint class name ``` -Use lowercase `provider: local`; `replicas` is an integer; `requirements` is a -list of distribution-name strings. Put `env_file` at config top level when the +`entrypoint` and `workflow_file` are relative to `.car/app/` and must stay +inside it. Use lowercase `provider: local`; `replicas` is an integer; +`requirements` is a list of distribution-name strings. Put `env_file` at config top level when the runtime capability is available. Omit `policy.yaml` unless access must be restricted; if present, give it a non-empty `rules` list. @@ -169,27 +248,36 @@ source-integrity rules. The owner column states where each is decided. | M11 | Workflow MUST expose `main(query)`; extra parameters MUST default | V016 | | M12 | Workflow MUST NEVER contain a main guard | V017 | | M13 | Fan-out MUST dispatch all calls before resolving any | V018 | -| M14 | Project modules MUST not take runtime or generated-stub flat names | V019/V020 | -| M15 | Workflow MUST import stubs from `agents.` | V023 | +| M14 | A module at the root of the copy MUST not take a runtime flat name | V019 | +| M15 | Workflow MUST import each agent from its own `entrypoint` module | V023 | | M16 | Policy MUST be absent or contain a non-empty `rules` list | deploy preflight | | M17 | EC2 entries MUST satisfy the EC2 deployment contract | deploy preflight | | M18 | NEVER copy source prompts, tools, schemas, or model calls | review | | M19 | NEVER hardcode or bake a real credential into an image | W003 | -| M20 | NEVER edit or vendor the source tree | `git status` | +| M20 | NEVER write outside `.car`; the application source stays untouched | `git status` | | M21 | NEVER swap the source LLM provider | review | | M22 | NEVER silently move, drop, or reclassify source dependencies | review | | M23 | Framework control flow MUST be rewritten; source node logic MUST be imported | review | -| M24 | A non-resolving source import MUST have usable root packaging metadata when editable install is supported | V031 | +| M24 | A non-resolving source import MUST have usable packaging metadata at the root of the copy when editable install is supported | V031 | +| M25 | Two agents MUST NOT share one `entrypoint` | V020 | +| M26 | `.car` MUST hold `config/` beside `app/`, the source copy | V032 | +| M27 | `.car/app` MUST be rooted at the source's import root | V031 | ## 4. Validate, build, and probe -Run static preflight, then let the build own build-time validation: +Run static preflight, then let the build own build-time validation. Both run +from the application root: ```bash -python /validate.py . -ventis build -c config/global_controller.yaml +python /validate.py .car +ventis build ``` +Fix every ERROR and re-run the validator until it reports 0 errors before +running `ventis build`. A build that skips this passes, and the port then fails +at `docker run` or on the first request, where the message names a container +rather than the mistake. + A green build never imports the adapter. Probe each agent image in this order: ```bash @@ -197,10 +285,11 @@ A green build never imports the adapter. Probe each agent image in this order: docker run --rm ventis- \ python -c "import local_controller" -# Agent load path; include --env-file when configured +# Agent load path; the entrypoint keeps its path from the source copy. +# Include --env-file when configured. docker run --rm --env-file ventis- \ python -c "import importlib.util,sys; \ -s=importlib.util.spec_from_file_location('m','.py'); \ +s=importlib.util.spec_from_file_location('m',''); \ m=importlib.util.module_from_spec(s);sys.modules['m']=m;s.loader.exec_module(m); \ m.();print('ok')" ``` @@ -211,7 +300,7 @@ its own dependency resolve and generated-stub imports. Then deploy, send a representative request, and poll its status: ```bash -ventis deploy -c config/global_controller.yaml +ventis deploy curl -X POST http://localhost:8080/main \ -H 'Content-Type: application/json' -d '{"query":""}' curl http://localhost:8080/status/ @@ -231,10 +320,13 @@ ventis clean docker image rm ventis- \ ventis- -test ! -e stubs && test ! -e grpc_stubs && test ! -e docker_container +test ! -e .car/stubs && test ! -e .car/grpc_stubs && test ! -e .car/docker_container docker ps -a --format '{{.Names}}' ``` -`ventis clean` removes only `stubs/`, `grpc_stubs/`, and `docker_container/`; it -does not remove containers or images. Keep port scaffolding, untouched source, -and requested logs or reports. +`ventis clean` removes only `.car/stubs/`, `.car/grpc_stubs/`, and +`.car/docker_container/`; it does not remove containers or images. Keep +`.car/config`, `.car/app`, and requested logs or reports. + +Finally, confirm the decoupling held: `git status` outside `.car` reports no +change to any file the developer owns. diff --git a/.claude/skills/porting-to-canyonos-core/references/example-port.md b/.claude/skills/porting-to-canyonos-core/references/example-port.md new file mode 100644 index 0000000..1a67d48 --- /dev/null +++ b/.claude/skills/porting-to-canyonos-core/references/example-port.md @@ -0,0 +1,119 @@ +# One port, end to end + +A LangGraph email assistant, ported and deployed. Read this for the shape of the +decisions; the rules themselves are in SKILL.md. + +## Contents + +- The source +- Decision 1: where to root the copy +- Decision 2: one agent, two methods +- The files +- The evidence + +## The source + +A single-file LangGraph app under `src/`, plus its own packages: + +```text +src/email_assistant.py triage_router + a ReAct loop (llm_call/tool_node) +src/prompts.py src/schemas.py src/utils.py src/tools/ +pyproject.toml package-dir = {"" = "src"} +.env OPENAI_API_KEY +``` + +Two graphs. Outer: `START -> triage_router -> (END | response_agent)`, routed by +a `Command(goto=...)`. Inner: `llm_call -> should_continue -> tool_node`, looping +until the model calls `Done`. + +## Decision 1: where to root the copy + +`email_assistant.py` imports `from tools import ...` and `from prompts import +...`, and `pyproject.toml` says `package-dir = {"" = "src"}`. So the import root +is `src/`, not the repository root: + +```bash +rsync -a --exclude '.git' --exclude '.car' --exclude '__pycache__' \ + --exclude '.env' src/ .car/app/ +``` + +Copying the repository root instead puts those modules at `/app/src/tools` while +`/app` is the only entry on `sys.path`. The build stays green, the replica +reports healthy, and the first request answers `No agent loaded` with +`No module named 'tools'` in the container log. The validator reports this as +V031 before any of that happens. + +`pyproject.toml` was left out of the copy on purpose: this runtime runs no +editable install, and its `package-dir = {"" = "src"}` is false of a copy that +is already rooted at `src/`. + +## Decision 2: one agent, two methods + +The outer graph is framework control flow, so it became an `if` in the workflow. +The inner ReAct loop stayed inside one agent method: every turn needs the whole +message history, so splitting `llm_call` from `tool_node` would push a growing +message list through Redis for no parallelism. + +The adapter is appended to the bottom of the copied `email_assistant.py`, so it +calls `triage_router`, `llm_call`, `should_continue` and `tool_node` as +module-level names. No prompt, tool, schema or model call is restated. + +```python +class EmailAgent: + def __init__(self): + self.recursion_limit = int(os.environ.get("VENTIS_RECURSION_LIMIT", "25")) + + def triage(self, email_input: dict) -> dict: + command = triage_router({"email_input": email_input, "messages": []}) + update = command.update or {} + return {"goto": command.goto, **update} + + def respond(self, messages: list) -> dict: + state = {"messages": add_messages([], messages)} + for _ in range(self.recursion_limit): + state["messages"] = add_messages(state["messages"], llm_call(state)["messages"]) + if should_continue(state) != "Action": + return {"messages": messages_to_dict(state["messages"])} + state["messages"] = add_messages(state["messages"], tool_node(state)["messages"]) + raise RuntimeError(f"agent did not call Done within {self.recursion_limit} turns") +``` + +`Command` and LangChain message objects are framework types, so they are +unpacked and serialized before they cross the boundary. + +## The files + +```text +.car/config/global_controller.yaml EmailAgent + Workflow, env_file: .env +.car/config/email_agent.yaml triage(email_input: dict), respond(messages: list) +.car/app/email_assistant.py source + the adapter above +.car/app/email_workflow.py the outer graph as an if; deploy(main, port=8080) +.car/app/prompts.py schemas.py utils.py tools/ untouched copies +``` + +`entrypoint: email_assistant.py` and `workflow_file: email_workflow.py`, both +relative to `.car/app`. The workflow imports the agent from its entrypoint -- +`from email_assistant import EmailAgent` -- which is the one module the build +replaces with a stub. + +The platform sends `{query: string}` only, so the four email fields ride inside +`query` as JSON and the workflow unpacks them. `main` returns a dict, and +`GET /status/` hands it back under `result`. + +## The evidence + +```text +validate.py .car 0 errors +ventis build ventis-emailagent, ventis-workflow +docker run ... import local_controller both images +docker run --env-file .env ... EmailAgent() loads +ventis deploy 2 replicas ready +POST /main 202 {"request_id": ...} +GET /status/ status: error, 401 from OpenAI +``` + +The last line is the interesting one. The agent log showed the request arriving +over gRPC (`route_to: :8000`, `function: triage`), the agent loading, and the +source's own model call returning 401 on an expired key. A source-level failure +behind a working boundary still closes the port: record it as such rather than +calling the port broken. diff --git a/.claude/skills/porting-to-canyonos-core/references/packaging.md b/.claude/skills/porting-to-canyonos-core/references/packaging.md index 8520c4a..cd8265a 100644 --- a/.claude/skills/porting-to-canyonos-core/references/packaging.md +++ b/.claude/skills/porting-to-canyonos-core/references/packaging.md @@ -3,10 +3,20 @@ Read this reference when an adapter imports nested source code, the source uses a `src/` layout, or V031 reports an import-root problem. +## Contents + +- What `/app` can import +- Re-root the copy before reaching for metadata +- Detect support, do not infer it from release history +- Root metadata is the trigger +- Dependencies in nested metadata +- Validation boundary + ## What `/app` can import -CanyonOS Core preserves project-relative paths in the image and starts Python at -`/app`. Without an editable install, Python resolves names rooted there: +CanyonOS Core copies `.car/app/` into the image with its paths intact and +starts Python at `/app`, so `/app` is that copy. Without an editable install, +Python resolves names rooted there: - `/app/tools.py` as `import tools` - `/app/pkg/__init__.py` as `import pkg` @@ -16,12 +26,31 @@ CanyonOS Core preserves project-relative paths in the image and starts Python at It does not resolve `/app/source/pkg` as `import pkg`; `/app/source` must become an import root first. +## Re-root the copy before reaching for metadata + +`.car/app/` is a copy Canyon owns, so the cheapest fix is usually to root it +where the source already imports from. A project laid out as + +```text +repo/src/email_assistant.py imports `tools`, `prompts`, `utils` +repo/src/tools/ +repo/pyproject.toml +``` + +has `src/` as its import root. Copy `src/`'s contents to `.car/app/` and every +one of those imports resolves from `/app` with no metadata, no editable install +and no `sys.path` hack. `entrypoint` and `workflow_file` then name modules +relative to that root, and the workflow imports the agent the same way. + +Reach for the metadata below only when one copy root cannot serve every import +-- for instance when the source imports both `tools` and `src.tools`. + ## Detect support, do not infer it from release history Run: ```bash -python /validate.py . +python /validate.py .car ``` Read the `editable_install` capability. If it is unavailable and the original @@ -30,16 +59,17 @@ Do not add a `sys.path` hack or relocate source files. ## Root metadata is the trigger -When editable install is supported, only packaging metadata at the **port root** -triggers `pip install -e .`: +When editable install is supported, only packaging metadata at the **root of +the copy** triggers `pip install -e .`: ```text -port-root/pyproject.toml detected -port-root/source/pyproject.toml ignored as an install trigger +.car/app/pyproject.toml detected +.car/app/source/pyproject.toml ignored as an install trigger ``` -A nested source repository may remain untouched. Add minimal root scaffolding -that points package discovery at the existing source package: +If the application keeps its metadata deeper in the tree, that copy stays where +it is. Add minimal scaffolding at `.car/app/` that points package discovery at +the existing package: ```toml [build-system] @@ -69,8 +99,9 @@ distributions in each relevant config entry's `requirements` list. This is compatibility scaffolding, not permission to drop, move, or reclassify declared dependencies. -If source metadata is already at the port root, do not create a wrapper. Its -project dependencies participate in the same resolver as config requirements. +If the application's own metadata already sits at the root of the copy, do not +create a wrapper. Its project dependencies participate in the same resolver as +config requirements. Report declared-but-unused toolchain dependencies and their image cost; let the owner decide whether source metadata should change. diff --git a/.claude/skills/porting-to-canyonos-core/references/runtime-contract.md b/.claude/skills/porting-to-canyonos-core/references/runtime-contract.md index 94f7401..6d7715a 100644 --- a/.claude/skills/porting-to-canyonos-core/references/runtime-contract.md +++ b/.claude/skills/porting-to-canyonos-core/references/runtime-contract.md @@ -9,17 +9,36 @@ finding. Runtime-dependent behavior is expressed as capabilities; run `validate.py` against the target environment instead of inferring support from release history. -## Project root and discovery +## Contents -`ventis build` uses the current working directory as the project root. +- Artifact root and discovery +- Agent yaml and generated stubs +- Agent loading and execution +- Workflow execution +- Build context and collisions +- Dependencies and protobuf +- Credentials capability +- Policy and provider behavior +- Cleanup boundary + +## Artifact root and discovery + +`ventis build` runs from the application root and reads `.car` below it. That +artifact root holds `config/` beside `app/`, the copy of the application source +that becomes `/app` inside every image. Paths in the config are relative to +`app/` and may not escape it. | Input | Discovery | |---|---| -| `agents/*.yaml` | direct yaml glob under the project root | -| `config/global_controller.yaml` | default config, overridable with `-c` | -| workflow | `workflow_file` on a `type: workflow` entry | +| agent declarations | any `.car/config/*.yaml` with a top-level `agent.name` | +| `.car/config/global_controller.yaml` | default config, overridable with `-c` | +| entrypoint | `entrypoint` on an agent entry, relative to `app/` | +| workflow | `workflow_file` on a `type: workflow` entry, relative to `app/` | | policy | `policy.yaml` beside the selected config file | -| generated files | `stubs/`, `grpc_stubs/`, `docker_container/` | +| generated files | `stubs/`, `grpc_stubs/`, `docker_container/`, all under `.car` beside `app/` | + +Because build products sit next to the copy rather than inside it, an +application directory named `stubs/` or `build/` survives into the image. The config name, yaml `agent.name`, and entrypoint class name form one binding: @@ -51,10 +70,12 @@ builtins. Generated methods have no defaults, so every declared argument is required at the stub call site. `returns` does not control runtime conversion; it documents whether workflow code should parse the returned string. -Stub destinations differ by runtime capability and entrypoint layout. Workflow -code in this port convention imports the generated class from -`agents.`. The validator checks that import against declarations -before the workflow image starts. +A stub has exactly one destination: the agent's own `entrypoint` path. In every +image except that agent's own, the stub is written over the real module there, +so an import of the agent from its source location resolves to the stub and +travels over gRPC. The agent's own image keeps its real module and receives +only its peers' stubs. The validator checks workflow imports against those +entrypoints before the workflow image starts. ## Agent loading and execution @@ -105,11 +126,11 @@ package resolution. Probe it independently from agent images. ## Build context and collisions -The runtime copies project files while preserving relative paths, then writes -shared runtime modules, generated stubs, and entrypoints into the image. Later -writes can shadow project files. +The runtime sweeps `app/` while preserving relative paths, then writes shared +runtime modules, generated stubs, and entrypoints into the image. Later writes +can shadow swept files. -Avoid root project modules named like runtime files, including: +Avoid modules at the root of the copy named like runtime files, including: ```text future.py @@ -124,8 +145,9 @@ session_logging.py workflow_launcher.py ``` -Also avoid a yaml basename that shadows a different source module imported by an -adapter. The validator checks deterministic flat-name collisions. +Two agents also may not share one entrypoint: each stub is written over its own +entrypoint, so the second lands on the first and every caller reaches whichever +was built last. The validator checks both collisions. File sweep and editable-install behavior are runtime capabilities. For nested imports, follow [packaging.md](packaging.md). @@ -153,7 +175,9 @@ source dependencies silently. ## Credentials capability When `env_file` capability is available, the top-level config path is resolved -against the project root and passed at container start. Hidden env files are not +against the application root -- the directory the command runs from, not `.car` +-- and passed at container start. A `.env` beside the source stays out of the +artifacts. Hidden env files are not copied into images. Invalid paths are deploy-preflight errors. When the capability is unavailable, declaring `env_file` has no effect. If the @@ -182,6 +206,5 @@ containers and Redis. Hard kills and failures before resource registration may leave resources behind. `ventis clean` removes generated `stubs/`, `grpc_stubs/`, and -`docker_container/`. It does not remove containers or images. Remove exact -leftovers explicitly and preserve source, port scaffolding, and requested -evidence. +`docker_container/` under `.car`. It does not remove containers or images. Remove exact leftovers explicitly and preserve `app/`, +`config/`, and requested evidence. diff --git a/.claude/skills/porting-to-canyonos-core/references/troubleshooting.md b/.claude/skills/porting-to-canyonos-core/references/troubleshooting.md index 361acf9..d2877c1 100644 --- a/.claude/skills/porting-to-canyonos-core/references/troubleshooting.md +++ b/.claude/skills/porting-to-canyonos-core/references/troubleshooting.md @@ -8,7 +8,10 @@ read [llm-proxy.md](llm-proxy.md) or [ec2.md](ec2.md). | Symptom | Likely cause | |---|---| -| Agent image is missing | Config name matched no yaml name, entrypoint is absent, or build skipped it; inspect build warnings | +| `Application source copy not found` | The command ran somewhere other than the application root, or `.car/app/` was never created | +| `Config file not found: .car/config/...` while `.car` exists | The command ran inside `.car`; it belongs one level up | +| Agent image is missing | Config name matched no declaration in `config/`, entrypoint is absent, or build skipped it; inspect build warnings | +| Wrong declaration is used | Two files in `config/` declare the same `agent.name`; the later filename silently wins | | Two services produce one image | Config names collide after lowercase normalization | | `generated grpc_stubs are missing` | Build did not complete on this host, or generated output was cleaned before deploy | | `int(... NoneType)` during local deploy | Local provider was not written exactly as lowercase `local` | @@ -24,11 +27,13 @@ read [llm-proxy.md](llm-proxy.md) or [ec2.md](ec2.md). | First request says `No agent loaded` | Entrypoint import, class lookup, or constructor failed and the controller swallowed the exception; inspect container logs | | Replica is healthy but serves nothing | Health publication does not prove successful agent loading | | Missing credentials while loading | Env injection is unavailable/misconfigured or the source reads another variable | -| Source module is missing | Original import does not resolve from `/app`; read [packaging.md](packaging.md) | +| Source module is missing | `.car/app` is not rooted at the source's import root, so the original import does not resolve from `/app`; read [packaging.md](packaging.md) | | Third-party module is missing | Distribution is absent from source metadata and config requirements | | Stub import raises `NameError` | yaml argument type is not a bare builtin | -| Source module behaves like an empty stub | A generated stub basename shadowed the source module | -| Runtime-named project module disappears | Shared runtime copy overwrote a root project module with the same name | +| An agent runs in the workflow process instead of a container | The workflow reached the class by a path other than the agent's `entrypoint`, so it got the real module rather than the stub | +| Calls to one agent reach another | Two config entries share an `entrypoint`, so one stub was written over the other | +| Runtime-named module disappears | Shared runtime copy overwrote a module at the root of the source copy with the same name | +| An application file is missing from the image | Only `.py` files are swept out of the copy unless the `sweeps_all_files` capability is available | ## Request is accepted, then fails diff --git a/.claude/skills/porting-to-canyonos-core/validate.py b/.claude/skills/porting-to-canyonos-core/validate.py index 04baf37..70f7b26 100755 --- a/.claude/skills/porting-to-canyonos-core/validate.py +++ b/.claude/skills/porting-to-canyonos-core/validate.py @@ -8,9 +8,12 @@ serves its first request. A replica is not evidence: the controller writes `healthy` to Redis before `_load_agent` runs. - python validate.py [project_dir] [-c config/global_controller.yaml] + python validate.py [artifact_root] [-c config/global_controller.yaml] [--json] [--strict] +`artifact_root` is the `.car` directory: `config/` beside `app/`, the copy of +the application source that becomes /app inside every container. + Exit 1 if any ERROR was reported, 0 otherwise. --strict also fails on warnings. Runtime capabilities vary across CanyonOS Core installations. This script probes @@ -35,6 +38,8 @@ DEFAULT_CONFIG_PATH = "config/global_controller.yaml" +# ventis/cli.py SOURCE_DIR_NAME -- the duplicated application source. +SOURCE_DIR_NAME = "app" # Copied flat into every image over the swept project tree, so a project module # landing flat under one of these names is overwritten. @@ -110,7 +115,6 @@ "env_file": "runtime env-file injection", "editable_install": "editable project installation", "sweeps_all_files": "full project-file sweep", - "stub_two_destinations": "flat and package stub destinations", } @@ -126,7 +130,6 @@ def probe_capabilities(): caps["ventis"] = True caps["editable_install"] = hasattr(stub_generator, "_install_step") caps["sweeps_all_files"] = hasattr(stub_generator, "_sweep_project_files") - caps["stub_two_destinations"] = hasattr(stub_generator, "_stub_destinations") import importlib @@ -202,10 +205,6 @@ def __init__(self, project_dir, capabilities): self.project_dir = project_dir self.capabilities = capabilities self.findings = [] - # A peer agent is imported by the name of its generated stub, which the - # build copies flat into every image. Those are not project modules and - # need no requirement. - self.stub_module_names = set() def add(self, check, level, path, line, summary, mechanism): self.findings.append( @@ -482,76 +481,49 @@ def check_method(report, entrypoint_path, agent_yaml_path, class_name, func, met # ------------------------------------------------------------------ # -def check_stub_imports(report, workflow_path, tree, stub_classes): - """V023 -- the workflow must import a stub as `from agents. import `. +def check_stub_imports(report, workflow_path, tree, stub_modules): + """V023 -- the workflow must import each agent from its own entrypoint module. - The build copies each stub to exactly one path, and for the workflow image - that path is agents/.py. Two ways of writing this line fail, and - the project walks you into both: the flat form is what examples/ uses, and - the class name is the one `ventis build` prints, which is not the one it - writes. + The build writes a stub over exactly one path: the agent's `entrypoint` + inside the source copy. An import that reaches the class any other way -- + flat, through a package re-export, or from a second copy of the module -- + resolves to the real class instead, and the workflow runs the agent + in-process with none of the deployment behind it. The class name is another + trap: `ventis build` prints one with a `Stub` suffix that it never writes. """ for node in ast.walk(tree): - if isinstance(node, ast.Import): - for alias in node.names: - base = alias.name.split(".")[0] - if base in stub_classes: - report.error( - "V023", workflow_path, node.lineno, - f"`import {alias.name}` -- the stub is at " - f"agents/{base}.py, not flat", - "The build copies a stub to one path, and for the " - "workflow that path is under agents/. This is a " - "ModuleNotFoundError the moment the workflow runs. " - f"Write `from agents.{base} import {stub_classes[base]}`.", - ) - continue - if not isinstance(node, ast.ImportFrom) or not node.module: continue - - module = node.module - if module in stub_classes: - report.error( - "V023", workflow_path, node.lineno, - f"`from {module} import ...` -- the stub is at " - f"agents/{module}.py, not flat", - "The build copies a stub to one path, and for the workflow " - "that path is under agents/. The flat form is what this " - "repository's own examples use and it raises " - "ModuleNotFoundError in the workflow image. Write " - f"`from agents.{module} import {stub_classes[module]}`.", - ) - continue - - if not module.startswith("agents."): - continue - base = module.split(".", 1)[1] - expected = stub_classes.get(base) - if expected is None: - continue for alias in node.names: - if alias.name == expected: + name = alias.name + base = name[: -len("Stub")] if name.endswith("Stub") else name + expected = stub_modules.get(base) + if expected is None: continue - if alias.name == f"{expected}Stub": + if name.endswith("Stub"): report.error( "V023", workflow_path, node.lineno, - f"`{alias.name}` is the name the build prints, not the " - f"class it writes", - "generate_agent_stub sets class_name = agent_config['name'] " - "and then recomputes it with a 'Stub' suffix for the log " - "line only. The message names a class that does not exist; " - f"the class is `{expected}`.", + f"`{name}` is the name the build prints, not the class it " + f"writes", + "generate_stub sets class_name = agent_config['name'] and " + "then recomputes it with a 'Stub' suffix for the log line " + "only. The message names a class that does not exist; the " + f"class is `{base}`.", ) - else: + elif node.module != expected: report.error( "V023", workflow_path, node.lineno, - f"`{alias.name}` is not a class the stub for {base} defines", - f"The stub's class carries the agent's own name: `{expected}`.", + f"`from {node.module} import {name}` -- the stub for {name} " + f"is written to {expected.replace('.', '/')}.py", + "The build replaces the module at the agent's entrypoint " + "and nothing else, so this import reaches the real class " + "and runs the agent in this process instead of over gRPC. " + f"Import it from `{expected}`, where the source already " + "keeps it.", ) -def check_workflow(report, workflow_path, stub_classes=None): +def check_workflow(report, workflow_path, stub_modules=None): """V016 V017 V018 V023.""" tree, error = parse_python(workflow_path) if tree is None: @@ -586,8 +558,8 @@ def check_workflow(report, workflow_path, stub_classes=None): else: check_main_signature(report, workflow_path, main) - if stub_classes: - check_stub_imports(report, workflow_path, tree, stub_classes) + if stub_modules: + check_stub_imports(report, workflow_path, tree, stub_modules) # V016 -- deploy() is what starts Flask. if not any( @@ -719,10 +691,10 @@ def check_fused_fanout(report, workflow_path, tree): # ------------------------------------------------------------------ # -def check_flat_collisions(report, project_dir, yaml_paths, entrypoints): +def check_flat_collisions(report, source_dir, entrypoints): """V019 V020 -- later copies land on earlier ones at the context root.""" - for entry in sorted(os.listdir(project_dir)): - path = os.path.join(project_dir, entry) + for entry in sorted(os.listdir(source_dir)): + path = os.path.join(source_dir, entry) if not os.path.isfile(path) or not entry.endswith(".py"): continue @@ -732,32 +704,30 @@ def check_flat_collisions(report, project_dir, yaml_paths, entrypoints): "V019", path, 1, - f"a project module named `{entry}` sits at the project root", + f"a project module named `{entry}` sits at the root of the " + "source copy", "The shared CanyonOS Core runtime is copied flat into the image after " "the project sweep, so this file is overwritten by CanyonOS Core's own " f"{entry}. Rename it or move it into a package directory.", ) - # V020 -- a stub is copied flat under its yaml's basename. - entrypoint_basenames = {os.path.basename(e) for e in entrypoints if e} - for yaml_path in yaml_paths: - stem = os.path.splitext(os.path.basename(yaml_path))[0] - module = f"{stem}.py" - if module in entrypoint_basenames: - continue # the entrypoint is copied last and wins its flat name back - candidate = os.path.join(project_dir, module) - if os.path.isfile(candidate): - report.error( - "V020", - candidate, - 1, - f"`{os.path.basename(yaml_path)}` generates a stub that lands on " - f"`{module}`", - "The yaml's basename names the stub, and the stub is copied flat " - "over the swept tree. Anything importing this module inside the " - "container gets the generated stub instead of the real code. " - "Rename the yaml to match its own entrypoint.", - ) + # V020 -- one module cannot be the entrypoint of two agents. + owners = {} + for name, entrypoint in entrypoints: + owners.setdefault(entrypoint, []).append(name) + for entrypoint, names in sorted(owners.items()): + if len(names) < 2: + continue + report.error( + "V020", + os.path.join(source_dir, entrypoint), + 1, + f"{' and '.join(sorted(names))} both declare `{entrypoint}` as their " + "entrypoint", + "Each agent's stub is written over its own entrypoint, so the two " + "land on one path and the last one built wins. Every caller then " + "reaches whichever agent that was. Give each agent its own module.", + ) # ------------------------------------------------------------------ # @@ -765,7 +735,7 @@ def check_flat_collisions(report, project_dir, yaml_paths, entrypoints): # ------------------------------------------------------------------ # -def check_env_file(report, config, config_path, project_dir): +def check_env_file(report, config, config_path, artifact_dir): """V030 -- gated on detected env-file injection support.""" declared = config.get("env_file") supported = report.capabilities.get("env_file") @@ -806,11 +776,11 @@ def check_env_file(report, config, config_path, project_dir): # duplicate them here. -def check_import_root(report, project_dir, entrypoint_paths): +def check_import_root(report, source_dir, entrypoint_paths): """V031 -- gated on detected editable-install support.""" supported = report.capabilities.get("editable_install") has_metadata = any( - os.path.isfile(os.path.join(project_dir, name)) + os.path.isfile(os.path.join(source_dir, name)) for name in ("pyproject.toml", "setup.py", "setup.cfg") ) @@ -820,11 +790,11 @@ def check_import_root(report, project_dir, entrypoint_paths): if tree is None: continue for name, lineno in toplevel_import_names(tree).items(): - if name in report.stub_module_names or f"{name}.py" in RUNTIME_FLAT_NAMES: + if f"{name}.py" in RUNTIME_FLAT_NAMES: continue - if _resolves_flat(project_dir, name): + if _resolves_flat(source_dir, name): continue - location = _resolves_nested(project_dir, name) + location = _resolves_nested(source_dir, name) if location: non_flat.append((path, lineno, name, location)) @@ -840,7 +810,7 @@ def check_import_root(report, project_dir, entrypoint_paths): path, lineno, f"`import {name}` resolves to {location}, which is not at the " - "project root", + "root of the source copy", "sys.path[0] is /app and this CanyonOS Core runs no editable install, " "so only modules swept to the root import. The adapter raises " "ModuleNotFoundError inside _load_agent and the first request " @@ -854,10 +824,10 @@ def check_import_root(report, project_dir, entrypoint_paths): "V031", path, lineno, - f"`import {name}` resolves to {location}, and the project root " - "has no packaging metadata", - "A pyproject.toml, setup.py or setup.cfg at the port root is " - "what adds `-e .`; metadata nested inside the untouched source " + f"`import {name}` resolves to {location}, and the source copy's " + "root has no packaging metadata", + "A pyproject.toml, setup.py or setup.cfg at the root of the " + "source copy is what adds `-e .`; metadata nested deeper in the " "tree is ignored. Add minimal root metadata pointing at the " "existing package directory. Without it the install is skipped " "silently.", @@ -1001,9 +971,10 @@ def check_requirements_coverage( for name, lineno in sorted(toplevel_import_names(tree).items()): if name in stdlib or name == "ventis": continue - # Provided by the image itself: the shared runtime is copied flat, and - # every agents/*.yaml generates a stub that is copied flat too. - if f"{name}.py" in RUNTIME_FLAT_NAMES or name in report.stub_module_names: + # Provided by the image itself: the shared runtime is copied flat over + # the swept tree. A stub is not listed here -- it replaces a module the + # source copy already carries, so the tree checks below cover it. + if f"{name}.py" in RUNTIME_FLAT_NAMES: continue if _resolves_flat(project_dir, name) or _resolves_nested(project_dir, name): continue @@ -1048,9 +1019,35 @@ def _normalize_distribution(name): # ------------------------------------------------------------------ # -def validate(project_dir, config_path, capabilities): +def find_agent_declarations(config_dir): + """Map agent name -> declaration, for every declaration in `config/`. + + Mirrors ventis/cli.py: declarations sit in `config/` beside the manifest, + which -- like `policy.yaml` -- carries no top-level `agent.name` and so + drops out here. + """ + import glob + + declarations = {} + for path in sorted(glob.glob(os.path.join(config_dir, "*.yaml"))): + data, error = load_yaml(path) + if error is not None or not isinstance(data, dict): + continue + agent = data.get("agent") + name = agent.get("name") if isinstance(agent, dict) else None + if isinstance(name, str) and name: + declarations[name] = (path, agent) + return declarations + + +def module_path(entrypoint): + """Dotted module name an entrypoint has inside the container.""" + return os.path.splitext(entrypoint)[0].replace("\\", "/").replace("/", ".") + + +def validate(artifact_dir, config_path, capabilities): """Inspect only failures hidden behind a successful image build.""" - report = Report(project_dir, capabilities) + report = Report(artifact_dir, capabilities) # The build owns config/YAML syntax and shape validation. We read only enough # valid structure to locate code for the deeper checks below. @@ -1063,23 +1060,21 @@ def validate(project_dir, config_path, capabilities): ) return report - import glob - - yaml_paths = sorted(glob.glob(os.path.join(project_dir, "agents", "*.yaml"))) - report.stub_module_names = { - os.path.splitext(os.path.basename(path))[0] for path in yaml_paths - } + source_dir = os.path.join(artifact_dir, SOURCE_DIR_NAME) + if not os.path.isdir(source_dir): + report.error( + "V032", + artifact_dir, + 0, + f"no `{SOURCE_DIR_NAME}/` beside `config/`", + "The artifact root holds the application source it deploys: " + f"`{SOURCE_DIR_NAME}/` is the copy that becomes /app, and every " + "entrypoint is relative to it. Without it the port has nothing to " + "build and nothing to keep it decoupled from the developer's tree.", + ) + return report - agents_by_name = {} - stub_classes = {} - for path in yaml_paths: - data, yaml_error = load_yaml(path) - agent = data.get("agent") if isinstance(data, dict) else None - name = agent.get("name") if isinstance(agent, dict) else None - if yaml_error is not None or not isinstance(name, str): - continue # ventis build reports malformed agent declarations - agents_by_name[name] = (path, agent) - stub_classes[os.path.splitext(os.path.basename(path))[0]] = name + agents_by_name = find_agent_declarations(os.path.dirname(config_path)) entries = config.get("agents") if not isinstance(entries, list): @@ -1097,42 +1092,50 @@ def validate(project_dir, config_path, capabilities): name = entry.get("name") entrypoint = entry.get("entrypoint") if isinstance(entrypoint, str): - entrypoints.append(entrypoint) + entrypoints.append((name, entrypoint)) if name in agents_by_name: yaml_path, agent_block = agents_by_name[name] - check_adapter(report, yaml_path, agent_block, entry, project_dir) - entrypoint_path = os.path.join(project_dir, entrypoint or "") + check_adapter(report, yaml_path, agent_block, entry, source_dir) + entrypoint_path = os.path.join(source_dir, entrypoint or "") if isinstance(entrypoint, str) and os.path.isfile(entrypoint_path): check_requirements_coverage( - report, project_dir, entry, entrypoint_path, config_path + report, source_dir, entry, entrypoint_path, config_path ) + # Where each agent's stub is written, and therefore the only import that + # reaches it over gRPC. + stub_modules = { + name: module_path(entrypoint) + for name, entrypoint in entrypoints + if name in agents_by_name + } + for entry in entries: if not isinstance(entry, dict) or entry.get("type", "agent") != "workflow": continue workflow_file = entry.get("workflow_file") if not isinstance(workflow_file, str): continue - workflow_path = os.path.join(project_dir, workflow_file) + workflow_path = os.path.join(source_dir, workflow_file) if os.path.isfile(workflow_path): - check_workflow(report, workflow_path, stub_classes) + check_workflow(report, workflow_path, stub_modules) # These survive a green build and otherwise surface only in a container or # on its first request. - check_flat_collisions(report, project_dir, yaml_paths, entrypoints) - check_env_file(report, config, config_path, project_dir) + check_flat_collisions(report, source_dir, entrypoints) + check_env_file(report, config, config_path, artifact_dir) entrypoint_paths = [ - os.path.join(project_dir, e) - for e in entrypoints - if os.path.isfile(os.path.join(project_dir, e)) + os.path.join(source_dir, e) + for _, e in entrypoints + if os.path.isfile(os.path.join(source_dir, e)) ] - check_import_root(report, project_dir, entrypoint_paths) + check_import_root(report, source_dir, entrypoint_paths) port_paths = list(entrypoint_paths) for entry in entries: if isinstance(entry, dict) and isinstance(entry.get("workflow_file"), str): - candidate = os.path.join(project_dir, entry["workflow_file"]) + candidate = os.path.join(source_dir, entry["workflow_file"]) if os.path.isfile(candidate): port_paths.append(candidate) @@ -1142,7 +1145,6 @@ def validate(project_dir, config_path, capabilities): return report - # ------------------------------------------------------------------ # # Output # # ------------------------------------------------------------------ # @@ -1166,7 +1168,7 @@ def _wrap(text, width, indent): return lines -def print_report(report, project_dir): +def print_report(report, artifact_root): caps = report.capabilities if not caps.get("ventis"): print("ventis is not importable here -- capability-gated rules are") @@ -1197,7 +1199,7 @@ def print_report(report, project_dir): errors, warnings = report.counts() if not findings: - print(f"{project_dir}: clean.") + print(f"{artifact_root}: clean.") return print(f"{errors} error(s), {warnings} warning(s).") @@ -1207,13 +1209,16 @@ def main(argv=None): description="Check a CanyonOS Core port against the rules in SKILL.md." ) parser.add_argument( - "project_dir", nargs="?", default=".", help="the port's project root" + "artifact_root", + nargs="?", + default=".", + help="the .car directory holding config/ and app/ (default: the cwd)", ) parser.add_argument( "-c", "--config", default=DEFAULT_CONFIG_PATH, - help=f"config path relative to project_dir (default: {DEFAULT_CONFIG_PATH})", + help=f"config path relative to artifact_root (default: {DEFAULT_CONFIG_PATH})", ) parser.add_argument("--json", action="store_true", help="emit findings as JSON") parser.add_argument( @@ -1221,22 +1226,22 @@ def main(argv=None): ) args = parser.parse_args(argv) - project_dir = os.path.abspath(args.project_dir) + artifact_root = os.path.abspath(args.artifact_root) config_path = ( args.config if os.path.isabs(args.config) - else os.path.join(project_dir, args.config) + else os.path.join(artifact_root, args.config) ) capabilities = probe_capabilities() - report = validate(project_dir, config_path, capabilities) + report = validate(artifact_root, config_path, capabilities) errors, warnings = report.counts() if args.json: print( json.dumps( { - "project_dir": project_dir, + "artifact_root": artifact_root, "capabilities": capabilities, "errors": errors, "warnings": warnings, @@ -1246,7 +1251,7 @@ def main(argv=None): ) ) else: - print_report(report, report.rel(project_dir) or project_dir) + print_report(report, report.rel(artifact_root) or artifact_root) if errors or (args.strict and warnings): return 1 From ab68b58c4cc12fb44822ebd8815493249b474490 Mon Sep 17 00:00:00 2001 From: Nick Huo Date: Wed, 2 Sep 2026 17:47:39 -0700 Subject: [PATCH 02/14] Close the porting-skill gaps the 45-repo corpus surfaced 50 of the corpus's 73 findings are on this skill, and none is a broken feature: every one is "the document did not say, so the porter worked it out." Six of the seven clusters are closed here. Stub-overwrites-entrypoint (12 findings / 9 repos). The build replaces an agent's entrypoint module wholesale in every other image, and the skill documented the fact without its consequences. SKILL.md now gates the entrypoint choice on the five properties that break: another image reading the module for real, a package __init__ re-exporting from it, a path segment that is not an identifier, relative imports in the entrypoint, and module-scope code that performs a real run. runtime-contract.md says what survives around a stub. The joke_writer example claimed the yaml basename names the stub; it does not, and it contradicted runtime-contract.md:73. requirements (7 findings / 7 repos). M10 said only "a list of strings". Added how to build the list from the import graph the image executes rather than the code the porter wrote, how to version it against the source's own lockfile and era, and a resolve-before-you-build step. validate.py false positives (7 findings / 6 repos). `import autogen` is shipped by three distributions and llama-index by a family, so both were permanent W006 noise; the table now maps an import to every distribution that provides it. The import check walked only the entrypoint's own AST, so a dependency one hop away was invisible -- it now walks the transitive graph inside the copy, package __init__ files included, and covers the workflow entry too. New V033/V034/V035 turn three container-startup crashes into preflight errors. sys.stdlib_module_names is 3.10+, so the stdlib filter was empty on older interpreters and the wider walk would have flagged every `import os`; derived it from the interpreter instead. llm_proxy env spellings (5 findings / 5 repos, 3 blocking). One spelling per provider was wrong: langchain_anthropic reads ANTHROPIC_API_URL first, llama-index reads OPENAI_API_BASE and never OPENAI_BASE_URL. Set every spelling, with a table of which wins. A source that builds the HTTP call itself has no hook at all -- that is now report-and-stop. "Streaming is unsupported" was too broad: buffered consumption works, token-by-token does not. global_controller.yaml (3 findings / 3 repos). No complete example existed anywhere. Added one, plus why `database` stays out. Unclustered: asyncio bridge granularity and loop-bound state, probe 2 not reproducing _load_agent's module naming, --env-file as the common case not an aside, session state inside `query`, and the StateGraph rewrite being conditioned on crossing a service boundary. Not closed: non-.py assets (cluster 2, CAN-282). The sweep is .py-only, troubleshooting.md describes it as conditional on a capability that has never existed, and validate.py probes a function ventis does not define. Deferred deliberately: core is taking the second half of can-228, and writing ".py only" now would be wrong as soon as it lands. All three descriptions get aligned in one pass then. This edit changes the skill's tree hash, so it forks the corpus experiment: the 45 rows pinned to skill_sha c249d63512 stop being comparable with anything run after it. --- .../skills/porting-to-canyonos-core/SKILL.md | 228 ++++++++++-- .../references/example-port.md | 5 +- .../references/llm-proxy.md | 53 ++- .../references/runtime-contract.md | 23 ++ .../references/troubleshooting.md | 9 + .../porting-to-canyonos-core/validate.py | 342 ++++++++++++++++-- 6 files changed, 597 insertions(+), 63 deletions(-) diff --git a/.claude/skills/porting-to-canyonos-core/SKILL.md b/.claude/skills/porting-to-canyonos-core/SKILL.md index 1776b98..b5854d9 100644 --- a/.claude/skills/porting-to-canyonos-core/SKILL.md +++ b/.claude/skills/porting-to-canyonos-core/SKILL.md @@ -25,7 +25,7 @@ Port progress: - [ ] 3. Write declarations, adapters, workflow, config - [ ] 4. validate.py reports 0 errors - [ ] 5. ventis build succeeds -- [ ] 6. Both images pass their probes +- [ ] 6. Every image passes its probes, including the peer import - [ ] 7. A real request returns through /status - [ ] 8. Clean up; git status shows nothing outside .car ``` @@ -149,10 +149,20 @@ a distinct resource/replica profile. - Do not create a one-replica service with no distinct resource profile merely to mirror every source graph node. -Rewrite framework-owned edges as ordinary Python. Import the connected node -functions unchanged. Construct runtime-injected service objects from source -configuration; do not invent models, dimensions, stores, or defaults silently. -Report any choice the source does not specify. +Rewrite framework-owned edges as ordinary Python **only where they cross a +service boundary you chose**. A graph whose nodes all land in one agent has no +boundary to express: keep `graph.compile().invoke(...)` and wrap it. Rewriting +it anyway restates control flow the source already had working and buys no +deployment. Import the connected node functions unchanged wherever you do +rewrite. + +Construct runtime-injected service objects from source configuration; do not +invent models, dimensions, stores, or defaults silently. Report any choice the +source does not specify. A service object that holds state across requests -- a +vector store, a memory, a checkpointer built in `__init__` -- makes +`replicas: 1` a correctness requirement rather than a sizing choice, because +the controller picks a replica per call and the others cannot see that state. +Say so in the report; do not leave it implied. ## 3. Write declarations and adapters @@ -169,21 +179,79 @@ required by the generated stub. `returns.type` is documentation; use `dict` or ### Adapter -Write the adapter where the code it wraps already lives, and give the module a -name of its own -- two agents may not share one entrypoint, because each -agent's stub is written over its own entrypoint and the second would land on -the first. Prefer editing the copied module in place over adding a parallel -one; that is what the copy is for. +Write the adapter where the code it wraps already lives, then choose which +module `entrypoint` names with the gates below. The entrypoint exposes a module-level class named exactly `agent.name`. It constructs with no arguments and its declared methods are synchronous. Read -configuration from the environment in `__init__`. Bridge source coroutines -inside a synchronous method with `asyncio.run(...)`. Serialize framework objects +configuration from the environment in `__init__`. Serialize framework objects with their own JSON-safe serializer before returning. Do not duplicate source prompts, tools, schemas, or model calls. Keep the source provider and SDK. +#### Choosing the entrypoint + +The entrypoint is the one module in the copy the build destroys: each agent's +stub is written over its own `entrypoint` path in every image except that +agent's own, so everywhere else that path holds a generated class and nothing +else. Every gate below applies to whichever module you pick; each one +disqualifies it as it stands, so fix the module or point `entrypoint` at +another. + +1. **Does anything else the deployment imports read this module for its real + contents?** The workflow, or any module the workflow imports at module scope. + If yes, the stub replaces it there and the workflow dies at container + startup. Put the adapter in a sibling module that imports this one and point + `entrypoint` at the sibling. This is why "edit the copied module in place" is + a preference and not a rule: it is right only for a module nothing else + imports. +2. **Does the module's package `__init__.py` re-export a name from it?** + (`from .graph import graph`) Python runs `__init__.py` before any submodule, + so every peer image that touches that package re-runs a re-export the stub + cannot satisfy and raises ImportError at startup. Point `entrypoint` at a + module the `__init__` does not re-export from; add one if it re-exports from + all of them. V033. +3. **Is every segment of the path a Python identifier?** `travel-planner.py` and + `steps/06_agent.py` load fine -- the controller loads by file path -- but the + workflow's `from steps.06_agent import X` is a SyntaxError, which no import + guard catches. Rename the file inside the copy, or add a normally-named + sibling that loads it by path and re-exposes the class. V034. +4. **Does the module use relative imports?** (`from . import data_service`) The + controller loads the entrypoint with `spec_from_file_location`, which leaves + `__package__` empty, so every relative import *in the entrypoint itself* + fails at agent load. Make its top-level imports absolute; the modules it + imports keep theirs. V035. +5. **Does module-level code perform a real run?** A script ending in + `result = crew.kickoff(...)` / `print(result)` fires that run on every + container start and every probe, before a request exists. Delete the + invocation and keep the construction: M18 protects prompts, tools, schemas + and model calls, not a script's own main body. + +#### Bridging async + +Use one `asyncio.run(...)` per declared method, at the method boundary, around +the whole call. One per awaited coroutine builds a fresh event loop and a fresh +connection pool per graph superstep. + +If the instance holds anything bound to a loop -- an `asyncio.Lock`, a client +constructed inside a coroutine -- `asyncio.run` cannot be used at all. The +runtime calls declared methods repeatedly on one instance, and the second call +raises `Lock is bound to a different event loop`. Run one persistent loop on a +background thread and submit with `run_coroutine_threadsafe`. Seed any +`ContextVar` the source's async code reads on the calling thread immediately +before submitting: `call_soon_threadsafe` copies the context at schedule time, +not inside the loop. + +#### Multi-turn and session state + +The platform sends one `{query: string}` per request and keeps nothing between +them. A source with per-conversation state -- a `thread_id`, a checkpointer, a +memory keyed by session -- carries that id *inside* `query`: accept either a +bare string or a JSON object in that one field and pass the id through to the +source unchanged. Do not add a second workflow parameter for it; the platform +never sends one. + ### Workflow Expose `main(query: str)` and call `deploy(main, port=...)` at module scope. @@ -214,6 +282,56 @@ Do not fuse dispatch and `.value()` in one comprehension; that silently serializes fan-out. Do not add an `if __name__ == "__main__":` block: the workflow is executed with `__name__ == "__main__"` in production. +### Requirements + +Each image installs the runtime's base list plus that entry's `requirements:` +and nothing else. The source's own `requirements.txt` is never installed -- the +generator writes its own -- and its `pyproject.toml` is installed only where the +editable-install capability is available. Re-declare every runtime distribution +by hand, per entry. + +Build each entry's list from the imports its image *executes*, not from the code +you wrote: + +1. Follow module-scope imports out of the entrypoint (or `workflow_file`) into + the copy, transitively. A workflow that imports `benchmark.py`, which imports + `agent.py`, needs `agent.py`'s distributions even though the workflow makes + no model call. +2. Include the `__init__.py` of every package on those paths -- it runs first. + In a peer image the entrypoint is a stub, but its package `__init__` and its + siblings are real, so that image still installs what they import. +3. Omit distributions reachable only from source files no image imports, such as + a Gradio or Streamlit UI beside the agent. M22 forbids reclassifying a + declared dependency, not declining to ship an unreachable one; name what you + left out in the report. + +`validate.py` walks the same graph and reports what is missing as W006. + +Version them the way the source resolved them, not the way PyPI resolves them +today: + +- **The source has a lockfile** (`poetry.lock`, `uv.lock`, a pinned + `requirements.txt`): copy those exact versions. Repeating the bare names + resolved `langchain` 1.x for one port, which no longer has + `langchain.agents.agent_toolkits` -- the untouched source's own import. +- **The source pins nothing**: cap every fast-moving distribution below its next + major (`langchain<1.0`, `openai<2`). Unpinned means "whatever existed when + this was written", which is not what pip installs today. +- **The source predates a known SDK break**: pin contemporaneous with its last + commit. A 2023 AutoGen script passing `request_timeout=` needs + `pyautogen==0.1.14`, which depends on `openai<1`, not `autogen==0.7.5`, which + floors on `openai>=1.58` where that kwarg is `timeout`. M23 forbids rewriting + the source call, so the pin has to absorb the difference. Compare the source's + commit date against the pin's release date whenever the source hardcodes SDK + kwargs. + +Resolve the list before writing any adapter -- `uv pip compile`, or +`pip install --dry-run -r` into a scratch environment. A source whose own locked +graph is no longer installable (a yanked release series that an unconditional +transitive pin still requires) is a port blocker; one command finds it instead +of one build-fail/pin/rebuild cycle per attempt. Report it and stop rather than +upgrading the source out of the problem. + ### Config For each service, keep these names aligned: @@ -222,11 +340,51 @@ For each service, keep these names aligned: config entry name == yaml agent.name == entrypoint class name ``` -`entrypoint` and `workflow_file` are relative to `.car/app/` and must stay -inside it. Use lowercase `provider: local`; `replicas` is an integer; -`requirements` is a list of distribution-name strings. Put `env_file` at config top level when the -runtime capability is available. Omit `policy.yaml` unless access must be -restricted; if present, give it a non-empty `rules` list. +`.car/config/global_controller.yaml` in full -- every key the runtime reads, +and no others: + +```yaml +agents: + - name: EmailAgent # == yaml agent.name == entrypoint class name + entrypoint: email_assistant.py # relative to .car/app, may not escape it + provider: local # lowercase; `Local` fails an equality test + replicas: 1 # integer + redis_port: 6379 # host port for this node's Redis; default 6379 + resources: # optional; defaults are cpu 1, memory 512 + cpu: 1 + memory: 1024 # MiB + requirements: # see Requirements above + - langgraph + - langchain-openai + + - name: Workflow + type: workflow # the one entry that carries this key + workflow_file: email_workflow.py # relative to .car/app + api_port: 8080 # where /main is served + provider: local + replicas: 1 + redis_port: 6379 + requirements: # its own list; the agent's does not apply here + - langgraph + +poll_interval: 5 # seconds between metrics polls; default 5 + +redis: + host: localhost + port: 6379 + db: 0 + +env_file: .env # relative to the application root, not .car +``` + +Omit `database`. Without it every metrics poll logs `Could not parse SQLAlchemy +URL from given URL string`, once per replica every `poll_interval` seconds -- +expected noise, not a failure, and not a reason to add the key. Adding it drops +a sqlite file at the application root, outside `.car`, which the step-8 +`git status` check then fails on. + +Omit `policy.yaml` unless access must be restricted; if present, give it a +non-empty `rules` list. ## Hard rules @@ -262,6 +420,11 @@ source-integrity rules. The owner column states where each is decided. | M25 | Two agents MUST NOT share one `entrypoint` | V020 | | M26 | `.car` MUST hold `config/` beside `app/`, the source copy | V032 | | M27 | `.car/app` MUST be rooted at the source's import root | V031 | +| M28 | `requirements` MUST cover every distribution that entry's import graph reaches, transitively | W006, probe 2 | +| M29 | The entrypoint MUST NOT be a module another image imports for its real contents | probe 3 | +| M30 | The entrypoint's package `__init__.py` MUST NOT re-export from it | V033 | +| M31 | Every segment of the `entrypoint` path MUST be a Python identifier | V034 | +| M32 | The entrypoint's own imports MUST be absolute | V035 | ## 4. Validate, build, and probe @@ -278,24 +441,39 @@ running `ventis build`. A build that skips this passes, and the port then fails at `docker run` or on the first request, where the message names a container rather than the mistake. -A green build never imports the adapter. Probe each agent image in this order: +A green build never imports the adapter. Probe in this order: ```bash -# Runtime startup path +# 1. Runtime startup path. Every image, agent and workflow alike. docker run --rm ventis- \ python -c "import local_controller" -# Agent load path; the entrypoint keeps its path from the source copy. -# Include --env-file when configured. +# 2. Agent load path. Name the module after the entrypoint's own path, exactly +# as _load_agent does -- spec_from_file_location('m', ...) sets a __name__ the +# runtime never uses and hides relative-import failures until deploy. docker run --rm --env-file ventis- \ python -c "import importlib.util,sys; \ -s=importlib.util.spec_from_file_location('m',''); \ -m=importlib.util.module_from_spec(s);sys.modules['m']=m;s.loader.exec_module(m); \ +p='';n=p[:-3]; \ +s=importlib.util.spec_from_file_location(n,p); \ +m=importlib.util.module_from_spec(s);sys.modules[n]=m;s.loader.exec_module(m); \ m.();print('ok')" + +# 3. Peer-import path, in the workflow image: here the stub stands where the +# entrypoint was and the package around it is real. + docker run --rm ventis- \ + python -c "from import ;print('ok')" ``` -Also probe the workflow image with `python -c "import local_controller"`; it has -its own dependency resolve and generated-stub imports. +Probe 2 needs `--env-file` in the ordinary case, not the exceptional one: a +source that builds its client at module scope (`client = Anthropic()`) fails at +import without it, and the SDK raises on a missing key even when the key is a +placeholder pointed at a proxy. + +Probe 3 is the only one that exercises what the workflow container does at +startup. It is what catches a package `__init__` re-export against a stub, and a +distribution the workflow image needs only because the entrypoint's package +siblings import it. Also probe the workflow image with +`python -c "import local_controller"`; it has its own dependency resolve. Then deploy, send a representative request, and poll its status: diff --git a/.claude/skills/porting-to-canyonos-core/references/example-port.md b/.claude/skills/porting-to-canyonos-core/references/example-port.md index 1a67d48..8308f18 100644 --- a/.claude/skills/porting-to-canyonos-core/references/example-port.md +++ b/.claude/skills/porting-to-canyonos-core/references/example-port.md @@ -105,8 +105,9 @@ The platform sends `{query: string}` only, so the four email fields ride inside ```text validate.py .car 0 errors ventis build ventis-emailagent, ventis-workflow -docker run ... import local_controller both images -docker run --env-file .env ... EmailAgent() loads +docker run ... import local_controller both images +docker run --env-file .env ... EmailAgent() loads +docker run ventis-workflow ... from email_assistant import EmailAgent stub imports ventis deploy 2 replicas ready POST /main 202 {"request_id": ...} GET /status/ status: error, 401 from OpenAI diff --git a/.claude/skills/porting-to-canyonos-core/references/llm-proxy.md b/.claude/skills/porting-to-canyonos-core/references/llm-proxy.md index f726bd7..dbbd6d6 100644 --- a/.claude/skills/porting-to-canyonos-core/references/llm-proxy.md +++ b/.claude/skills/porting-to-canyonos-core/references/llm-proxy.md @@ -8,18 +8,55 @@ explicitly routes model SDKs through it. The proxy redirects provider endpoints; it does not convert providers. Keep the source SDK, model ID, request body, and response parsing unchanged. -Configure only the provider variables the source uses: +## Set every spelling, not the one you expect + +Each SDK generation reads a different base-URL variable, and a wrapper library +reads a different one from the SDK it wraps. Set only the name this reference +used to give and the container reaches the real provider with a placeholder key: +a 401 that reads like a broken port, after validate, build and both probes have +passed. Set all of them for whichever providers the source uses: ```dotenv OPENAI_BASE_URL=http://host.docker.internal:8081/openai/v1 +OPENAI_API_BASE=http://host.docker.internal:8081/openai/v1 ANTHROPIC_BASE_URL=http://host.docker.internal:8081/anthropic +ANTHROPIC_API_URL=http://host.docker.internal:8081/anthropic +ANTHROPIC_API_BASE=http://host.docker.internal:8081/anthropic AWS_ENDPOINT_URL_BEDROCK_RUNTIME=http://host.docker.internal:8081/bedrock ``` +Which name actually wins, for when a call still escapes: + +| Caller | Reads | +|---|---| +| `openai` SDK | `OPENAI_BASE_URL` | +| `langchain_openai` | `OPENAI_API_BASE` | +| `llama_index.llms.openai`, `llama_index.embeddings.openai` | `OPENAI_API_BASE` only -- `resolve_openai_credentials()` never looks at `OPENAI_BASE_URL` | +| `anthropic` SDK | `ANTHROPIC_BASE_URL` | +| `langchain_anthropic` | `ANTHROPIC_API_URL` first, `ANTHROPIC_BASE_URL` as fallback | +| LiteLLM | `ANTHROPIC_API_BASE` | + +Confirm the route rather than assuming it: the proxy logs one line per forwarded +call, so an empty proxy log after a successful request means the container went +straight to the provider. + Some SDKs refuse to initialize without caller credentials. Give agent containers non-secret placeholders only when required. Keep real OpenAI, Anthropic, or AWS credentials in the separate proxy process, not in the port's `env_file`. +## A source with no env hook cannot be proxied + +Some sources build the HTTP call themselves -- `urllib.request` against a module +constant like `API = "https://api.openai.com/v1/responses"` -- and read no +base-URL variable at all. Editing that constant is a source edit M18 and M21 +forbid, so the `env_file` is inert and the container can only ever reach the +real provider. Report this as a proxy blocker and stop. Do not hand the +container a real upstream credential instead. + +Detect it before deploying: grep the source for the provider hostname. A literal +`api.openai.com` or `api.anthropic.com` outside a comment means the call bypasses +the SDK's base-URL resolution entirely. + ## Start locally The proxy defaults conflict with a typical deployment: host loopback is not @@ -42,12 +79,20 @@ The implementation buffers complete requests and responses: - OpenAI and Anthropic non-streaming HTTP calls are forwarded. - Bedrock `invoke` is reissued through the proxy's boto3 client. -- OpenAI/Anthropic streaming is unsupported. - Bedrock `invoke-with-response-stream`, `converse`, and `converse-stream` are unsupported. -Survey the source before selecting the proxy. Do not silently disable streaming; -report the unsupported behavior and stop. +Streaming splits by how the source **consumes** the response, not by whether a +`streaming` flag is set. The proxy buffers, so it forwards anything that reads a +complete response and breaks anything that reads tokens as they arrive: + +- `ChatOpenAI(streaming=True)` reached through `.invoke()` works. LangChain + drains the stream inside the call and returns one message; the proxy sees an + ordinary buffered request. Verified end to end against this proxy. +- `.stream()`, `.astream()`, and a raw `stream=True` read token by token do not. + +Read the call site before deciding. Report and stop only for the second kind; +never silently disable streaming to make the first kind fit. ## Credential behavior diff --git a/.claude/skills/porting-to-canyonos-core/references/runtime-contract.md b/.claude/skills/porting-to-canyonos-core/references/runtime-contract.md index 6d7715a..eb6ce0b 100644 --- a/.claude/skills/porting-to-canyonos-core/references/runtime-contract.md +++ b/.claude/skills/porting-to-canyonos-core/references/runtime-contract.md @@ -77,6 +77,24 @@ travels over gRPC. The agent's own image keeps its real module and receives only its peers' stubs. The validator checks workflow imports against those entrypoints before the workflow image starts. +The `entrypoint` path is the whole of it. The declaration file's own basename +names neither the stub nor the agent, and has no runtime meaning beyond being +discovered in `config/`. + +What the stub replaces is one whole module, and everything else in the copy +still runs around it. In a peer image: + +- the entrypoint's package `__init__.py` is real and runs before the stub is + reached, so a re-export from the stubbed module (`from .graph import graph`) + raises ImportError at container startup -- V033; +- the entrypoint's sibling modules are real, so that image installs *their* + dependencies even though its own code never names them -- W006; +- the stub defines the declared class and nothing else: no module-level + constants, no helper functions, no other class the source module exported. + +The third has no static check. Read what the workflow imports out of that +module. + ## Agent loading and execution The local controller effectively performs: @@ -92,6 +110,11 @@ Consequences: - The class is module-level and named exactly as configured. - Construction takes no arguments. +- The module name is `VENTIS_AGENT_FILE` with `.py` stripped -- directory + separators and all -- so an entrypoint at `pkg/agent.py` loads as the module + `pkg/agent`, which has no parent package. Relative imports in the entrypoint + raise `attempted relative import with no known parent package`; modules it + imports absolutely are unaffected. V035. - Declared methods accept yaml argument names as keyword arguments. - Methods are synchronous; this path does not await a coroutine. - Dicts and lists are JSON-encoded before entering Redis; other results become diff --git a/.claude/skills/porting-to-canyonos-core/references/troubleshooting.md b/.claude/skills/porting-to-canyonos-core/references/troubleshooting.md index d2877c1..3744ec2 100644 --- a/.claude/skills/porting-to-canyonos-core/references/troubleshooting.md +++ b/.claude/skills/porting-to-canyonos-core/references/troubleshooting.md @@ -34,6 +34,13 @@ read [llm-proxy.md](llm-proxy.md) or [ec2.md](ec2.md). | Calls to one agent reach another | Two config entries share an `entrypoint`, so one stub was written over the other | | Runtime-named module disappears | Shared runtime copy overwrote a module at the root of the source copy with the same name | | An application file is missing from the image | Only `.py` files are swept out of the copy unless the `sweeps_all_files` capability is available | +| Peer container raises `ImportError` for a name in another agent's module | That agent's package `__init__.py` re-exports from its entrypoint, which is a stub in this image; V033 | +| `attempted relative import with no known parent package` | The entrypoint's own imports are relative, and it is loaded by path with no parent package; V035 | +| `ModuleNotFoundError` for a distribution this image's own code never imports | The entrypoint's package `__init__` or a sibling imports it; add it to this entry's `requirements:` | +| `SyntaxError` on the workflow's agent import | An `entrypoint` path segment is not a Python identifier; V034 | +| A model call fires at container start, before any request | Module-level code in the entrypoint performs a real run | +| `ModuleNotFoundError` for a submodule that used to exist | An unpinned requirement resolved to a newer major; pin it to what the source resolved | +| `unexpected keyword argument` inside an SDK call | The pinned distribution is newer than the source; pin contemporaneous with the source's commit date | ## Request is accepted, then fails @@ -47,6 +54,8 @@ read [llm-proxy.md](llm-proxy.md) or [ec2.md](ec2.md). | Redis contains a coroutine repr | Adapter method is async and the controller did not await it | | Fan-out is no faster | Dispatch and `.value()` were fused, serializing the calls | | Debug block runs at startup | Workflow is executed with `__name__ == "__main__"` | +| `Lock is bound to a different event loop` on the second request | `asyncio.run` per call, while the instance holds loop-bound state; use one persistent background loop | +| Model call reaches the real provider with the proxy configured | The SDK reads a base-URL variable the env file does not set; read [llm-proxy.md](llm-proxy.md) | ## Deployment platform endpoint diff --git a/.claude/skills/porting-to-canyonos-core/validate.py b/.claude/skills/porting-to-canyonos-core/validate.py index 70f7b26..698f9aa 100755 --- a/.claude/skills/porting-to-canyonos-core/validate.py +++ b/.claude/skills/porting-to-canyonos-core/validate.py @@ -71,26 +71,71 @@ "ipython", "boto3", ] -# Import name -> distribution name, for the handful where they differ and the -# mismatch would otherwise be reported as a missing requirement. +BASE_WORKFLOW_REQUIREMENTS = BASE_AGENT_REQUIREMENTS + [ + "flask", + "sqlalchemy", + "psycopg", +] +# Import name -> every distribution that provides it. A tuple rather than a +# string because more than one distribution can ship the same import name, and +# reporting a correct declaration as a gap teaches the reader to dismiss W006. IMPORT_TO_DISTRIBUTION = { - "attr": "attrs", - "bs4": "beautifulsoup4", - "cv2": "opencv-python", - "dateutil": "python-dateutil", - "dotenv": "python-dotenv", - "grpc": "grpcio", - "grpc_tools": "grpcio-tools", - "jwt": "pyjwt", - "PIL": "pillow", - "psycopg": "psycopg", - "psycopg2": "psycopg2-binary", - "pydantic_settings": "pydantic-settings", - "sklearn": "scikit-learn", - "typing_extensions": "typing-extensions", - "yaml": "pyyaml", + "attr": ("attrs",), + # `import autogen` is shipped by three unrelated distributions: pyautogen + # (Microsoft's original), ag2 (the community continuation), and a package + # literally named autogen. Any of them satisfies the import. + "autogen": ("pyautogen", "ag2", "autogen", "autogen-agentchat"), + "bs4": ("beautifulsoup4",), + "cv2": ("opencv-python",), + "dateutil": ("python-dateutil",), + "dotenv": ("python-dotenv",), + "grpc": ("grpcio",), + "grpc_tools": ("grpcio-tools",), + "jwt": ("pyjwt",), + "PIL": ("pillow",), + "psycopg": ("psycopg",), + "psycopg2": ("psycopg2-binary",), + "pydantic_settings": ("pydantic-settings",), + "sklearn": ("scikit-learn",), + "typing_extensions": ("typing-extensions",), + "yaml": ("pyyaml",), +} +# Import name -> distribution prefix, where the top-level package is shipped by +# a family of distributions rather than one. `import llama_index.llms.openai` +# collapses to `llama_index`, which no correctly scoped requirements list ever +# names: it declares llama-index-core, llama-index-llms-openai and so on. Any +# member of the family satisfies the import. +NAMESPACE_DISTRIBUTIONS = { + "llama_index": "llama-index", } +def _stdlib_names(): + """Module names the interpreter provides without any distribution. + + `sys.stdlib_module_names` exists only from 3.10. Below that, derive the set + from the interpreter's own library directory rather than shipping a list + that rots -- without it every `import os` in the copy becomes a W006, and a + check that cries wolf is a check the reader stops reading. + """ + names = getattr(sys, "stdlib_module_names", None) + if names: + return frozenset(names) + found = set(sys.builtin_module_names) + library = os.path.dirname(os.__file__) + try: + entries = os.listdir(library) + except OSError: + return frozenset(found) + for entry in entries: + if entry.endswith(".py"): + found.add(entry[:-3]) + elif "." not in entry and "-" not in entry: + found.add(entry) + return frozenset(found) + + +STDLIB_MODULE_NAMES = _stdlib_names() + SECRET_PATTERNS = [ (re.compile(r"sk-[A-Za-z0-9_-]{20,}"), "an OpenAI-style secret key"), (re.compile(r"AKIA[0-9A-Z]{16}"), "an AWS access key id"), @@ -309,6 +354,104 @@ def toplevel_import_names(tree): return names +def dotted_import_names(tree): + """Every absolute import in the module as its full dotted path, with lines.""" + names = {} + for node in ast.walk(tree): + if isinstance(node, ast.Import): + for alias in node.names: + names.setdefault(alias.name, node.lineno) + elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module: + names.setdefault(node.module, node.lineno) + return names + + +def _local_module_files(project_dir, dotted): + """Files inside the copy that `import ` executes, outermost first. + + Python runs every package `__init__.py` on the way down before the leaf + module. That is how an image ends up executing code it never names: the + workflow imports `pkg.agent`, `pkg/__init__.py` runs first, and whatever it + imports runs with it. + """ + parts = dotted.split(".") + found = [] + prefix = project_dir + for depth, part in enumerate(parts): + if os.path.isdir(os.path.join(prefix, part)): + init_path = os.path.join(prefix, part, "__init__.py") + if os.path.isfile(init_path): + found.append(init_path) + prefix = os.path.join(prefix, part) + continue + leaf = os.path.join(prefix, part + ".py") + if depth == len(parts) - 1 and os.path.isfile(leaf): + found.append(leaf) + return found + return found + + +def _relative_import_files(project_dir, path, tree): + """Files a module's own `from .sibling import x` imports execute.""" + root = os.path.realpath(project_dir) + found = [] + for node in ast.walk(tree): + if not isinstance(node, ast.ImportFrom) or not node.level: + continue + base = os.path.dirname(path) + for _ in range(node.level - 1): + base = os.path.dirname(base) + resolved = os.path.realpath(base) + if resolved != root and not resolved.startswith(root + os.sep): + continue + target = os.path.join(base, *(node.module.split(".") if node.module else [])) + candidates = [target + ".py", os.path.join(target, "__init__.py")] + candidates += [ + os.path.join(target, alias.name + ".py") for alias in node.names + ] + candidates += [ + os.path.join(target, alias.name, "__init__.py") for alias in node.names + ] + found += [c for c in candidates if os.path.isfile(c)] + return found + + +def reachable_imports(project_dir, root_path): + """Every third-party import the image executes from `root_path`, transitively. + + An image runs far more than the file the config names. `tools/parser.py` is + one hop from an entrypoint and its pdfplumber import is invisible to an AST + walk of the entrypoint alone; a package `__init__.py` three hops up drags in + graphql-core. Both surface only as a ModuleNotFoundError deep inside an + import chain at agent load, long after a green build. + + Returns {dotted import name: (file that imports it, line)} for names that do + not resolve inside the copy. + """ + external = {} + seen = set() + queue = [os.path.realpath(root_path)] + while queue: + path = queue.pop() + if path in seen or not os.path.isfile(path): + continue + seen.add(path) + tree, _ = parse_python(path) + if tree is None: + continue + for dotted, lineno in dotted_import_names(tree).items(): + local = _local_module_files(project_dir, dotted) + if local: + queue += [os.path.realpath(p) for p in local] + else: + external.setdefault(dotted, (path, lineno)) + queue += [ + os.path.realpath(p) + for p in _relative_import_files(project_dir, path, tree) + ] + return external + + # ------------------------------------------------------------------ # # V006-V010 adapter failures hidden by _load_agent # # ------------------------------------------------------------------ # @@ -834,6 +977,102 @@ def check_import_root(report, source_dir, entrypoint_paths): ) +# ------------------------------------------------------------------ # +# V033-V035 traps set by where the entrypoint sits # +# ------------------------------------------------------------------ # + + +def check_entrypoint_module(report, source_dir, name, entrypoint): + """V033 V034 V035. + + Two runtime facts collide here. The build writes this agent's stub over + `entrypoint` in every image except this agent's own, and the controller + loads the real file by path rather than by import. Each breaks a module + layout that is correct everywhere else in Python. + """ + path = os.path.join(source_dir, entrypoint) + if not os.path.isfile(path): + return + + segments = os.path.splitext(entrypoint)[0].replace("\\", "/").split("/") + invalid = [part for part in segments if not part.isidentifier()] + if invalid: + report.error( + "V034", + path, + 0, + f"`{invalid[0]}` in the entrypoint path is not a Python identifier", + "The controller loads the entrypoint by file path, so this file runs " + "-- but the workflow has to import the class from " + f"`{module_path(entrypoint)}` (V023), and that is a SyntaxError, not " + "an ImportError. Rename the file inside the copy, or point " + "`entrypoint` at a normally-named sibling that loads this file by " + "path and re-exposes the class.", + ) + + tree, _ = parse_python(path) + if tree is not None: + for node in ast.walk(tree): + if isinstance(node, ast.ImportFrom) and node.level: + spelling = "." * node.level + (node.module or "") + report.error( + "V035", + path, + node.lineno, + f"the entrypoint's own `from {spelling} import ...` is relative", + "_load_agent loads this file with spec_from_file_location(" + "VENTIS_AGENT_FILE.replace('.py', ''), path). That name keeps " + "the entrypoint's directory separator, so it has no parent " + "package and __package__ is empty: every relative import in " + "this file raises 'attempted relative import with no known " + "parent package' at agent load, behind 'No agent loaded'. " + "Make this file's own top-level imports absolute; modules it " + "imports may keep theirs.", + ) + break + + directory = os.path.dirname(entrypoint) + if not directory: + return + init_path = os.path.join(source_dir, directory, "__init__.py") + if not os.path.isfile(init_path): + return + module = os.path.splitext(os.path.basename(entrypoint))[0] + package = directory.replace("\\", "/").replace("/", ".") + init_tree, _ = parse_python(init_path) + if init_tree is None: + return + for node in ast.walk(init_tree): + if not isinstance(node, ast.ImportFrom): + continue + target = node.module or "" + hit = target == module if node.level else target in ( + module, + f"{package}.{module}", + ) + if not hit and node.level and not node.module: + hit = any(alias.name == module for alias in node.names) + if not hit: + continue + report.error( + "V033", + init_path, + node.lineno, + f"`{package}/__init__.py` re-exports from `{module}`, the entrypoint " + f"for {name}", + "Python runs a package's __init__.py before any of its submodules, " + "and in every image except this agent's own the module at the " + "entrypoint is the generated stub, which defines the agent class and " + f"nothing else. Any peer image that imports anything from `{package}` " + "-- the workflow importing the agent class included -- re-runs this " + "re-export against the stub and dies at container startup with " + f"ImportError. Point `entrypoint` at a module `{package}/__init__.py` " + "does not re-export from; add one that imports the real module if " + "every existing module is re-exported.", + ) + break + + def _resolves_flat(project_dir, name): """Whether Python can resolve `name` with /app as its import root. @@ -936,13 +1175,15 @@ def _pyproject_dependencies(project_dir): def check_requirements_coverage( - report, project_dir, entry, entrypoint_path, config_path + report, project_dir, entry, root_path, config_path, base_requirements ): - """W006 -- an import the container cannot satisfy.""" - tree, _ = parse_python(entrypoint_path) - if tree is None: - return + """W006 -- an import the container cannot satisfy. + Walks the whole import graph the image executes from `root_path`, not just + that one file: a distribution reached through a local module or a package + __init__ is exactly as missing, and exactly as invisible until the container + starts. + """ declared = { _normalize_distribution(item) for item in (entry.get("requirements") or []) @@ -965,11 +1206,13 @@ def check_requirements_coverage( unreadable_metadata = True else: declared |= project_deps - base = {_normalize_distribution(item) for item in BASE_AGENT_REQUIREMENTS} - stdlib = getattr(sys, "stdlib_module_names", frozenset()) + base = {_normalize_distribution(item) for item in base_requirements} + satisfied = base | declared - for name, lineno in sorted(toplevel_import_names(tree).items()): - if name in stdlib or name == "ventis": + external = reachable_imports(project_dir, root_path) + for dotted, (where, lineno) in sorted(external.items()): + name = dotted.split(".")[0] + if name in STDLIB_MODULE_NAMES or name == "ventis": continue # Provided by the image itself: the shared runtime is copied flat over # the swept tree. A stub is not listed here -- it replaces a module the @@ -978,8 +1221,12 @@ def check_requirements_coverage( continue if _resolves_flat(project_dir, name) or _resolves_nested(project_dir, name): continue - distribution = _normalize_distribution(IMPORT_TO_DISTRIBUTION.get(name, name)) - if distribution in base or distribution in declared: + prefix = NAMESPACE_DISTRIBUTIONS.get(name) + if prefix and any( + item == prefix or item.startswith(prefix + "-") for item in declared + ): + continue + if _candidate_distributions(name) & satisfied: continue if unreadable_metadata: mechanism = ( @@ -998,16 +1245,30 @@ def check_requirements_coverage( f"distribution is named something other than `{name}`, declare " f"that name in {report.rel(config_path)}." ) + if os.path.realpath(where) != os.path.realpath(root_path): + mechanism += ( + f" This image never names `{name}` in {report.rel(root_path)}; " + f"it runs {report.rel(where)} on the way there, and that module " + "needs it." + ) report.warn( "W006", - entrypoint_path, + where, lineno, - f"`import {name}` is in neither the runtime's base list nor this " - "entry's `requirements:`", + f"`import {dotted}` is in neither the runtime's base list nor " + f"{entry.get('name') or 'this entry'}'s `requirements:`", mechanism, ) +def _candidate_distributions(name): + """Every distribution name that would satisfy `import `.""" + return { + _normalize_distribution(item) + for item in IMPORT_TO_DISTRIBUTION.get(name, (name,)) + } + + def _normalize_distribution(name): return re.split(r"[<>=!\[;\s]", name.strip().lower(), maxsplit=1)[0].replace( "_", "-" @@ -1098,8 +1359,14 @@ def validate(artifact_dir, config_path, capabilities): check_adapter(report, yaml_path, agent_block, entry, source_dir) entrypoint_path = os.path.join(source_dir, entrypoint or "") if isinstance(entrypoint, str) and os.path.isfile(entrypoint_path): + check_entrypoint_module(report, source_dir, name, entrypoint) check_requirements_coverage( - report, source_dir, entry, entrypoint_path, config_path + report, + source_dir, + entry, + entrypoint_path, + config_path, + BASE_AGENT_REQUIREMENTS, ) # Where each agent's stub is written, and therefore the only import that @@ -1119,6 +1386,17 @@ def validate(artifact_dir, config_path, capabilities): workflow_path = os.path.join(source_dir, workflow_file) if os.path.isfile(workflow_path): check_workflow(report, workflow_path, stub_modules) + # The workflow image installs its own list. A module it imports for + # a helper drags that module's dependencies in even though the + # workflow makes no model call of its own. + check_requirements_coverage( + report, + source_dir, + entry, + workflow_path, + config_path, + BASE_WORKFLOW_REQUIREMENTS, + ) # These survive a green build and otherwise surface only in a container or # on its first request. From 988d247de0bf6705fbe0dc271160ec3d3be31402 Mon Sep 17 00:00:00 2001 From: Nick Huo Date: Wed, 2 Sep 2026 17:47:39 -0700 Subject: [PATCH 03/14] Split step 3 out of SKILL.md into two references SKILL.md had grown to 509 lines and step 3 was 213 of them -- the entrypoint gates, async bridging, session state, the requirements procedure and the full manifest were all inline. That is reference weight sitting in the file that loads on every invocation. Split along the boundary the port already has, one file per authored directory: references/adapter.md what goes into .car/app references/manifest.md what goes into .car/config SKILL.md keeps what has to survive a skim: the name binding, the workflow rules and their two code snippets, and the hard-rules table, which is the one-line-per-rule index into both new files. The reference list now separates mandatory from triggered. Every other reference fires on a symptom -- a failed build, a nested import -- but the rules in these two build green and fail in a container, so a porter cannot know to look them up. Their trigger is step 3 itself, named on the checklist line. 509 -> 364 lines, no content dropped. --- .../skills/porting-to-canyonos-core/SKILL.md | 246 ++++-------------- .../references/adapter.md | 73 ++++++ .../references/manifest.md | 117 +++++++++ 3 files changed, 240 insertions(+), 196 deletions(-) create mode 100644 .claude/skills/porting-to-canyonos-core/references/adapter.md create mode 100644 .claude/skills/porting-to-canyonos-core/references/manifest.md diff --git a/.claude/skills/porting-to-canyonos-core/SKILL.md b/.claude/skills/porting-to-canyonos-core/SKILL.md index b5854d9..f02ff65 100644 --- a/.claude/skills/porting-to-canyonos-core/SKILL.md +++ b/.claude/skills/porting-to-canyonos-core/SKILL.md @@ -22,7 +22,7 @@ maps to one line here. Port progress: - [ ] 1. Copy the source into .car/app, rooted at its import root - [ ] 2. Survey the copy and choose service boundaries -- [ ] 3. Write declarations, adapters, workflow, config +- [ ] 3. Read adapter.md + manifest.md; write declarations, adapters, workflow, config - [ ] 4. validate.py reports 0 errors - [ ] 5. ventis build succeeds - [ ] 6. Every image passes its probes, including the peer import @@ -33,21 +33,32 @@ Port progress: Do not skip step 4. Every failure mode it reports survives a green build and a healthy replica, and then costs a deploy cycle to rediscover. -## Load references only when needed +## References -- Read [references/packaging.md](references/packaging.md) when a source import - does not resolve from `/app`, the source is nested, or packaging metadata is +Two are mandatory, and their trigger is a step rather than a symptom -- a porter +cannot know to look up a rule whose violation builds green: + +- [references/adapter.md](references/adapter.md) -- choosing the entrypoint, + bridging async, session state. Read before writing into `.car/app`. +- [references/manifest.md](references/manifest.md) -- the agent yaml, the + complete manifest, and how to build a `requirements` list. Read before writing + into `.car/config`. + +The rest have triggers: + +- [references/packaging.md](references/packaging.md) when a source import does + not resolve from `/app`, the source is nested, or packaging metadata is involved. -- Read [references/llm-proxy.md](references/llm-proxy.md) only when the target +- [references/llm-proxy.md](references/llm-proxy.md) only when the target includes `llm_proxy`. -- Read [references/ec2.md](references/ec2.md) only when any config entry uses +- [references/ec2.md](references/ec2.md) only when any config entry uses `provider: EC2`. -- Read [references/troubleshooting.md](references/troubleshooting.md) after a - failed build, image probe, deploy, or request. -- Read [references/runtime-contract.md](references/runtime-contract.md) when a +- [references/troubleshooting.md](references/troubleshooting.md) after a failed + build, image probe, deploy, or request. +- [references/runtime-contract.md](references/runtime-contract.md) when a validator finding needs explanation or the runtime mechanism is unclear. -- Read [references/example-port.md](references/example-port.md) for one port end - to end -- the decisions, the files, and the evidence that closed it. +- [references/example-port.md](references/example-port.md) for one port end to + end -- the decisions, the files, and the evidence that closed it. ## Goal: a self-contained `.car`, and a source tree that never learns about it @@ -94,14 +105,11 @@ caches, build outputs, and `.env` files holding real credentials. **Root the copy at the source's import root, which is not always its repository root.** `/app` is the copy, and without the editable-install capability it is -the only entry on `sys.path`. A source whose modules import each other as -`from tools import ...` while living under `src/` has `src/` as its import -root: copy the *contents* of `src/` to `.car/app/`, or every one of those -imports raises `ModuleNotFoundError` inside `_load_agent` and the first -request answers `No agent loaded`. Read the source's own imports, not its -directory names, to decide. Re-rooting is free here in a way it never was -before: `.car/app` is a copy Canyon owns, so nothing in the developer's tree -moves. +the only entry on `sys.path`, so a source under `src/` that imports +`from tools import ...` needs the *contents* of `src/` at `.car/app/`. Read the +source's own imports, not its directory names, to decide. Getting it wrong +builds green and answers `No agent loaded` on the first request; +[references/packaging.md](references/packaging.md) works the case through. ```bash mkdir -p .car/config @@ -166,21 +174,23 @@ Say so in the report; do not leave it implied. ## 3. Write declarations and adapters -### Agent yaml +Read [references/adapter.md](references/adapter.md) before writing an adapter, +and [references/manifest.md](references/manifest.md) before writing +`.car/config`. Neither is optional and neither is triggered by a symptom: every +rule in them builds green and fails inside a container. -Declarations go in `.car/config/`, beside the manifest. The build reads every -`*.yaml` there and keeps the ones with a top-level `agent.name`, so the -manifest and `policy.yaml` drop out on their own. +For each service, keep these names aligned: -Use one yaml per deployed service. Argument types are bare builtins only: -`str`, `int`, `float`, `bool`, `dict`, or `list`. Every declared argument is -required by the generated stub. `returns.type` is documentation; use `dict` or -`list` to signal that workflow callers must `json.loads` the returned string. +```text +config entry name == yaml agent.name == entrypoint class name +``` ### Adapter -Write the adapter where the code it wraps already lives, then choose which -module `entrypoint` names with the gates below. +Write the adapter where the code it wraps already lives. *Which* module +`entrypoint` then names is the decision adapter.md gates: the build writes that +agent's stub over that path in every other image, so it is the one module in +the copy the port destroys. The entrypoint exposes a module-level class named exactly `agent.name`. It constructs with no arguments and its declared methods are synchronous. Read @@ -190,81 +200,19 @@ with their own JSON-safe serializer before returning. Do not duplicate source prompts, tools, schemas, or model calls. Keep the source provider and SDK. -#### Choosing the entrypoint - -The entrypoint is the one module in the copy the build destroys: each agent's -stub is written over its own `entrypoint` path in every image except that -agent's own, so everywhere else that path holds a generated class and nothing -else. Every gate below applies to whichever module you pick; each one -disqualifies it as it stands, so fix the module or point `entrypoint` at -another. - -1. **Does anything else the deployment imports read this module for its real - contents?** The workflow, or any module the workflow imports at module scope. - If yes, the stub replaces it there and the workflow dies at container - startup. Put the adapter in a sibling module that imports this one and point - `entrypoint` at the sibling. This is why "edit the copied module in place" is - a preference and not a rule: it is right only for a module nothing else - imports. -2. **Does the module's package `__init__.py` re-export a name from it?** - (`from .graph import graph`) Python runs `__init__.py` before any submodule, - so every peer image that touches that package re-runs a re-export the stub - cannot satisfy and raises ImportError at startup. Point `entrypoint` at a - module the `__init__` does not re-export from; add one if it re-exports from - all of them. V033. -3. **Is every segment of the path a Python identifier?** `travel-planner.py` and - `steps/06_agent.py` load fine -- the controller loads by file path -- but the - workflow's `from steps.06_agent import X` is a SyntaxError, which no import - guard catches. Rename the file inside the copy, or add a normally-named - sibling that loads it by path and re-exposes the class. V034. -4. **Does the module use relative imports?** (`from . import data_service`) The - controller loads the entrypoint with `spec_from_file_location`, which leaves - `__package__` empty, so every relative import *in the entrypoint itself* - fails at agent load. Make its top-level imports absolute; the modules it - imports keep theirs. V035. -5. **Does module-level code perform a real run?** A script ending in - `result = crew.kickoff(...)` / `print(result)` fires that run on every - container start and every probe, before a request exists. Delete the - invocation and keep the construction: M18 protects prompts, tools, schemas - and model calls, not a script's own main body. - -#### Bridging async - -Use one `asyncio.run(...)` per declared method, at the method boundary, around -the whole call. One per awaited coroutine builds a fresh event loop and a fresh -connection pool per graph superstep. - -If the instance holds anything bound to a loop -- an `asyncio.Lock`, a client -constructed inside a coroutine -- `asyncio.run` cannot be used at all. The -runtime calls declared methods repeatedly on one instance, and the second call -raises `Lock is bound to a different event loop`. Run one persistent loop on a -background thread and submit with `run_coroutine_threadsafe`. Seed any -`ContextVar` the source's async code reads on the calling thread immediately -before submitting: `call_soon_threadsafe` copies the context at schedule time, -not inside the loop. - -#### Multi-turn and session state - -The platform sends one `{query: string}` per request and keeps nothing between -them. A source with per-conversation state -- a `thread_id`, a checkpointer, a -memory keyed by session -- carries that id *inside* `query`: accept either a -bare string or a JSON object in that one field and pass the id through to the -source unchanged. Do not add a second workflow parameter for it; the platform -never sends one. - ### Workflow Expose `main(query: str)` and call `deploy(main, port=...)` at module scope. -Import each agent from its own `entrypoint`, exactly where the source copy -keeps it -- that is the one module the build replaces with a stub: +Import each agent from its own `entrypoint`, exactly where the source copy keeps +it -- that is the one module the build replaces with a stub: ```python from deploy import deploy from . import # the agent's entrypoint path ``` -Any other route to the class -- a flat name, a package re-export, a second -copy of the module -- reaches the real class and runs the agent in the workflow +Any other route to the class -- a flat name, a package re-export, a second copy +of the module -- reaches the real class and runs the agent in the workflow process with none of the deployment behind it. That import needs no rewriting when the source already imported the agent from there. @@ -282,109 +230,15 @@ Do not fuse dispatch and `.value()` in one comprehension; that silently serializes fan-out. Do not add an `if __name__ == "__main__":` block: the workflow is executed with `__name__ == "__main__"` in production. -### Requirements - -Each image installs the runtime's base list plus that entry's `requirements:` -and nothing else. The source's own `requirements.txt` is never installed -- the -generator writes its own -- and its `pyproject.toml` is installed only where the -editable-install capability is available. Re-declare every runtime distribution -by hand, per entry. - -Build each entry's list from the imports its image *executes*, not from the code -you wrote: - -1. Follow module-scope imports out of the entrypoint (or `workflow_file`) into - the copy, transitively. A workflow that imports `benchmark.py`, which imports - `agent.py`, needs `agent.py`'s distributions even though the workflow makes - no model call. -2. Include the `__init__.py` of every package on those paths -- it runs first. - In a peer image the entrypoint is a stub, but its package `__init__` and its - siblings are real, so that image still installs what they import. -3. Omit distributions reachable only from source files no image imports, such as - a Gradio or Streamlit UI beside the agent. M22 forbids reclassifying a - declared dependency, not declining to ship an unreachable one; name what you - left out in the report. - -`validate.py` walks the same graph and reports what is missing as W006. - -Version them the way the source resolved them, not the way PyPI resolves them -today: - -- **The source has a lockfile** (`poetry.lock`, `uv.lock`, a pinned - `requirements.txt`): copy those exact versions. Repeating the bare names - resolved `langchain` 1.x for one port, which no longer has - `langchain.agents.agent_toolkits` -- the untouched source's own import. -- **The source pins nothing**: cap every fast-moving distribution below its next - major (`langchain<1.0`, `openai<2`). Unpinned means "whatever existed when - this was written", which is not what pip installs today. -- **The source predates a known SDK break**: pin contemporaneous with its last - commit. A 2023 AutoGen script passing `request_timeout=` needs - `pyautogen==0.1.14`, which depends on `openai<1`, not `autogen==0.7.5`, which - floors on `openai>=1.58` where that kwarg is `timeout`. M23 forbids rewriting - the source call, so the pin has to absorb the difference. Compare the source's - commit date against the pin's release date whenever the source hardcodes SDK - kwargs. - -Resolve the list before writing any adapter -- `uv pip compile`, or -`pip install --dry-run -r` into a scratch environment. A source whose own locked -graph is no longer installable (a yanked release series that an unconditional -transitive pin still requires) is a port blocker; one command finds it instead -of one build-fail/pin/rebuild cycle per attempt. Report it and stop rather than -upgrading the source out of the problem. - ### Config -For each service, keep these names aligned: - -```text -config entry name == yaml agent.name == entrypoint class name -``` - -`.car/config/global_controller.yaml` in full -- every key the runtime reads, -and no others: - -```yaml -agents: - - name: EmailAgent # == yaml agent.name == entrypoint class name - entrypoint: email_assistant.py # relative to .car/app, may not escape it - provider: local # lowercase; `Local` fails an equality test - replicas: 1 # integer - redis_port: 6379 # host port for this node's Redis; default 6379 - resources: # optional; defaults are cpu 1, memory 512 - cpu: 1 - memory: 1024 # MiB - requirements: # see Requirements above - - langgraph - - langchain-openai - - - name: Workflow - type: workflow # the one entry that carries this key - workflow_file: email_workflow.py # relative to .car/app - api_port: 8080 # where /main is served - provider: local - replicas: 1 - redis_port: 6379 - requirements: # its own list; the agent's does not apply here - - langgraph - -poll_interval: 5 # seconds between metrics polls; default 5 - -redis: - host: localhost - port: 6379 - db: 0 - -env_file: .env # relative to the application root, not .car -``` - -Omit `database`. Without it every metrics poll logs `Could not parse SQLAlchemy -URL from given URL string`, once per replica every `poll_interval` seconds -- -expected noise, not a failure, and not a reason to add the key. Adding it drops -a sqlite file at the application root, outside `.car`, which the step-8 -`git status` check then fails on. - -Omit `policy.yaml` unless access must be restricted; if present, give it a -non-empty `rules` list. +`entrypoint` and `workflow_file` are relative to `.car/app/` and may not escape +it. `provider` is lowercase `local`, `replicas` is an integer, and +`requirements` is a per-entry list of distribution names -- the source's own +`requirements.txt` is never installed into any image. Omit `policy.yaml` unless +access must be restricted; if present, give it a non-empty `rules` list. +manifest.md carries the complete manifest and how to build each `requirements` +list. ## Hard rules diff --git a/.claude/skills/porting-to-canyonos-core/references/adapter.md b/.claude/skills/porting-to-canyonos-core/references/adapter.md new file mode 100644 index 0000000..61d6c07 --- /dev/null +++ b/.claude/skills/porting-to-canyonos-core/references/adapter.md @@ -0,0 +1,73 @@ +# Writing what goes into `.car/app` + +Read this before writing any adapter. Every rule here builds green, passes +`ventis build`, and fails inside a container -- which is why the trigger is the +step, not a symptom. + +## Contents + +- Choosing the entrypoint +- Bridging async +- Multi-turn and session state + +## Choosing the entrypoint + +The entrypoint is the one module in the copy the build destroys: each agent's +stub is written over its own `entrypoint` path in every image except that +agent's own, so everywhere else that path holds a generated class and nothing +else. Every gate below applies to whichever module you pick; each one +disqualifies it as it stands, so fix the module or point `entrypoint` at +another. + +1. **Does anything else the deployment imports read this module for its real + contents?** The workflow, or any module the workflow imports at module scope. + If yes, the stub replaces it there and the workflow dies at container + startup. Put the adapter in a sibling module that imports this one and point + `entrypoint` at the sibling. This is why "edit the copied module in place" is + a preference and not a rule: it is right only for a module nothing else + imports. +2. **Does the module's package `__init__.py` re-export a name from it?** + (`from .graph import graph`) Python runs `__init__.py` before any submodule, + so every peer image that touches that package re-runs a re-export the stub + cannot satisfy and raises ImportError at startup. Point `entrypoint` at a + module the `__init__` does not re-export from; add one if it re-exports from + all of them. V033. +3. **Is every segment of the path a Python identifier?** `travel-planner.py` and + `steps/06_agent.py` load fine -- the controller loads by file path -- but the + workflow's `from steps.06_agent import X` is a SyntaxError, which no import + guard catches. Rename the file inside the copy, or add a normally-named + sibling that loads it by path and re-exposes the class. V034. +4. **Does the module use relative imports?** (`from . import data_service`) The + controller loads the entrypoint with `spec_from_file_location`, which leaves + `__package__` empty, so every relative import *in the entrypoint itself* + fails at agent load. Make its top-level imports absolute; the modules it + imports keep theirs. V035. +5. **Does module-level code perform a real run?** A script ending in + `result = crew.kickoff(...)` / `print(result)` fires that run on every + container start and every probe, before a request exists. Delete the + invocation and keep the construction: M18 protects prompts, tools, schemas + and model calls, not a script's own main body. + +## Bridging async + +Use one `asyncio.run(...)` per declared method, at the method boundary, around +the whole call. One per awaited coroutine builds a fresh event loop and a fresh +connection pool per graph superstep. + +If the instance holds anything bound to a loop -- an `asyncio.Lock`, a client +constructed inside a coroutine -- `asyncio.run` cannot be used at all. The +runtime calls declared methods repeatedly on one instance, and the second call +raises `Lock is bound to a different event loop`. Run one persistent loop on a +background thread and submit with `run_coroutine_threadsafe`. Seed any +`ContextVar` the source's async code reads on the calling thread immediately +before submitting: `call_soon_threadsafe` copies the context at schedule time, +not inside the loop. + +## Multi-turn and session state + +The platform sends one `{query: string}` per request and keeps nothing between +them. A source with per-conversation state -- a `thread_id`, a checkpointer, a +memory keyed by session -- carries that id *inside* `query`: accept either a +bare string or a JSON object in that one field and pass the id through to the +source unchanged. Do not add a second workflow parameter for it; the platform +never sends one. diff --git a/.claude/skills/porting-to-canyonos-core/references/manifest.md b/.claude/skills/porting-to-canyonos-core/references/manifest.md new file mode 100644 index 0000000..e1ffb7d --- /dev/null +++ b/.claude/skills/porting-to-canyonos-core/references/manifest.md @@ -0,0 +1,117 @@ +# Writing what goes into `.car/config` + +Read this before writing the manifest or an agent declaration. `ventis build` +owns yaml syntax; nothing here is syntax. These are the values that build green +and then decide whether a container can import its own dependencies. + +## Contents + +- Agent yaml +- Requirements +- The manifest, in full + +## Agent yaml + +Declarations go in `.car/config/`, beside the manifest. The build reads every +`*.yaml` there and keeps the ones with a top-level `agent.name`, so the +manifest and `policy.yaml` drop out on their own. + +Use one yaml per deployed service. Argument types are bare builtins only: +`str`, `int`, `float`, `bool`, `dict`, or `list`. Every declared argument is +required by the generated stub. `returns.type` is documentation; use `dict` or +`list` to signal that workflow callers must `json.loads` the returned string. + +## Requirements + +Each image installs the runtime's base list plus that entry's `requirements:` +and nothing else. The source's own `requirements.txt` is never installed -- the +generator writes its own -- and its `pyproject.toml` is installed only where the +editable-install capability is available. Re-declare every runtime distribution +by hand, per entry. + +Build each entry's list from the imports its image *executes*, not from the code +you wrote: + +1. Follow module-scope imports out of the entrypoint (or `workflow_file`) into + the copy, transitively. A workflow that imports `benchmark.py`, which imports + `agent.py`, needs `agent.py`'s distributions even though the workflow makes + no model call. +2. Include the `__init__.py` of every package on those paths -- it runs first. + In a peer image the entrypoint is a stub, but its package `__init__` and its + siblings are real, so that image still installs what they import. +3. Omit distributions reachable only from source files no image imports, such as + a Gradio or Streamlit UI beside the agent. M22 forbids reclassifying a + declared dependency, not declining to ship an unreachable one; name what you + left out in the report. + +`validate.py` walks the same graph and reports what is missing as W006. + +Version them the way the source resolved them, not the way PyPI resolves them +today: + +- **The source has a lockfile** (`poetry.lock`, `uv.lock`, a pinned + `requirements.txt`): copy those exact versions. Repeating the bare names + resolved `langchain` 1.x for one port, which no longer has + `langchain.agents.agent_toolkits` -- the untouched source's own import. +- **The source pins nothing**: cap every fast-moving distribution below its next + major (`langchain<1.0`, `openai<2`). Unpinned means "whatever existed when + this was written", which is not what pip installs today. +- **The source predates a known SDK break**: pin contemporaneous with its last + commit. A 2023 AutoGen script passing `request_timeout=` needs + `pyautogen==0.1.14`, which depends on `openai<1`, not `autogen==0.7.5`, which + floors on `openai>=1.58` where that kwarg is `timeout`. M23 forbids rewriting + the source call, so the pin has to absorb the difference. Compare the source's + commit date against the pin's release date whenever the source hardcodes SDK + kwargs. + +Resolve the list before writing any adapter -- `uv pip compile`, or +`pip install --dry-run -r` into a scratch environment. A source whose own locked +graph is no longer installable (a yanked release series that an unconditional +transitive pin still requires) is a port blocker; one command finds it instead +of one build-fail/pin/rebuild cycle per attempt. Report it and stop rather than +upgrading the source out of the problem. + +## The manifest, in full + +`.car/config/global_controller.yaml` in full -- every key the runtime reads, +and no others: + +```yaml +agents: + - name: EmailAgent # == yaml agent.name == entrypoint class name + entrypoint: email_assistant.py # relative to .car/app, may not escape it + provider: local # lowercase; `Local` fails an equality test + replicas: 1 # integer + redis_port: 6379 # host port for this node's Redis; default 6379 + resources: # optional; defaults are cpu 1, memory 512 + cpu: 1 + memory: 1024 # MiB + requirements: # see Requirements above + - langgraph + - langchain-openai + + - name: Workflow + type: workflow # the one entry that carries this key + workflow_file: email_workflow.py # relative to .car/app + api_port: 8080 # where /main is served + provider: local + replicas: 1 + redis_port: 6379 + requirements: # its own list; the agent's does not apply here + - langgraph + +poll_interval: 5 # seconds between metrics polls; default 5 + +redis: + host: localhost + port: 6379 + db: 0 + +env_file: .env # relative to the application root, not .car +``` + +Omit `database`. Without it every metrics poll logs `Could not parse SQLAlchemy +URL from given URL string`, once per replica every `poll_interval` seconds -- +expected noise, not a failure, and not a reason to add the key. Adding it drops +a sqlite file at the application root, outside `.car`, which the step-8 +`git status` check then fails on. From 02738144407581d58ecfa679e11696587f189ae9 Mon Sep 17 00:00:00 2001 From: Nick Huo Date: Wed, 2 Sep 2026 17:47:39 -0700 Subject: [PATCH 04/14] Classify the references by the kind of trigger that fires them The reference index was a flat list whose entries all read as "read this when..." while the triggers were not the same kind of thing at all. Grouped into four, because which group a reference is in decides when it gets opened: Before you write adapter.md, manifest.md -- triggered by the step When the target has packaging.md, llm-proxy.md, ec2.md -- triggered by this shape a fact knowable at step 1 After something failed troubleshooting.md, runtime-contract.md -- triggered by a symptom For orientation example-port.md Each entry now says what is in the file, so the file does not have to be opened to find out whether it is the right one. llm-proxy.md crossed 100 lines when the env-var table went in; added the Contents section the length calls for. Every reference over 100 lines now has one. Verified mechanically: SKILL.md body 370 lines, name 24 chars, description 618 chars, no reference reachable only through another reference, no Windows-style paths. validate.py stays at the skill root rather than moving to scripts/. The convention fits a skill with several scripts; this one has exactly one, and the corpus harness invokes it by that path (canyonos-core-porting-tests/.claude/skills/testing-porting-to-canyonos-core/SKILL.md:106). Moving one file to satisfy a directory convention is not worth breaking a caller in another repo. --- .../skills/porting-to-canyonos-core/SKILL.md | 39 ++++++++++++------- .../references/llm-proxy.md | 9 +++++ 2 files changed, 34 insertions(+), 14 deletions(-) diff --git a/.claude/skills/porting-to-canyonos-core/SKILL.md b/.claude/skills/porting-to-canyonos-core/SKILL.md index f02ff65..3b8f261 100644 --- a/.claude/skills/porting-to-canyonos-core/SKILL.md +++ b/.claude/skills/porting-to-canyonos-core/SKILL.md @@ -35,8 +35,12 @@ healthy replica, and then costs a deploy cycle to rediscover. ## References -Two are mandatory, and their trigger is a step rather than a symptom -- a porter -cannot know to look up a rule whose violation builds green: +Every reference is linked from here and read whole when its trigger fires. What +differs between the groups is the *kind* of trigger. + +**Before you write.** Triggered by the step, not by a symptom: a porter cannot +look up a rule whose violation builds green and fails in a container. Neither is +optional. - [references/adapter.md](references/adapter.md) -- choosing the entrypoint, bridging async, session state. Read before writing into `.car/app`. @@ -44,21 +48,28 @@ cannot know to look up a rule whose violation builds green: complete manifest, and how to build a `requirements` list. Read before writing into `.car/config`. -The rest have triggers: +**When the target has this shape.** Triggered by a fact about the source or the +deployment, all three knowable at step 1. -- [references/packaging.md](references/packaging.md) when a source import does - not resolve from `/app`, the source is nested, or packaging metadata is - involved. -- [references/llm-proxy.md](references/llm-proxy.md) only when the target +- [references/packaging.md](references/packaging.md) -- read when a source + import does not resolve from `/app`, the source is nested, or packaging + metadata is involved. +- [references/llm-proxy.md](references/llm-proxy.md) -- read when the target includes `llm_proxy`. -- [references/ec2.md](references/ec2.md) only when any config entry uses +- [references/ec2.md](references/ec2.md) -- read when any config entry uses `provider: EC2`. -- [references/troubleshooting.md](references/troubleshooting.md) after a failed - build, image probe, deploy, or request. -- [references/runtime-contract.md](references/runtime-contract.md) when a - validator finding needs explanation or the runtime mechanism is unclear. -- [references/example-port.md](references/example-port.md) for one port end to - end -- the decisions, the files, and the evidence that closed it. + +**After something failed.** Triggered by a symptom. + +- [references/troubleshooting.md](references/troubleshooting.md) -- read after a + failed build, image probe, deploy, or request; symptom-to-cause tables. +- [references/runtime-contract.md](references/runtime-contract.md) -- read when + a validator finding needs explanation or the runtime mechanism is unclear. + +**For orientation.** + +- [references/example-port.md](references/example-port.md) -- one LangGraph port + end to end: the decisions, the files, and the evidence that closed it. ## Goal: a self-contained `.car`, and a source tree that never learns about it diff --git a/.claude/skills/porting-to-canyonos-core/references/llm-proxy.md b/.claude/skills/porting-to-canyonos-core/references/llm-proxy.md index dbbd6d6..e43885c 100644 --- a/.claude/skills/porting-to-canyonos-core/references/llm-proxy.md +++ b/.claude/skills/porting-to-canyonos-core/references/llm-proxy.md @@ -3,6 +3,15 @@ Read this only when the target checkout contains `llm_proxy` or the deployment explicitly routes model SDKs through it. +## Contents + +- Preserve provider protocols +- Set every spelling, not the one you expect +- A source with no env hook cannot be proxied +- Start locally +- Supported call shape +- Credential behavior + ## Preserve provider protocols The proxy redirects provider endpoints; it does not convert providers. Keep the From f248baa39b05a5748f190ada999acf1e6cc00ea0 Mon Sep 17 00:00:00 2001 From: Nick Huo Date: Wed, 2 Sep 2026 17:47:39 -0700 Subject: [PATCH 05/14] Stop validate.py copying facts out of ventis BASE_AGENT_REQUIREMENTS and BASE_WORKFLOW_REQUIREMENTS now come from the importable stub_generator, with the literals kept only as a fallback for a machine where ventis is not importable. This is what the file already claims to do for capabilities -- probe the runtime, don't restate it -- and the copy had already drifted: it said `psycopg` where the generator says `psycopg[binary]`. The failure mode is not hypothetical. `sweeps_all_files` probes `_sweep_project_files`, a function ventis has never defined, so the capability reported `no` across all 45 corpus runs and nobody noticed. Checking the other two probes the same way turns up a second instance: `editable_install` probes `_install_step`, which exists on no branch, and the generated Dockerfile runs no editable install at all -- so `no` is the right answer reached the wrong way, and the probe would not notice the capability landing. `env_file` is the one that works, probing `resolve_env_file` which PR #53 defines. Both stale probes are left as found rather than rewritten. packaging.md's "report a capability blocker and stop" and M24's "when editable install is supported" describe a capability that has never existed in any form -- the same shape as the sweep half-state in CAN-282, and the same call: report it, don't paper over it. Also fixes the one ruff finding in validate.py (FURB188), which predates this branch. The tests covering all of this live on nickhuo/close-porting-skill-gaps, out of scope for this skill-only PR. --- .../porting-to-canyonos-core/validate.py | 50 ++++++++++++------- 1 file changed, 33 insertions(+), 17 deletions(-) diff --git a/.claude/skills/porting-to-canyonos-core/validate.py b/.claude/skills/porting-to-canyonos-core/validate.py index 698f9aa..edc21c6 100755 --- a/.claude/skills/porting-to-canyonos-core/validate.py +++ b/.claude/skills/porting-to-canyonos-core/validate.py @@ -60,22 +60,38 @@ } ) -# ventis/stub_generator.py BASE_AGENT_REQUIREMENTS / BASE_WORKFLOW_REQUIREMENTS. -BASE_AGENT_REQUIREMENTS = [ - "grpcio", - "grpcio-tools", - "redis", - "pyyaml", - "psutil", - "ipdb", - "ipython", - "boto3", -] -BASE_WORKFLOW_REQUIREMENTS = BASE_AGENT_REQUIREMENTS + [ - "flask", - "sqlalchemy", - "psycopg", -] +def _base_requirements(): + """What the generator preinstalls, taken from the importable runtime. + + The literals below are a fallback for a machine where `ventis` is not + importable. They are also the only copy of a runtime fact in this file that + nothing checks at run time, and a copied fact rots: the `sweeps_all_files` + probe named a function that never existed and reported `no` for an entire + 45-repository corpus before a porter caught it. Prefer the live values, and + let tests/test_porting_skill_validate.py hold the fallback to them. + """ + agent = [ + "grpcio", + "grpcio-tools", + "redis", + "pyyaml", + "psutil", + "ipdb", + "ipython", + "boto3", + ] + workflow = [*agent, "flask", "sqlalchemy", "psycopg[binary]"] + try: + from ventis import stub_generator + except Exception: # noqa: BLE001 - a broken install must not crash the check + return agent, workflow + return ( + list(getattr(stub_generator, "BASE_AGENT_REQUIREMENTS", agent)), + list(getattr(stub_generator, "BASE_WORKFLOW_REQUIREMENTS", workflow)), + ) + + +BASE_AGENT_REQUIREMENTS, BASE_WORKFLOW_REQUIREMENTS = _base_requirements() # Import name -> every distribution that provides it. A tuple rather than a # string because more than one distribution can ship the same import name, and # reporting a correct declaration as a gap teaches the reader to dismiss W006. @@ -639,7 +655,7 @@ def check_stub_imports(report, workflow_path, tree, stub_modules): continue for alias in node.names: name = alias.name - base = name[: -len("Stub")] if name.endswith("Stub") else name + base = name.removesuffix("Stub") expected = stub_modules.get(base) if expected is None: continue From 2ec4f8a8328143043d014f75e79c9f9266e55212 Mon Sep 17 00:00:00 2001 From: Nick Huo Date: Thu, 3 Sep 2026 12:42:00 -0700 Subject: [PATCH 06/14] Guide porters through canyonos config choices --- .../skills/porting-to-canyonos-core/SKILL.md | 36 +++++++++++++++++- .../references/ec2.md | 6 +++ .../references/manifest.md | 38 +++++++++++++++++++ 3 files changed, 79 insertions(+), 1 deletion(-) diff --git a/.claude/skills/porting-to-canyonos-core/SKILL.md b/.claude/skills/porting-to-canyonos-core/SKILL.md index 3b8f261..11b1250 100644 --- a/.claude/skills/porting-to-canyonos-core/SKILL.md +++ b/.claude/skills/porting-to-canyonos-core/SKILL.md @@ -22,7 +22,8 @@ maps to one line here. Port progress: - [ ] 1. Copy the source into .car/app, rooted at its import root - [ ] 2. Survey the copy and choose service boundaries -- [ ] 3. Read adapter.md + manifest.md; write declarations, adapters, workflow, config +- [ ] 3. Read adapter.md + manifest.md; write declarations, adapters, workflow; + use the `canyonos config` flow to review deployment choices, then write config - [ ] 4. validate.py reports 0 errors - [ ] 5. ventis build succeeds - [ ] 6. Every image passes its probes, including the peer import @@ -251,6 +252,39 @@ access must be restricted; if present, give it a non-empty `rules` list. manifest.md carries the complete manifest and how to build each `requirements` list. +Part of the manifest is derived from the copy; the rest is the developer's +deployment choice, and no reading of the source produces it. Derive what the +source decides. Do not derive what it does not. + +Use the interaction exposed by `cli/canyonos/config.py` as the configuration UX. +Before writing `.car/config/global_controller.yaml`, present the same two +choices -- **View** and **Change** -- rather than silently choosing deployment +settings: + +1. Build the complete candidate manifest in memory from derived values plus the + defaults in manifest.md's ownership table. +2. **View** prints the whole candidate, not a summary. Clearly annotate defaults + and values constrained by the source, such as `replicas: 1` for stateful + in-memory services. +3. **Change** asks, in one batch, only for developer-owned values: provider and + its EC2 block, replicas that are not constrained, resources, ports, secret + file location, and whether access needs restricting. Show the current/default + value for every choice. Apply the answers and show the resulting manifest. +4. Write the reviewed candidate, then run the validator. + +If the coding environment can launch an interactive command, prefer running +`canyonos config` for this review. Otherwise reproduce its View/Change flow in +the conversation; do not skip the review merely because the CLI has no TTY. +Do not ask the developer for derived values such as `entrypoint` or +`requirements`: walking the copied source gives a more reliable answer. + +Never block an unattended port on this interaction. `canyonos integrate` +launches this skill from a prompt, and a corpus run has no one at the terminal. +If no answer is available, write the displayed defaults, name them in the final +report, and keep going. The one exception is a value with no safe default: +`provider: EC2` needs infrastructure identifiers that are wrong to invent, so +an unanswered EC2 choice leaves the entry `local` and says so. + ## Hard rules Capitalized **MUST** and **NEVER** are reserved for port-breaking or diff --git a/.claude/skills/porting-to-canyonos-core/references/ec2.md b/.claude/skills/porting-to-canyonos-core/references/ec2.md index e06daa0..3a80363 100644 --- a/.claude/skills/porting-to-canyonos-core/references/ec2.md +++ b/.claude/skills/porting-to-canyonos-core/references/ec2.md @@ -9,6 +9,12 @@ top-level `ec2` block supplies the runtime's required infrastructure and SSH settings. Read the target checkout's deploy preflight and EC2 runtime before writing the block; do not copy values from an example environment. +These identifiers come from the developer, in step 3's config round -- they are +the one part of the manifest with no safe default. If the round produces no +answer, leave the entry `provider: local` and report that EC2 was requested but +not configured. Never fill the block from an example, a previous port, or +another entry in the same manifest. + Typical required categories are: - AMI and instance type diff --git a/.claude/skills/porting-to-canyonos-core/references/manifest.md b/.claude/skills/porting-to-canyonos-core/references/manifest.md index e1ffb7d..a098469 100644 --- a/.claude/skills/porting-to-canyonos-core/references/manifest.md +++ b/.claude/skills/porting-to-canyonos-core/references/manifest.md @@ -6,10 +6,48 @@ and then decide whether a container can import its own dependencies. ## Contents +- Who decides each key - Agent yaml - Requirements - The manifest, in full +## Who decides each key + +Two kinds of key share one file. A **derived** key has exactly one right answer +and the copy holds it; asking the developer can only make it worse. A +**developer** key is a deployment choice the source does not contain, and +deriving it means guessing and presenting the guess as a reading. + +SKILL.md step 3 shows the whole manifest and then asks about the second column +only, in one round, carrying these defaults. + +| Key | Decided by | Default when unanswered | +|---|---|---| +| `name`, `entrypoint`, `workflow_file`, `type` | derived — service boundaries, step 2 | — | +| `requirements` | derived — the entry's import graph | — | +| `database` | neither; omit it always (see below) | absent | +| `provider` | developer | `local` | +| `ec2:` block, `instance_type` | developer — no default is safe | entry stays `local` | +| `replicas` | developer, *unless* cross-request state forces `1` | `1` | +| `resources.cpu` / `resources.memory` | developer | `1` / `512` MiB | +| `api_port` | developer | `8080` | +| `redis_port`, `redis.host` / `.port` / `.db` | developer | `6379`, `localhost` / `6379` / `0` | +| `poll_interval` | developer | `5` | +| `env_file` | developer — the file's location and whether it exists | `.env` when the survey found credential reads, else absent | +| `policy.yaml` | developer | absent | + +Two entries in that table are not free choices, and saying so is part of showing +the config rather than asking about it: + +- **`replicas` stops being a choice once a service holds cross-request state.** + Where the step-2 survey found such state, SKILL.md already fixes `replicas: 1` + as a correctness requirement, so `1` is derived: report it as a constraint and + do not offer to raise it. +- **EC2 identifiers are wrong to invent.** ec2.md forbids copying them from an + example environment, and a wrong AMI, subnet, or security group fails at + deploy preflight or, worse, provisions something unreachable. Unanswered + means the entry stays `local`. + ## Agent yaml Declarations go in `.car/config/`, beside the manifest. The build reads every From 3528a9ad6d2b5043f6e6265addef8b3ec87a6db3 Mon Sep 17 00:00:00 2001 From: Nick Huo Date: Thu, 3 Sep 2026 14:49:27 -0700 Subject: [PATCH 07/14] Ignore stub-shadowed dependencies in workflow validation --- .../porting-to-canyonos-core/validate.py | 28 +++++++++++++++---- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/.claude/skills/porting-to-canyonos-core/validate.py b/.claude/skills/porting-to-canyonos-core/validate.py index edc21c6..a8f38c9 100755 --- a/.claude/skills/porting-to-canyonos-core/validate.py +++ b/.claude/skills/porting-to-canyonos-core/validate.py @@ -432,8 +432,8 @@ def _relative_import_files(project_dir, path, tree): return found -def reachable_imports(project_dir, root_path): - """Every third-party import the image executes from `root_path`, transitively. +def reachable_imports(project_dir, root_path, shadowed_paths=()): + """Every third-party import the built image executes, transitively. An image runs far more than the file the config names. `tools/parser.py` is one hop from an entrypoint and its pdfplumber import is invisible to an AST @@ -441,15 +441,21 @@ def reachable_imports(project_dir, root_path): graphql-core. Both surface only as a ModuleNotFoundError deep inside an import chain at agent load, long after a green build. + `shadowed_paths` are real source modules replaced in this image before it + runs. In particular, a workflow image receives generated stubs over every + agent entrypoint, so following the overwritten implementations would report + dependencies that image never imports. + Returns {dotted import name: (file that imports it, line)} for names that do not resolve inside the copy. """ external = {} seen = set() + shadowed = {os.path.realpath(path) for path in shadowed_paths} queue = [os.path.realpath(root_path)] while queue: path = queue.pop() - if path in seen or not os.path.isfile(path): + if path in shadowed or path in seen or not os.path.isfile(path): continue seen.add(path) tree, _ = parse_python(path) @@ -1191,7 +1197,13 @@ def _pyproject_dependencies(project_dir): def check_requirements_coverage( - report, project_dir, entry, root_path, config_path, base_requirements + report, + project_dir, + entry, + root_path, + config_path, + base_requirements, + shadowed_paths=(), ): """W006 -- an import the container cannot satisfy. @@ -1225,7 +1237,7 @@ def check_requirements_coverage( base = {_normalize_distribution(item) for item in base_requirements} satisfied = base | declared - external = reachable_imports(project_dir, root_path) + external = reachable_imports(project_dir, root_path, shadowed_paths) for dotted, (where, lineno) in sorted(external.items()): name = dotted.split(".")[0] if name in STDLIB_MODULE_NAMES or name == "ventis": @@ -1392,6 +1404,11 @@ def validate(artifact_dir, config_path, capabilities): for name, entrypoint in entrypoints if name in agents_by_name } + stubbed_entrypoint_paths = [ + os.path.join(source_dir, entrypoint) + for name, entrypoint in entrypoints + if name in agents_by_name + ] for entry in entries: if not isinstance(entry, dict) or entry.get("type", "agent") != "workflow": @@ -1412,6 +1429,7 @@ def validate(artifact_dir, config_path, capabilities): workflow_path, config_path, BASE_WORKFLOW_REQUIREMENTS, + shadowed_paths=stubbed_entrypoint_paths, ) # These survive a green build and otherwise surface only in a container or From 8caed7e57cfe49c7ed62500d364d815985466621 Mon Sep 17 00:00:00 2001 From: Nick Huo Date: Thu, 3 Sep 2026 16:03:50 -0700 Subject: [PATCH 08/14] Standardize CanyonOS artifact preparation --- .../SKILL.md | 46 +++--- .claude/skills/porting-to-canyonos/prepare.py | 156 ++++++++++++++++++ .../references/adapter.md | 0 .../references/ec2.md | 0 .../references/example-port.md | 6 +- .../references/llm-proxy.md | 0 .../references/manifest.md | 0 .../references/packaging.md | 0 .../references/runtime-contract.md | 0 .../references/troubleshooting.md | 0 .../validate.py | 0 11 files changed, 186 insertions(+), 22 deletions(-) rename .claude/skills/{porting-to-canyonos-core => porting-to-canyonos}/SKILL.md (91%) create mode 100755 .claude/skills/porting-to-canyonos/prepare.py rename .claude/skills/{porting-to-canyonos-core => porting-to-canyonos}/references/adapter.md (100%) rename .claude/skills/{porting-to-canyonos-core => porting-to-canyonos}/references/ec2.md (100%) rename .claude/skills/{porting-to-canyonos-core => porting-to-canyonos}/references/example-port.md (96%) rename .claude/skills/{porting-to-canyonos-core => porting-to-canyonos}/references/llm-proxy.md (100%) rename .claude/skills/{porting-to-canyonos-core => porting-to-canyonos}/references/manifest.md (100%) rename .claude/skills/{porting-to-canyonos-core => porting-to-canyonos}/references/packaging.md (100%) rename .claude/skills/{porting-to-canyonos-core => porting-to-canyonos}/references/runtime-contract.md (100%) rename .claude/skills/{porting-to-canyonos-core => porting-to-canyonos}/references/troubleshooting.md (100%) rename .claude/skills/{porting-to-canyonos-core => porting-to-canyonos}/validate.py (100%) diff --git a/.claude/skills/porting-to-canyonos-core/SKILL.md b/.claude/skills/porting-to-canyonos/SKILL.md similarity index 91% rename from .claude/skills/porting-to-canyonos-core/SKILL.md rename to .claude/skills/porting-to-canyonos/SKILL.md index 11b1250..9f8ddc1 100644 --- a/.claude/skills/porting-to-canyonos-core/SKILL.md +++ b/.claude/skills/porting-to-canyonos/SKILL.md @@ -5,8 +5,8 @@ description: Ports existing LangChain, LangGraph, CrewAI, AutoGen, and hand-roll # Port an agent project to CanyonOS Core -Requires Python, Docker, and the `ventis` CLI. `validate.py` in this skill needs -Python 3 and `pyyaml`. +Requires Python, Docker, and the `ventis` CLI. `prepare.py` uses only the Python +standard library; `validate.py` needs Python 3 and `pyyaml`. CanyonOS Core is the product name. Its compatibility executable and Python package remain `ventis`; environment variables and Docker resources retain the @@ -20,7 +20,7 @@ maps to one line here. ``` Port progress: -- [ ] 1. Copy the source into .car/app, rooted at its import root +- [ ] 1. Choose the import root; run prepare.py to create .car/config and .car/app - [ ] 2. Survey the copy and choose service boundaries - [ ] 3. Read adapter.md + manifest.md; write declarations, adapters, workflow; use the `canyonos config` flow to review deployment choices, then write config @@ -109,29 +109,35 @@ model clients, and node bodies—is imported from where the copy keeps it. The port re-expresses only the CanyonOS Core boundary and framework-owned orchestration. -## 1. Duplicate the source, then survey it +## 1. Prepare the artifact tree, then survey it -Copy the application source into `.car/app/`, preserving its structure. Leave -out only what no container should carry: `.git/`, `.car/` itself, virtualenvs, -caches, build outputs, and `.env` files holding real credentials. - -**Root the copy at the source's import root, which is not always its repository -root.** `/app` is the copy, and without the editable-install capability it is -the only entry on `sys.path`, so a source under `src/` that imports -`from tools import ...` needs the *contents* of `src/` at `.car/app/`. Read the -source's own imports, not its directory names, to decide. Getting it wrong -builds green and answers `No agent loaded` on the first request; +**Choose the source's import root, which is not always its repository root.** +`/app` is the copy, and without the editable-install capability it is the only +entry on `sys.path`, so a source under `src/` that imports `from tools import +...` needs the *contents* of `src/` at `.car/app/`. Read the source's own +imports, not its directory names, to decide. Getting it wrong builds green and +answers `No agent loaded` on the first request; [references/packaging.md](references/packaging.md) works the case through. +Once the import root is known, use the skill's preparation script rather than +assembling `.car` with ad hoc copy commands: + ```bash -mkdir -p .car/config -rsync -a --exclude '.git' --exclude '.car' --exclude '.venv' --exclude 'venv' \ - --exclude '__pycache__' --exclude '.env' / .car/app/ +python /prepare.py .car ``` -Every edit from here on is inside `.car`. The application source outside it is -read-only for the rest of the port -- `git status` at the end shows `.car/` and -nothing else. +The script creates `.car/config/` and copies the import root's **contents** into +`.car/app/`, preserving its structure. It excludes version-control data, +`.car`, virtualenvs, caches, build outputs, bytecode, and credential-bearing +`.env*` files while retaining `.env.example`, `.env.sample`, and +`.env.template`. It refuses to merge into an existing `.car/app`; use `--force` +only when intentionally replacing the entire source copy. `--force` leaves an +existing `.car/config/` unchanged. + +Choosing the import root remains a porter decision; the script standardizes +only directory creation and copying. After it runs, every edit is inside +`.car`. The application source outside it is read-only for the rest of the port +-- `git status` at the end shows `.car/` and nothing else. Then survey the copy. Identify: diff --git a/.claude/skills/porting-to-canyonos/prepare.py b/.claude/skills/porting-to-canyonos/prepare.py new file mode 100755 index 0000000..9f282b7 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/prepare.py @@ -0,0 +1,156 @@ +#!/usr/bin/env python3 +"""Create a CanyonOS Core artifact tree from a chosen Python import root. + +The script owns the mechanical part of step 1: it creates ``.car/config`` and +copies the selected import root to ``.car/app`` with development artifacts and +credential files excluded. Choosing the correct import root still requires +reading the application's imports. +""" + +import argparse +import shutil +import sys +import uuid +from pathlib import Path + +EXCLUDED_DIRECTORIES = frozenset( + { + ".car", + ".git", + ".hg", + ".mypy_cache", + ".pytest_cache", + ".ruff_cache", + ".tox", + ".venv", + ".svn", + "__pycache__", + "build", + "dist", + "env", + "htmlcov", + "node_modules", + "venv", + } +) +EXCLUDED_FILE_SUFFIXES = (".pyc", ".pyo") +ENV_TEMPLATES = frozenset({".env.example", ".env.sample", ".env.template"}) + + +def _is_relative_to(path: Path, parent: Path) -> bool: + try: + path.relative_to(parent) + except ValueError: + return False + return True + + +def _ignore_factory(artifact_root: Path): + def ignore(directory: str, names: list[str]) -> set[str]: + directory_path = Path(directory) + ignored = set() + for name in names: + path = directory_path / name + if path.resolve() == artifact_root: + ignored.add(name) + elif path.is_dir() and name in EXCLUDED_DIRECTORIES: + ignored.add(name) + elif name.startswith(".env") and name not in ENV_TEMPLATES: + ignored.add(name) + elif name.endswith(EXCLUDED_FILE_SUFFIXES): + ignored.add(name) + return ignored + + return ignore + + +def prepare(import_root: Path, artifact_root: Path, force: bool = False) -> None: + import_root = import_root.expanduser().resolve() + artifact_root = artifact_root.expanduser().resolve() + app_dir = artifact_root / "app" + config_dir = artifact_root / "config" + + if not import_root.is_dir(): + raise ValueError(f"import root is not a directory: {import_root}") + if import_root == artifact_root: + raise ValueError("artifact root cannot also be the import root") + if _is_relative_to(import_root, artifact_root): + raise ValueError("import root cannot be inside the artifact root") + if app_dir.exists() and not app_dir.is_dir(): + raise ValueError(f"app path exists but is not a directory: {app_dir}") + if app_dir.exists() and not force: + raise FileExistsError( + f"{app_dir} already exists; use --force to replace only the source copy" + ) + if config_dir.exists() and not config_dir.is_dir(): + raise ValueError(f"config path exists but is not a directory: {config_dir}") + + artifact_root.mkdir(parents=True, exist_ok=True) + temporary_app = artifact_root / f".app-{uuid.uuid4().hex}.tmp" + previous_app = artifact_root / f".app-{uuid.uuid4().hex}.previous" + + installed_new_app = False + try: + shutil.copytree( + import_root, + temporary_app, + ignore=_ignore_factory(artifact_root), + copy_function=shutil.copy2, + symlinks=True, + ) + if app_dir.exists(): + app_dir.rename(previous_app) + temporary_app.rename(app_dir) + installed_new_app = True + config_dir.mkdir(exist_ok=True) + except Exception: + if temporary_app.exists(): + shutil.rmtree(temporary_app) + if installed_new_app and app_dir.exists(): + shutil.rmtree(app_dir) + if previous_app.exists(): + previous_app.rename(app_dir) + raise + + if previous_app.exists(): + shutil.rmtree(previous_app) + + +def parse_args(argv=None): + parser = argparse.ArgumentParser( + description="Create .car/config and copy an import root into .car/app." + ) + parser.add_argument( + "import_root", + help="directory whose contents should become the contents of .car/app", + ) + parser.add_argument( + "artifact_root", + nargs="?", + default=".car", + help="artifact directory to create (default: .car)", + ) + parser.add_argument( + "--force", + action="store_true", + help="replace an existing app/ copy; leave config/ unchanged", + ) + return parser.parse_args(argv) + + +def main(argv=None) -> int: + args = parse_args(argv) + try: + prepare(Path(args.import_root), Path(args.artifact_root), args.force) + except (OSError, ValueError) as error: + sys.stderr.write(f"prepare.py: {error}\n") + return 1 + + artifact_root = Path(args.artifact_root) + print(f"Created {artifact_root / 'config'}") + print(f"Copied {Path(args.import_root)} to {artifact_root / 'app'}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/.claude/skills/porting-to-canyonos-core/references/adapter.md b/.claude/skills/porting-to-canyonos/references/adapter.md similarity index 100% rename from .claude/skills/porting-to-canyonos-core/references/adapter.md rename to .claude/skills/porting-to-canyonos/references/adapter.md diff --git a/.claude/skills/porting-to-canyonos-core/references/ec2.md b/.claude/skills/porting-to-canyonos/references/ec2.md similarity index 100% rename from .claude/skills/porting-to-canyonos-core/references/ec2.md rename to .claude/skills/porting-to-canyonos/references/ec2.md diff --git a/.claude/skills/porting-to-canyonos-core/references/example-port.md b/.claude/skills/porting-to-canyonos/references/example-port.md similarity index 96% rename from .claude/skills/porting-to-canyonos-core/references/example-port.md rename to .claude/skills/porting-to-canyonos/references/example-port.md index 8308f18..56bb4f0 100644 --- a/.claude/skills/porting-to-canyonos-core/references/example-port.md +++ b/.claude/skills/porting-to-canyonos/references/example-port.md @@ -33,10 +33,12 @@ until the model calls `Done`. is `src/`, not the repository root: ```bash -rsync -a --exclude '.git' --exclude '.car' --exclude '__pycache__' \ - --exclude '.env' src/ .car/app/ +python /prepare.py src .car ``` +This creates `.car/config/` and copies the contents of `src/` into `.car/app/` +with the standard source and credential exclusions. + Copying the repository root instead puts those modules at `/app/src/tools` while `/app` is the only entry on `sys.path`. The build stays green, the replica reports healthy, and the first request answers `No agent loaded` with diff --git a/.claude/skills/porting-to-canyonos-core/references/llm-proxy.md b/.claude/skills/porting-to-canyonos/references/llm-proxy.md similarity index 100% rename from .claude/skills/porting-to-canyonos-core/references/llm-proxy.md rename to .claude/skills/porting-to-canyonos/references/llm-proxy.md diff --git a/.claude/skills/porting-to-canyonos-core/references/manifest.md b/.claude/skills/porting-to-canyonos/references/manifest.md similarity index 100% rename from .claude/skills/porting-to-canyonos-core/references/manifest.md rename to .claude/skills/porting-to-canyonos/references/manifest.md diff --git a/.claude/skills/porting-to-canyonos-core/references/packaging.md b/.claude/skills/porting-to-canyonos/references/packaging.md similarity index 100% rename from .claude/skills/porting-to-canyonos-core/references/packaging.md rename to .claude/skills/porting-to-canyonos/references/packaging.md diff --git a/.claude/skills/porting-to-canyonos-core/references/runtime-contract.md b/.claude/skills/porting-to-canyonos/references/runtime-contract.md similarity index 100% rename from .claude/skills/porting-to-canyonos-core/references/runtime-contract.md rename to .claude/skills/porting-to-canyonos/references/runtime-contract.md diff --git a/.claude/skills/porting-to-canyonos-core/references/troubleshooting.md b/.claude/skills/porting-to-canyonos/references/troubleshooting.md similarity index 100% rename from .claude/skills/porting-to-canyonos-core/references/troubleshooting.md rename to .claude/skills/porting-to-canyonos/references/troubleshooting.md diff --git a/.claude/skills/porting-to-canyonos-core/validate.py b/.claude/skills/porting-to-canyonos/validate.py similarity index 100% rename from .claude/skills/porting-to-canyonos-core/validate.py rename to .claude/skills/porting-to-canyonos/validate.py From d27ef4b310f57ae9f725308047243acf4ddfcc33 Mon Sep 17 00:00:00 2001 From: Nick Huo Date: Thu, 3 Sep 2026 18:00:45 -0700 Subject: [PATCH 09/14] Stop CanyonOS ports at validation --- .claude/skills/porting-to-canyonos/SKILL.md | 170 +++++++----------- .../porting-to-canyonos/references/adapter.md | 8 +- .../porting-to-canyonos/references/ec2.md | 19 +- .../references/example-port.md | 30 ++-- .../references/llm-proxy.md | 6 +- .../references/manifest.md | 7 +- .../references/packaging.md | 33 +++- .../references/runtime-contract.md | 44 ++--- .../references/troubleshooting.md | 13 +- 9 files changed, 169 insertions(+), 161 deletions(-) diff --git a/.claude/skills/porting-to-canyonos/SKILL.md b/.claude/skills/porting-to-canyonos/SKILL.md index 9f8ddc1..2f8f02a 100644 --- a/.claude/skills/porting-to-canyonos/SKILL.md +++ b/.claude/skills/porting-to-canyonos/SKILL.md @@ -1,17 +1,18 @@ --- -name: porting-to-canyonos-core -description: Ports existing LangChain, LangGraph, CrewAI, AutoGen, and hand-rolled Python agent projects onto CanyonOS Core, whose CLI is `ventis` and whose artifacts live in a `.car` directory. Writes the `.car/config` manifest and declarations, duplicates the source into `.car/app`, writes adapters and the workflow, then validates, builds, deploys and probes. Use when converting, migrating, adapting, packaging, building or deploying an existing agent or multi-agent project onto CanyonOS Core or ventis, when running `ventis build` or `ventis deploy`, or when a `.car` port fails to build, load an agent, or answer a request. +name: porting-to-canyonos +description: Ports existing LangChain, LangGraph, CrewAI, AutoGen, and hand-rolled Python agent projects onto CanyonOS Core, whose CLI is `canyonos` and whose artifacts live in a `.car` directory. Writes `.car/config`, copies source into `.car/app`, writes adapters and the workflow, and validates the port. Stops after validation and asks before running `canyonos deploy`, which performs both build and deployment. Use when converting, migrating, adapting, packaging, validating, or deploying an existing agent or multi-agent project onto CanyonOS Core, or when a `.car` port fails validation, build, load, or deployment. --- # Port an agent project to CanyonOS Core -Requires Python, Docker, and the `ventis` CLI. `prepare.py` uses only the Python -standard library; `validate.py` needs Python 3 and `pyyaml`. +Requires Python, Docker, and the `canyonos` CLI. `prepare.py` uses only the +Python standard library; `validate.py` needs Python 3 and `pyyaml`. -CanyonOS Core is the product name. Its compatibility executable and Python -package remain `ventis`; environment variables and Docker resources retain the -`VENTIS_*` and `ventis-*` prefixes. These are protocol identifiers, not branding -strings. Do not rename them. +CanyonOS Core is the product name and `canyonos` is its user-facing CLI. The +internal compatibility Python package, environment variables, and Docker +resources retain the `ventis`, `VENTIS_*`, and `ventis-*` names. These are +protocol identifiers, not CLI instructions or branding strings. Do not rename +them, and do not tell users to run the obsolete `ventis` CLI. ## Port checklist @@ -24,15 +25,12 @@ Port progress: - [ ] 2. Survey the copy and choose service boundaries - [ ] 3. Read adapter.md + manifest.md; write declarations, adapters, workflow; use the `canyonos config` flow to review deployment choices, then write config -- [ ] 4. validate.py reports 0 errors -- [ ] 5. ventis build succeeds -- [ ] 6. Every image passes its probes, including the peer import -- [ ] 7. A real request returns through /status -- [ ] 8. Clean up; git status shows nothing outside .car +- [ ] 4. validate.py reports 0 errors; report readiness and stop ``` -Do not skip step 4. Every failure mode it reports survives a green build and a -healthy replica, and then costs a deploy cycle to rediscover. +Do not skip step 4. The porting workflow ends when validation reports 0 errors: +report the files created, warnings and unresolved runtime blockers, then stop. +Never build or deploy as an implicit continuation of the port. ## References @@ -53,8 +51,8 @@ optional. deployment, all three knowable at step 1. - [references/packaging.md](references/packaging.md) -- read when a source - import does not resolve from `/app`, the source is nested, or packaging - metadata is involved. + import does not resolve from `/app`, the source is nested, packaging metadata + is involved, or the source reads non-Python files at runtime. - [references/llm-proxy.md](references/llm-proxy.md) -- read when the target includes `llm_proxy`. - [references/ec2.md](references/ec2.md) -- read when any config entry uses @@ -62,8 +60,9 @@ deployment, all three knowable at step 1. **After something failed.** Triggered by a symptom. -- [references/troubleshooting.md](references/troubleshooting.md) -- read after a - failed build, image probe, deploy, or request; symptom-to-cause tables. +- [references/troubleshooting.md](references/troubleshooting.md) -- read after an + explicitly approved deploy fails during build, startup, or a request; + symptom-to-cause tables. - [references/runtime-contract.md](references/runtime-contract.md) -- read when a validator finding needs explanation or the runtime mechanism is unclear. @@ -92,7 +91,7 @@ never inside it: declaration Canyon owns, and `app/`, the copy that becomes `/app` in every container. The container keeps the directory structure the application already had. Write adapters into that copy, in the module the code they wrap already -lives in -- not into new `agents/` and `workflow/` directories. `ventis` +lives in -- not into new `agents/` and `workflow/` directories. `canyonos` commands run from the application root and read `.car` below it. Nothing under `.car` points back out at the application source, and nothing in @@ -141,7 +140,10 @@ only directory creation and copying. After it runs, every edit is inside Then survey the copy. Identify: -1. The source entry point and callable input/output. +1. The source entry point and callable input/output. When the repository has + several plausible implementations, trace the imports from the production + route, CLI, or documented launch path; do not choose the most convenient + graph by filename. 2. Framework-owned control flow (`StateGraph`, `Crew`, `GroupChat`, routing, `Send`, `Command`, interrupts). 3. Runtime-injected services nodes read: stores, context, memory, sessions, or @@ -151,8 +153,20 @@ Then survey the copy. Identify: 6. Model provider, credential names, streaming use, and optional `llm_proxy`. 7. Whether independent work fans out and benefits from separate replicas. 8. Whether source imports resolve from `.car/app`, the root that becomes - `/app`. This is the check the validator turns into V031, and it is the one - most likely to survive a green build and a healthy replica. + `/app`. This is the check the validator turns into V031. +9. Every non-Python file reached at runtime: prompts, CrewAI + `agents.yaml`/`tasks.yaml`, PDFs, templates, schemas, and local corpora. If + `sweeps_all_files` is unavailable, these are runtime blockers even though + `prepare.py` copied them. Do not invent base64 embedding or rewrite hardcoded + paths; report and stop. +10. Whether every Python file on the selected import graph parses. A syntax + error already present in the source is a source defect, not permission to + repair behavior silently. Report it and obtain approval before fixing only + the `.car/app` copy. +11. Suspicious module-level behavior before any import or execution: obfuscated + payloads, network downloads, shell/process calls, credential harvesting, or + destructive filesystem operations. Stop and ask the user when found; do not + import, build, or deploy untrusted code merely because it is a port target. Run the validator now, and again after every change until it reports 0 errors. Execute it; do not read it. Its header detects capabilities directly from the @@ -162,8 +176,9 @@ importable runtime rather than from release history: python /validate.py .car ``` -If config or agent yaml is malformed, the validator defers to `ventis build`. -Capability-gated findings say which runtime behavior is available. +If config or agent yaml is malformed, the validator defers that failure to the +build phase inside a later `canyonos deploy`. Capability-gated findings say +which runtime behavior is available. ## 2. Choose service boundaries @@ -303,18 +318,18 @@ source-integrity rules. The owner column states where each is decided. | M3 | yaml argument names MUST match Python parameter names | V008 | | M4 | yaml argument types MUST be bare builtins | V010 | | M5 | Declared adapter methods MUST be synchronous | V009 | -| M6 | Config names MUST match yaml agent names | build | -| M7 | Config names MUST not collide after lowercase normalization | build output | -| M8 | Local provider MUST be lowercase `local` | deploy preflight | -| M9 | `replicas` MUST be an integer | deploy preflight | -| M10 | `requirements` MUST be a list of strings | build | +| M6 | Config names MUST match yaml agent names | `canyonos deploy` build phase | +| M7 | Config names MUST not collide after lowercase normalization | `canyonos deploy` build output | +| M8 | Local provider MUST be lowercase `local` | `canyonos deploy` preflight | +| M9 | `replicas` MUST be an integer | `canyonos deploy` preflight | +| M10 | `requirements` MUST be a list of strings | `canyonos deploy` build phase | | M11 | Workflow MUST expose `main(query)`; extra parameters MUST default | V016 | | M12 | Workflow MUST NEVER contain a main guard | V017 | | M13 | Fan-out MUST dispatch all calls before resolving any | V018 | | M14 | A module at the root of the copy MUST not take a runtime flat name | V019 | | M15 | Workflow MUST import each agent from its own `entrypoint` module | V023 | -| M16 | Policy MUST be absent or contain a non-empty `rules` list | deploy preflight | -| M17 | EC2 entries MUST satisfy the EC2 deployment contract | deploy preflight | +| M16 | Policy MUST be absent or contain a non-empty `rules` list | `canyonos deploy` preflight | +| M17 | EC2 entries MUST satisfy the EC2 deployment contract | `canyonos deploy` preflight | | M18 | NEVER copy source prompts, tools, schemas, or model calls | review | | M19 | NEVER hardcode or bake a real credential into an image | W003 | | M20 | NEVER write outside `.car`; the application source stays untouched | `git status` | @@ -325,91 +340,42 @@ source-integrity rules. The owner column states where each is decided. | M25 | Two agents MUST NOT share one `entrypoint` | V020 | | M26 | `.car` MUST hold `config/` beside `app/`, the source copy | V032 | | M27 | `.car/app` MUST be rooted at the source's import root | V031 | -| M28 | `requirements` MUST cover every distribution that entry's import graph reaches, transitively | W006, probe 2 | -| M29 | The entrypoint MUST NOT be a module another image imports for its real contents | probe 3 | +| M28 | `requirements` MUST cover every distribution that entry's import graph reaches, transitively | W006, dependency review | +| M29 | The entrypoint MUST NOT be a module another image imports for its real contents | adapter.md review | | M30 | The entrypoint's package `__init__.py` MUST NOT re-export from it | V033 | | M31 | Every segment of the `entrypoint` path MUST be a Python identifier | V034 | | M32 | The entrypoint's own imports MUST be absolute | V035 | -## 4. Validate, build, and probe +## 4. Validate and stop -Run static preflight, then let the build own build-time validation. Both run -from the application root: +Run static preflight from the application root: ```bash python /validate.py .car -ventis build ``` -Fix every ERROR and re-run the validator until it reports 0 errors before -running `ventis build`. A build that skips this passes, and the port then fails -at `docker run` or on the first request, where the message names a container -rather than the mistake. +Fix every ERROR and re-run until it reports 0 errors. Warnings and capability +limitations are not permission to hide risk: list each one in the handoff and +say whether it blocks this source. Confirm that `git status` outside `.car` +shows no change to a file the developer owns. -A green build never imports the adapter. Probe in this order: +At that point, report that the `.car` port is validated and stop. Ask the user a +direct yes/no question before taking the next step: -```bash -# 1. Runtime startup path. Every image, agent and workflow alike. - docker run --rm ventis- \ - python -c "import local_controller" - -# 2. Agent load path. Name the module after the entrypoint's own path, exactly -# as _load_agent does -- spec_from_file_location('m', ...) sets a __name__ the -# runtime never uses and hides relative-import failures until deploy. - docker run --rm --env-file ventis- \ - python -c "import importlib.util,sys; \ -p='';n=p[:-3]; \ -s=importlib.util.spec_from_file_location(n,p); \ -m=importlib.util.module_from_spec(s);sys.modules[n]=m;s.loader.exec_module(m); \ -m.();print('ok')" - -# 3. Peer-import path, in the workflow image: here the stub stands where the -# entrypoint was and the package around it is real. - docker run --rm ventis- \ - python -c "from import ;print('ok')" -``` - -Probe 2 needs `--env-file` in the ordinary case, not the exceptional one: a -source that builds its client at module scope (`client = Anthropic()`) fails at -import without it, and the SDK raises on a missing key even when the key is a -placeholder pointed at a proxy. +> Validation passed. Run `canyonos deploy` now? This will build images and start +> the deployment. -Probe 3 is the only one that exercises what the workflow container does at -startup. It is what catches a package `__init__` re-export against a stub, and a -distribution the workflow image needs only because the entrypoint's package -siblings import it. Also probe the workflow image with -`python -c "import local_controller"`; it has its own dependency resolve. +Do not run a standalone build first. `canyonos deploy` owns both build and +deployment, and must run only after explicit user approval. Silence, an +unattended run, or the original request to "port" is not approval. -Then deploy, send a representative request, and poll its status: +If the user approves, run from the application root: ```bash -ventis deploy -curl -X POST http://localhost:8080/main \ - -H 'Content-Type: application/json' -d '{"query":""}' -curl http://localhost:8080/status/ +canyonos deploy ``` -A successful outer request with a source-level failure still proves the port -reached and returned the source behavior. Record the distinction. - -## 5. Clean up - -After collecting evidence, stop foreground deploy with Ctrl+C and wait for -controller cleanup. Remove exact leftovers if startup crashed. Then remove build -products and exact images from this config: - -```bash -ventis clean -docker image rm ventis- \ - ventis- - -test ! -e .car/stubs && test ! -e .car/grpc_stubs && test ! -e .car/docker_container -docker ps -a --format '{{.Names}}' -``` - -`ventis clean` removes only `.car/stubs/`, `.car/grpc_stubs/`, and -`.car/docker_container/`; it does not remove containers or images. Keep -`.car/config`, `.car/app`, and requested logs or reports. - -Finally, confirm the decoupling held: `git status` outside `.car` reports no -change to any file the developer owns. +Report build or deployment failures without silently changing source behavior, +dependencies, provider, or deployment settings. `canyonos deploy` follows the +controller logs; Ctrl+C stops log monitoring, not necessarily the deployment. +Use `canyonos stop` only when the user asks to stop it. diff --git a/.claude/skills/porting-to-canyonos/references/adapter.md b/.claude/skills/porting-to-canyonos/references/adapter.md index 61d6c07..00e8d37 100644 --- a/.claude/skills/porting-to-canyonos/references/adapter.md +++ b/.claude/skills/porting-to-canyonos/references/adapter.md @@ -1,7 +1,7 @@ # Writing what goes into `.car/app` -Read this before writing any adapter. Every rule here builds green, passes -`ventis build`, and fails inside a container -- which is why the trigger is the +Read this before writing any adapter. Every rule here can pass static build +checks and fail only when a container loads -- which is why the trigger is the step, not a symptom. ## Contents @@ -43,8 +43,8 @@ another. fails at agent load. Make its top-level imports absolute; the modules it imports keep theirs. V035. 5. **Does module-level code perform a real run?** A script ending in - `result = crew.kickoff(...)` / `print(result)` fires that run on every - container start and every probe, before a request exists. Delete the + `result = crew.kickoff(...)` / `print(result)` fires that run whenever the + module loads, before a request exists. Delete the invocation and keep the construction: M18 protects prompts, tools, schemas and model calls, not a script's own main body. diff --git a/.claude/skills/porting-to-canyonos/references/ec2.md b/.claude/skills/porting-to-canyonos/references/ec2.md index 3a80363..486049d 100644 --- a/.claude/skills/porting-to-canyonos/references/ec2.md +++ b/.claude/skills/porting-to-canyonos/references/ec2.md @@ -22,7 +22,7 @@ Typical required categories are: - security groups - SSH user and credentials accepted by the runtime -`ventis deploy` owns basic EC2 config validation. A preflight pass is not proof +`canyonos deploy` owns basic EC2 config validation. A preflight pass is not proof that provisioning, SSH, image transfer, or remote container startup works. ## Networking @@ -35,13 +35,14 @@ The environment file may be copied temporarily to a remote host by runtimes that expose the `env_file` capability. Confirm behavior from the capability probe and target runtime rather than assuming local Docker semantics. -## Probes and cleanup +## Deployment and cleanup -Run the same runtime and adapter probes against the exact image before remote -deployment. After deploy, verify the remote container logs; controller health -can be green even when agent loading failed. +After the user explicitly approves `canyonos deploy`, verify the remote +container logs; controller health can be green even when agent loading failed. +Do not start a separate build or deployment as part of validation. -Stop foreground deploy normally so the controller can terminate recorded EC2 -instances. If provisioning or startup fails before an instance is recorded, -inspect the cloud provider directly and remove exact leaked resources. Never use -a broad cleanup command against unrelated instances. +Ctrl+C stops CLI log monitoring, not necessarily the deployment. Ask before +running `canyonos stop` so the controller can terminate recorded EC2 instances. +If provisioning or startup fails before an instance is recorded, inspect the +cloud provider directly and remove exact leaked resources. Never use a broad +cleanup command against unrelated instances. diff --git a/.claude/skills/porting-to-canyonos/references/example-port.md b/.claude/skills/porting-to-canyonos/references/example-port.md index 56bb4f0..326160a 100644 --- a/.claude/skills/porting-to-canyonos/references/example-port.md +++ b/.claude/skills/porting-to-canyonos/references/example-port.md @@ -1,7 +1,8 @@ # One port, end to end -A LangGraph email assistant, ported and deployed. Read this for the shape of the -decisions; the rules themselves are in SKILL.md. +A LangGraph email assistant, ported and validated, then deployed with explicit +approval. Read this for the shape of the decisions; the rules themselves are in +SKILL.md. ## Contents @@ -99,18 +100,27 @@ relative to `.car/app`. The workflow imports the agent from its entrypoint -- replaces with a stub. The platform sends `{query: string}` only, so the four email fields ride inside -`query` as JSON and the workflow unpacks them. `main` returns a dict, and -`GET /status/` hands it back under `result`. +`query` as JSON and the workflow unpacks them. The adapter returns a dict; the +runtime encodes it once for transport, so the workflow decodes the Future once +and returns an ordinary dict without another `json.dumps`: + +```python +def main(query: str) -> dict: + email = json.loads(query) + triage = json.loads(agent.triage(email_input=email).value()) + if triage["goto"] == "END": + return triage + return json.loads(agent.respond(messages=triage["messages"]).value()) +``` + +`GET /status/` hands that result back under `result`. ## The evidence ```text -validate.py .car 0 errors -ventis build ventis-emailagent, ventis-workflow -docker run ... import local_controller both images -docker run --env-file .env ... EmailAgent() loads -docker run ventis-workflow ... from email_assistant import EmailAgent stub imports -ventis deploy 2 replicas ready +validate.py .car 0 errors; porting workflow stopped +user approval yes, run deployment +canyonos deploy build complete; 2 replicas ready POST /main 202 {"request_id": ...} GET /status/ status: error, 401 from OpenAI ``` diff --git a/.claude/skills/porting-to-canyonos/references/llm-proxy.md b/.claude/skills/porting-to-canyonos/references/llm-proxy.md index e43885c..f660672 100644 --- a/.claude/skills/porting-to-canyonos/references/llm-proxy.md +++ b/.claude/skills/porting-to-canyonos/references/llm-proxy.md @@ -22,8 +22,8 @@ source SDK, model ID, request body, and response parsing unchanged. Each SDK generation reads a different base-URL variable, and a wrapper library reads a different one from the SDK it wraps. Set only the name this reference used to give and the container reaches the real provider with a placeholder key: -a 401 that reads like a broken port, after validate, build and both probes have -passed. Set all of them for whichever providers the source uses: +a 401 that reads like a broken port, after validation and the deployment build +have passed. Set all of them for whichever providers the source uses: ```dotenv OPENAI_BASE_URL=http://host.docker.internal:8081/openai/v1 @@ -79,7 +79,7 @@ curl http://127.0.0.1:8081/healthz Local CanyonOS Core containers resolve `host.docker.internal` through their Docker host mapping. On EC2 that name resolves to each EC2 Docker host, not the -machine running `ventis deploy`. Distributed deployments need a reachable proxy +machine running `canyonos deploy`. Distributed deployments need a reachable proxy address or one proxy on each host. ## Supported call shape diff --git a/.claude/skills/porting-to-canyonos/references/manifest.md b/.claude/skills/porting-to-canyonos/references/manifest.md index a098469..8dc7e07 100644 --- a/.claude/skills/porting-to-canyonos/references/manifest.md +++ b/.claude/skills/porting-to-canyonos/references/manifest.md @@ -1,8 +1,9 @@ # Writing what goes into `.car/config` -Read this before writing the manifest or an agent declaration. `ventis build` -owns yaml syntax; nothing here is syntax. These are the values that build green -and then decide whether a container can import its own dependencies. +Read this before writing the manifest or an agent declaration. The build phase +of `canyonos deploy` owns yaml syntax; nothing here is syntax. These are the +values that can build successfully and then decide whether a container imports +its own dependencies. ## Contents diff --git a/.claude/skills/porting-to-canyonos/references/packaging.md b/.claude/skills/porting-to-canyonos/references/packaging.md index cd8265a..2b7b727 100644 --- a/.claude/skills/porting-to-canyonos/references/packaging.md +++ b/.claude/skills/porting-to-canyonos/references/packaging.md @@ -1,7 +1,8 @@ # Packaging and import roots Read this reference when an adapter imports nested source code, the source uses a -`src/` layout, or V031 reports an import-root problem. +`src/` layout, V031 reports an import-root problem, or the source reads +non-Python files at runtime. ## Contents @@ -10,6 +11,7 @@ Read this reference when an adapter imports nested source code, the source uses - Detect support, do not infer it from release history - Root metadata is the trigger - Dependencies in nested metadata +- Runtime data and configuration files - Validation boundary ## What `/app` can import @@ -105,8 +107,31 @@ config requirements. Report declared-but-unused toolchain dependencies and their image cost; let the owner decide whether source metadata should change. +## Runtime data and configuration files + +`prepare.py` copies non-Python files into `.car/app`, but that does not prove the +runtime's image sweep carries them into a container. Inventory every file opened +by the selected import graph: prompt templates, JSON schemas, PDFs, local +corpora, certificates, and framework configuration such as CrewAI +`agents.yaml` and `tasks.yaml`. + +Run `validate.py` and read its `sweeps_all_files` capability: + +- When available, retain each asset at the same path relative to the chosen + import root. Check any path derived from the original repository root or + process working directory; the container starts from `/app`. +- When unavailable, a required non-Python asset is a runtime blocker. Report it + and stop after validation. Do not conceal the gap by base64-encoding the file + into Python, changing a hardcoded path, or duplicating framework config into + adapter code; those changes restate source-owned data and behavior. + +Do not treat successful construction as evidence that configuration loaded. +Frameworks such as CrewAI may warn about a missing yaml and create an empty +configuration, then fail only when the first agent or task is accessed. Inspect +those decorators and file references statically during the survey. + ## Validation boundary -`ventis build` owns packaging syntax and installation errors. `validate.py` -checks only whether adapter imports appear to require a nested root that the -runtime will not expose. +The build phase of `canyonos deploy` owns packaging syntax and installation +errors. `validate.py` checks only whether adapter imports appear to require a +nested root that the runtime will not expose. diff --git a/.claude/skills/porting-to-canyonos/references/runtime-contract.md b/.claude/skills/porting-to-canyonos/references/runtime-contract.md index eb6ce0b..9d96aff 100644 --- a/.claude/skills/porting-to-canyonos/references/runtime-contract.md +++ b/.claude/skills/porting-to-canyonos/references/runtime-contract.md @@ -1,8 +1,8 @@ # CanyonOS Core runtime contract -The product is CanyonOS Core. Compatibility identifiers remain `ventis` for the -CLI and Python package, `VENTIS_*` for runtime variables, and `ventis-*` for -Docker resources. +The product is CanyonOS Core and its user-facing CLI is `canyonos`. Compatibility +identifiers remain `ventis` for the internal Python package, `VENTIS_*` for +runtime variables, and `ventis-*` for Docker resources. Read this reference when implementing an adapter or explaining a validator finding. Runtime-dependent behavior is expressed as capabilities; run @@ -23,10 +23,10 @@ release history. ## Artifact root and discovery -`ventis build` runs from the application root and reads `.car` below it. That -artifact root holds `config/` beside `app/`, the copy of the application source -that becomes `/app` inside every image. Paths in the config are relative to -`app/` and may not escape it. +The build phase of `canyonos deploy` runs from the application root and reads +`.car` below it. That artifact root holds `config/` beside `app/`, the copy of +the application source that becomes `/app` inside every image. Paths in the +config are relative to `app/` and may not escape it. | Input | Discovery | |---|---| @@ -123,8 +123,8 @@ Consequences: Agent import and construction exceptions are caught by the controller. A failed agent may still advertise healthy because health is written independently of -successful agent loading. That is why image probes import both the runtime and -the entrypoint explicitly. +successful agent loading. After an explicitly approved deployment, inspect +container logs rather than treating health as proof that the entrypoint loaded. ## Workflow execution @@ -145,7 +145,7 @@ and resolving inside one comprehension serializes work without raising an error; dispatch all calls first, then resolve them. The workflow container also starts runtime controller code and has its own -package resolution. Probe it independently from agent images. +package resolution. A failure there can differ from failures in agent images. ## Build context and collisions @@ -191,9 +191,9 @@ produce a green image build that dies on: import local_controller ``` -Always run that probe before probing the entrypoint. Treat a generated-code / -runtime-version mismatch as a CanyonOS Core runtime issue, not a reason to alter -source dependencies silently. +Treat a generated-code/runtime-version mismatch during an explicitly approved +`canyonos deploy` as a CanyonOS Core runtime issue, not a reason to alter source +dependencies silently. ## Credentials capability @@ -208,8 +208,8 @@ source needs credentials, report the capability blocker rather than hardcoding or vendoring a secret. A source that constructs its client at import time works only when credentials -are already in the container environment. Image entrypoint probes therefore use -the same env file as deployment. +are already in the container environment. After an explicitly approved deploy, +check loading failures against the same env file configured for deployment. For proxy-specific credential separation, read [llm-proxy.md](llm-proxy.md). @@ -224,10 +224,12 @@ and remote networking are covered in [ec2.md](ec2.md). ## Cleanup boundary -Stopping foreground deploy normally invokes controller cleanup for recorded -containers and Redis. Hard kills and failures before resource registration may -leave resources behind. +`canyonos deploy` follows controller logs after starting the deployment. Ctrl+C +stops that log stream; use `canyonos stop` to request controller teardown when +the user asks to stop the deployment. Hard kills and failures before resource +registration may leave resources behind. -`ventis clean` removes generated `stubs/`, `grpc_stubs/`, and -`docker_container/` under `.car`. It does not remove containers or images. Remove exact leftovers explicitly and preserve `app/`, -`config/`, and requested evidence. +`canyonos clean` removes generated `stubs/`, `grpc_stubs/`, and +`docker_container/`; it does not remove containers or images. Remove exact +leftovers explicitly and preserve `.car/app`, `.car/config`, and requested +evidence. diff --git a/.claude/skills/porting-to-canyonos/references/troubleshooting.md b/.claude/skills/porting-to-canyonos/references/troubleshooting.md index 3744ec2..ef40ecd 100644 --- a/.claude/skills/porting-to-canyonos/references/troubleshooting.md +++ b/.claude/skills/porting-to-canyonos/references/troubleshooting.md @@ -1,7 +1,8 @@ # Troubleshooting -Read this after a failed build, image probe, deploy, or request. For mechanisms, -read [runtime-contract.md](runtime-contract.md). For proxy or remote-host failures, +Read this after an explicitly approved `canyonos deploy` fails during build, +startup, or a request. For mechanisms, read +[runtime-contract.md](runtime-contract.md). For proxy or remote-host failures, read [llm-proxy.md](llm-proxy.md) or [ec2.md](ec2.md). ## Build or deploy stops early @@ -18,6 +19,7 @@ read [llm-proxy.md](llm-proxy.md) or [ec2.md](ec2.md). | Replica conversion `TypeError` | `replicas` is not an integer | | Policy `AttributeError` | Policy exists but is empty or has null/non-list rules; remove it when unrestricted | | Port or container name already in use | A previous deployment did not complete cleanup | +| Build phase reports `No space left on device` | Docker layers and the package-manager cache exceeded the host budget; inspect `df -h` and `docker system df`, report the pressure, and ask before deleting caches or images | ## Container exits or serves nothing @@ -69,6 +71,7 @@ read [llm-proxy.md](llm-proxy.md) or [ec2.md](ec2.md). | Symptom | Likely cause | |---|---| -| `ventis clean` succeeds but containers remain | The command removes generated directories only | -| `ventis clean` succeeds but images remain | Image deletion is separate and requires exact tags | -| Next deployment collides with old resources | Foreground deploy was killed or crashed before controller cleanup | +| Ctrl+C leaves the deployment running | `canyonos deploy` follows logs; Ctrl+C stops monitoring, not the deployment. Ask before running `canyonos stop` | +| `canyonos clean` succeeds but containers remain | The command removes generated directories only | +| `canyonos clean` succeeds but images remain | Image deletion is separate and requires exact tags | +| Next deployment collides with old resources | The deployment was killed or crashed before controller cleanup | From 2e0c2bd32b44c87f648a24e2d7313c42cca64255 Mon Sep 17 00:00:00 2001 From: Nick Huo Date: Fri, 4 Sep 2026 10:32:46 -0700 Subject: [PATCH 10/14] Keep the porting skill focused on validation --- .claude/skills/porting-to-canyonos/SKILL.md | 10 ++-------- .../porting-to-canyonos/references/troubleshooting.md | 1 - 2 files changed, 2 insertions(+), 9 deletions(-) diff --git a/.claude/skills/porting-to-canyonos/SKILL.md b/.claude/skills/porting-to-canyonos/SKILL.md index 2f8f02a..bdd8890 100644 --- a/.claude/skills/porting-to-canyonos/SKILL.md +++ b/.claude/skills/porting-to-canyonos/SKILL.md @@ -163,10 +163,6 @@ Then survey the copy. Identify: error already present in the source is a source defect, not permission to repair behavior silently. Report it and obtain approval before fixing only the `.car/app` copy. -11. Suspicious module-level behavior before any import or execution: obfuscated - payloads, network downloads, shell/process calls, credential harvesting, or - destructive filesystem operations. Stop and ask the user when found; do not - import, build, or deploy untrusted code merely because it is a port target. Run the validator now, and again after every change until it reports 0 errors. Execute it; do not read it. Its header detects capabilities directly from the @@ -375,7 +371,5 @@ If the user approves, run from the application root: canyonos deploy ``` -Report build or deployment failures without silently changing source behavior, -dependencies, provider, or deployment settings. `canyonos deploy` follows the -controller logs; Ctrl+C stops log monitoring, not necessarily the deployment. -Use `canyonos stop` only when the user asks to stop it. +Do not add build, probe, deployment-debugging, or cleanup work to this skill's +porting flow. diff --git a/.claude/skills/porting-to-canyonos/references/troubleshooting.md b/.claude/skills/porting-to-canyonos/references/troubleshooting.md index ef40ecd..213163c 100644 --- a/.claude/skills/porting-to-canyonos/references/troubleshooting.md +++ b/.claude/skills/porting-to-canyonos/references/troubleshooting.md @@ -19,7 +19,6 @@ read [llm-proxy.md](llm-proxy.md) or [ec2.md](ec2.md). | Replica conversion `TypeError` | `replicas` is not an integer | | Policy `AttributeError` | Policy exists but is empty or has null/non-list rules; remove it when unrestricted | | Port or container name already in use | A previous deployment did not complete cleanup | -| Build phase reports `No space left on device` | Docker layers and the package-manager cache exceeded the host budget; inspect `df -h` and `docker system df`, report the pressure, and ask before deleting caches or images | ## Container exits or serves nothing From 61cc5afbfd63e999df1b85edbae4440c755df16b Mon Sep 17 00:00:00 2001 From: Nick Huo Date: Fri, 4 Sep 2026 13:02:07 -0700 Subject: [PATCH 11/14] Push the survey and the refresh flow out of SKILL.md, and split the validator SKILL.md was carrying three things that only matter once: the step-1 survey list, the rules for refreshing an existing `.car/app`, and adapter prose that adapter.md already gates. All three move out, and SKILL.md keeps the decision and the pointer. - references/source-survey.md: the survey, read after preparing the copy and before choosing service boundaries. - references/refresh.md: read when `.car/app` exists and the source moved on. prepare.py grows `--refresh`, which updates source-owned files while preserving port edits and stops atomically when both sides changed one path -- the case that silently loses an adapter. It now also rejects symbolic links, which either escape the self-contained artifact or are skipped by the runtime sweep, so neither outcome is worth carrying. validate.py splits into a validation package rather than growing further: core.py result and YAML primitives manifest.py the public artifact contract, fail-closed python_source.py static source discovery for adapter and packaging checks runtime.py runtime capabilities and dependency facts references/manifest.md gains the `canyonos config` View/Change review flow and re-scopes its opening: the validator checks YAML structure and the artifact contract before an approved deploy, so this reference explains how to derive the values rather than what the build will reject. (cherry picked from commit d2c82d6bfab4e0ffcb1d12b64b2470eb529be981) --- .claude/skills/porting-to-canyonos/SKILL.md | 209 ++----- .claude/skills/porting-to-canyonos/prepare.py | 239 +++++++- .../references/example-port.md | 2 +- .../references/manifest.md | 35 +- .../references/packaging.md | 2 +- .../porting-to-canyonos/references/refresh.md | 38 ++ .../references/source-survey.md | 29 + .../skills/porting-to-canyonos/validate.py | 535 ++---------------- .../validation/__init__.py | 2 + .../porting-to-canyonos/validation/core.py | 99 ++++ .../validation/manifest.py | 295 ++++++++++ .../validation/python_source.py | 152 +++++ .../porting-to-canyonos/validation/runtime.py | 123 ++++ 13 files changed, 1095 insertions(+), 665 deletions(-) create mode 100644 .claude/skills/porting-to-canyonos/references/refresh.md create mode 100644 .claude/skills/porting-to-canyonos/references/source-survey.md create mode 100644 .claude/skills/porting-to-canyonos/validation/__init__.py create mode 100644 .claude/skills/porting-to-canyonos/validation/core.py create mode 100644 .claude/skills/porting-to-canyonos/validation/manifest.py create mode 100644 .claude/skills/porting-to-canyonos/validation/python_source.py create mode 100644 .claude/skills/porting-to-canyonos/validation/runtime.py diff --git a/.claude/skills/porting-to-canyonos/SKILL.md b/.claude/skills/porting-to-canyonos/SKILL.md index bdd8890..1dcfb8f 100644 --- a/.claude/skills/porting-to-canyonos/SKILL.md +++ b/.claude/skills/porting-to-canyonos/SKILL.md @@ -25,10 +25,10 @@ Port progress: - [ ] 2. Survey the copy and choose service boundaries - [ ] 3. Read adapter.md + manifest.md; write declarations, adapters, workflow; use the `canyonos config` flow to review deployment choices, then write config -- [ ] 4. validate.py reports 0 errors; report readiness and stop +- [ ] 4. validate.py exits 0; report readiness and stop ``` -Do not skip step 4. The porting workflow ends when validation reports 0 errors: +Do not skip step 4. The porting workflow ends when validation exits 0: report the files created, warnings and unresolved runtime blockers, then stop. Never build or deploy as an implicit continuation of the port. @@ -53,6 +53,10 @@ deployment, all three knowable at step 1. - [references/packaging.md](references/packaging.md) -- read when a source import does not resolve from `/app`, the source is nested, packaging metadata is involved, or the source reads non-Python files at runtime. +- [references/source-survey.md](references/source-survey.md) -- read after + preparing the copy and before choosing service boundaries. +- [references/refresh.md](references/refresh.md) -- read when `.car/app` already + exists and the source has changed; preserve port edits while refreshing it. - [references/llm-proxy.md](references/llm-proxy.md) -- read when the target includes `llm_proxy`. - [references/ec2.md](references/ec2.md) -- read when any config entry uses @@ -122,59 +126,29 @@ Once the import root is known, use the skill's preparation script rather than assembling `.car` with ad hoc copy commands: ```bash -python /prepare.py .car +python3 /prepare.py .car ``` The script creates `.car/config/` and copies the import root's **contents** into `.car/app/`, preserving its structure. It excludes version-control data, `.car`, virtualenvs, caches, build outputs, bytecode, and credential-bearing `.env*` files while retaining `.env.example`, `.env.sample`, and -`.env.template`. It refuses to merge into an existing `.car/app`; use `--force` -only when intentionally replacing the entire source copy. `--force` leaves an -existing `.car/config/` unchanged. +`.env.template`. It rejects symbolic links because they either escape the +self-contained artifact or are skipped by the runtime source sweep. + +If `.car/app` already exists, read [references/refresh.md](references/refresh.md) +and use `--refresh`. It updates source-owned files while preserving port edits, +and stops atomically when both sides changed one path. Use `--force` only to +discard every edit in `.car/app`; it leaves `.car/config/` unchanged. Choosing the import root remains a porter decision; the script standardizes only directory creation and copying. After it runs, every edit is inside `.car`. The application source outside it is read-only for the rest of the port -- `git status` at the end shows `.car/` and nothing else. -Then survey the copy. Identify: - -1. The source entry point and callable input/output. When the repository has - several plausible implementations, trace the imports from the production - route, CLI, or documented launch path; do not choose the most convenient - graph by filename. -2. Framework-owned control flow (`StateGraph`, `Crew`, `GroupChat`, routing, - `Send`, `Command`, interrupts). -3. Runtime-injected services nodes read: stores, context, memory, sessions, or - callback managers. -4. Sync versus async boundaries. -5. Imports and declared runtime dependencies. -6. Model provider, credential names, streaming use, and optional `llm_proxy`. -7. Whether independent work fans out and benefits from separate replicas. -8. Whether source imports resolve from `.car/app`, the root that becomes - `/app`. This is the check the validator turns into V031. -9. Every non-Python file reached at runtime: prompts, CrewAI - `agents.yaml`/`tasks.yaml`, PDFs, templates, schemas, and local corpora. If - `sweeps_all_files` is unavailable, these are runtime blockers even though - `prepare.py` copied them. Do not invent base64 embedding or rewrite hardcoded - paths; report and stop. -10. Whether every Python file on the selected import graph parses. A syntax - error already present in the source is a source defect, not permission to - repair behavior silently. Report it and obtain approval before fixing only - the `.car/app` copy. - -Run the validator now, and again after every change until it reports 0 errors. -Execute it; do not read it. Its header detects capabilities directly from the -importable runtime rather than from release history: - -```bash -python /validate.py .car -``` - -If config or agent yaml is malformed, the validator defers that failure to the -build phase inside a later `canyonos deploy`. Capability-gated findings say -which runtime behavior is available. +Read [references/source-survey.md](references/source-survey.md), survey the +copy, and run the validator. The survey determines the source facts used in the +next two steps; do not infer them from framework conventions. ## 2. Choose service boundaries @@ -214,143 +188,52 @@ For each service, keep these names aligned: config entry name == yaml agent.name == entrypoint class name ``` -### Adapter - -Write the adapter where the code it wraps already lives. *Which* module -`entrypoint` then names is the decision adapter.md gates: the build writes that -agent's stub over that path in every other image, so it is the one module in -the copy the port destroys. - -The entrypoint exposes a module-level class named exactly `agent.name`. It -constructs with no arguments and its declared methods are synchronous. Read -configuration from the environment in `__init__`. Serialize framework objects -with their own JSON-safe serializer before returning. - -Do not duplicate source prompts, tools, schemas, or model calls. Keep the source -provider and SDK. +Write a no-argument, synchronous adapter class at the entrypoint selected by +`adapter.md`. Import source-owned behavior instead of duplicating it. Expose +`main(query: str)` in the workflow, import every service from its exact +entrypoint module, and call `deploy(main, port=...)` at module scope. -### Workflow - -Expose `main(query: str)` and call `deploy(main, port=...)` at module scope. -Import each agent from its own `entrypoint`, exactly where the source copy keeps -it -- that is the one module the build replaces with a stub: +For parallel remote calls, dispatch before resolving: ```python -from deploy import deploy -from . import # the agent's entrypoint path +futures = [agent.work(item=item) for item in items] +results = [json.loads(future.value()) for future in futures] ``` -Any other route to the class -- a flat name, a package re-export, a second copy -of the module -- reaches the real class and runs the agent in the workflow -process with none of the deployment behind it. That import needs no rewriting -when the source already imported the agent from there. +Do not fuse dispatch and `.value()` in one comprehension. Do not add a main +guard; the workflow executes as `__main__` in production. -The deployment platform sends `{query: string}` to `/main`. Pack richer input -inside `query`; any additional workflow parameter has a default. +Build declarations and per-image requirements from the copied import graph. +Then use the View/Change flow in `manifest.md` (and `canyonos config` when +interactive) to review developer-owned deployment choices. Write only the +reviewed candidate and rerun validation. -Dispatch every remote call before resolving any future: +## Source-integrity boundary -```python -futures = [agent.work(item=item) for item in items] -results = [json.loads(future.value()) for future in futures] -``` +The validator owns mechanical runtime rules; do not duplicate its check list in +the prompt. The porter owns the rules static analysis cannot prove: + +- Never edit outside `.car` or copy source-owned prompts, tools, schemas, model + calls, and node bodies into an adapter. +- Never swap the source provider, invent runtime configuration, or silently + move, drop, or reclassify a dependency. +- Rewrite framework control flow only where it crosses a service boundary; + preserve it inside a service. +- Never hardcode or bake a real credential into `.car`. -Do not fuse dispatch and `.value()` in one comprehension; that silently -serializes fan-out. Do not add an `if __name__ == "__main__":` block: the -workflow is executed with `__name__ == "__main__"` in production. - -### Config - -`entrypoint` and `workflow_file` are relative to `.car/app/` and may not escape -it. `provider` is lowercase `local`, `replicas` is an integer, and -`requirements` is a per-entry list of distribution names -- the source's own -`requirements.txt` is never installed into any image. Omit `policy.yaml` unless -access must be restricted; if present, give it a non-empty `rules` list. -manifest.md carries the complete manifest and how to build each `requirements` -list. - -Part of the manifest is derived from the copy; the rest is the developer's -deployment choice, and no reading of the source produces it. Derive what the -source decides. Do not derive what it does not. - -Use the interaction exposed by `cli/canyonos/config.py` as the configuration UX. -Before writing `.car/config/global_controller.yaml`, present the same two -choices -- **View** and **Change** -- rather than silently choosing deployment -settings: - -1. Build the complete candidate manifest in memory from derived values plus the - defaults in manifest.md's ownership table. -2. **View** prints the whole candidate, not a summary. Clearly annotate defaults - and values constrained by the source, such as `replicas: 1` for stateful - in-memory services. -3. **Change** asks, in one batch, only for developer-owned values: provider and - its EC2 block, replicas that are not constrained, resources, ports, secret - file location, and whether access needs restricting. Show the current/default - value for every choice. Apply the answers and show the resulting manifest. -4. Write the reviewed candidate, then run the validator. - -If the coding environment can launch an interactive command, prefer running -`canyonos config` for this review. Otherwise reproduce its View/Change flow in -the conversation; do not skip the review merely because the CLI has no TTY. -Do not ask the developer for derived values such as `entrypoint` or -`requirements`: walking the copied source gives a more reliable answer. - -Never block an unattended port on this interaction. `canyonos integrate` -launches this skill from a prompt, and a corpus run has no one at the terminal. -If no answer is available, write the displayed defaults, name them in the final -report, and keep going. The one exception is a value with no safe default: -`provider: EC2` needs infrastructure identifiers that are wrong to invent, so -an unanswered EC2 choice leaves the entry `local` and says so. - -## Hard rules - -Capitalized **MUST** and **NEVER** are reserved for port-breaking or -source-integrity rules. The owner column states where each is decided. - -| ID | Rule | Owner | -|---|---|---| -| M1 | Entrypoint MUST define a class named exactly `agent.name` | V006 | -| M2 | The class MUST construct with no arguments | V007 | -| M3 | yaml argument names MUST match Python parameter names | V008 | -| M4 | yaml argument types MUST be bare builtins | V010 | -| M5 | Declared adapter methods MUST be synchronous | V009 | -| M6 | Config names MUST match yaml agent names | `canyonos deploy` build phase | -| M7 | Config names MUST not collide after lowercase normalization | `canyonos deploy` build output | -| M8 | Local provider MUST be lowercase `local` | `canyonos deploy` preflight | -| M9 | `replicas` MUST be an integer | `canyonos deploy` preflight | -| M10 | `requirements` MUST be a list of strings | `canyonos deploy` build phase | -| M11 | Workflow MUST expose `main(query)`; extra parameters MUST default | V016 | -| M12 | Workflow MUST NEVER contain a main guard | V017 | -| M13 | Fan-out MUST dispatch all calls before resolving any | V018 | -| M14 | A module at the root of the copy MUST not take a runtime flat name | V019 | -| M15 | Workflow MUST import each agent from its own `entrypoint` module | V023 | -| M16 | Policy MUST be absent or contain a non-empty `rules` list | `canyonos deploy` preflight | -| M17 | EC2 entries MUST satisfy the EC2 deployment contract | `canyonos deploy` preflight | -| M18 | NEVER copy source prompts, tools, schemas, or model calls | review | -| M19 | NEVER hardcode or bake a real credential into an image | W003 | -| M20 | NEVER write outside `.car`; the application source stays untouched | `git status` | -| M21 | NEVER swap the source LLM provider | review | -| M22 | NEVER silently move, drop, or reclassify source dependencies | review | -| M23 | Framework control flow MUST be rewritten; source node logic MUST be imported | review | -| M24 | A non-resolving source import MUST have usable packaging metadata at the root of the copy when editable install is supported | V031 | -| M25 | Two agents MUST NOT share one `entrypoint` | V020 | -| M26 | `.car` MUST hold `config/` beside `app/`, the source copy | V032 | -| M27 | `.car/app` MUST be rooted at the source's import root | V031 | -| M28 | `requirements` MUST cover every distribution that entry's import graph reaches, transitively | W006, dependency review | -| M29 | The entrypoint MUST NOT be a module another image imports for its real contents | adapter.md review | -| M30 | The entrypoint's package `__init__.py` MUST NOT re-export from it | V033 | -| M31 | Every segment of the `entrypoint` path MUST be a Python identifier | V034 | -| M32 | The entrypoint's own imports MUST be absolute | V035 | +When a source defect or unsupported runtime capability requires breaking one of +these boundaries, report the blocker and obtain approval for that specific +change. Do not broaden approval to unrelated source edits. ## 4. Validate and stop Run static preflight from the application root: ```bash -python /validate.py .car +python3 /validate.py .car ``` -Fix every ERROR and re-run until it reports 0 errors. Warnings and capability +Fix every ERROR and re-run until it exits 0. Warnings and capability limitations are not permission to hide risk: list each one in the handoff and say whether it blocks this source. Confirm that `git status` outside `.car` shows no change to a file the developer owns. diff --git a/.claude/skills/porting-to-canyonos/prepare.py b/.claude/skills/porting-to-canyonos/prepare.py index 9f282b7..963b0c5 100755 --- a/.claude/skills/porting-to-canyonos/prepare.py +++ b/.claude/skills/porting-to-canyonos/prepare.py @@ -8,10 +8,13 @@ """ import argparse +import hashlib +import json +import os import shutil import sys import uuid -from pathlib import Path +from pathlib import Path, PurePosixPath EXCLUDED_DIRECTORIES = frozenset( { @@ -35,6 +38,7 @@ ) EXCLUDED_FILE_SUFFIXES = (".pyc", ".pyo") ENV_TEMPLATES = frozenset({".env.example", ".env.sample", ".env.template"}) +STATE_FILENAME = ".porting-state.json" def _is_relative_to(path: Path, parent: Path) -> bool: @@ -45,6 +49,17 @@ def _is_relative_to(path: Path, parent: Path) -> bool: return True +def _reject_symlinks(import_root: Path) -> None: + """Keep the copied artifact independent of paths outside the copy.""" + for path in import_root.rglob("*"): + if path.is_symlink(): + relative = path.relative_to(import_root) + raise ValueError( + f"import root contains a symbolic link: {relative}; replace it " + "with the intended file or directory before preparing the port" + ) + + def _ignore_factory(artifact_root: Path): def ignore(directory: str, names: list[str]) -> set[str]: directory_path = Path(directory) @@ -64,7 +79,168 @@ def ignore(directory: str, names: list[str]) -> set[str]: return ignore -def prepare(import_root: Path, artifact_root: Path, force: bool = False) -> None: +def _copy_source(import_root: Path, destination: Path, artifact_root: Path) -> None: + shutil.copytree( + import_root, + destination, + ignore=_ignore_factory(artifact_root), + copy_function=shutil.copy2, + symlinks=True, + ) + + +def _hash_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def _file_hashes(root: Path) -> dict[str, str]: + return { + path.relative_to(root).as_posix(): _hash_file(path) + for path in sorted(root.rglob("*")) + if path.is_file() and not path.is_symlink() + } + + +def _load_state(config_dir: Path) -> dict[str, str]: + state_path = config_dir / STATE_FILENAME + if not state_path.is_file(): + raise ValueError( + f"{state_path} is missing; this artifact predates refresh tracking. " + "Use --force only if discarding all edits in .car/app is intentional" + ) + try: + state = json.loads(state_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as error: + raise ValueError(f"cannot read refresh state {state_path}: {error}") from error + files = state.get("source_files") if isinstance(state, dict) else None + if ( + not isinstance(state, dict) + or state.get("version") != 1 + or not isinstance(files, dict) + or not all(_valid_state_entry(path, digest) for path, digest in files.items()) + ): + raise ValueError(f"invalid refresh state: {state_path}") + return files + + +def _valid_state_entry(path: object, digest: object) -> bool: + if not isinstance(path, str) or not isinstance(digest, str): + return False + relative = PurePosixPath(path) + return ( + path == relative.as_posix() + and not relative.is_absolute() + and path not in ("", ".") + and ".." not in relative.parts + and len(digest) == 64 + and all(character in "0123456789abcdef" for character in digest) + ) + + +def _write_state(path: Path, source_hashes: dict[str, str]) -> None: + path.write_text( + json.dumps( + {"version": 1, "source_files": source_hashes}, + indent=2, + sort_keys=True, + ) + + "\n", + encoding="utf-8", + ) + + +def _refresh_app( + app_dir: Path, + source_copy: Path, + staged_app: Path, + previous_source: dict[str, str], +) -> None: + current = _file_hashes(app_dir) + incoming = _file_hashes(source_copy) + conflicts = [] + + for relative in sorted(previous_source.keys() | incoming.keys() | current.keys()): + old = previous_source.get(relative) + new = incoming.get(relative) + edited = current.get(relative) + source_changed = new != old + app_changed = edited != old + if old is None and new is not None and edited not in (None, new): + conflicts.append(relative) + elif old is not None and source_changed and app_changed and edited != new: + conflicts.append(relative) + + # File/directory replacements need the same three-way protection. A new + # source file at `pkg` must not erase a port-only `pkg/adapter.py`, and a + # new source directory must not silently replace a port-authored file at + # `pkg`. + for incoming_path in incoming: + prefix = incoming_path + "/" + for current_path, edited in current.items(): + if not current_path.startswith(prefix): + continue + if edited != previous_source.get(current_path): + conflicts.append(current_path) + for current_path, edited in current.items(): + prefix = current_path + "/" + if any(incoming_path.startswith(prefix) for incoming_path in incoming): + if edited != previous_source.get(current_path): + conflicts.append(current_path) + + if conflicts: + conflicts = sorted(set(conflicts)) + shown = "\n ".join(conflicts[:20]) + suffix = "" if len(conflicts) <= 20 else f"\n ... and {len(conflicts) - 20} more" + raise ValueError( + "refresh found files changed in both the source and .car/app:\n " + f"{shown}{suffix}\nResolve them in .car/app, then update the source " + "or use --force only to discard all port edits" + ) + + shutil.copytree(app_dir, staged_app, copy_function=shutil.copy2, symlinks=True) + + # Apply safe source deletions first so file-to-directory changes have room. + for relative, old in previous_source.items(): + if relative in incoming or current.get(relative) != old: + continue + target = staged_app / relative + if target.is_file(): + target.unlink() + + for directory in sorted( + (path for path in staged_app.rglob("*") if path.is_dir()), + key=lambda path: len(path.parts), + reverse=True, + ): + try: + directory.rmdir() + except OSError: + pass + + for relative, new in incoming.items(): + old = previous_source.get(relative) + edited = current.get(relative) + if new == old or edited == new: + continue + if old is not None and edited != old: + continue + target = staged_app / relative + if target.is_dir(): + shutil.rmtree(target) + target.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(source_copy / relative, target) + + +def prepare( + import_root: Path, + artifact_root: Path, + force: bool = False, + refresh: bool = False, +) -> None: import_root = import_root.expanduser().resolve() artifact_root = artifact_root.expanduser().resolve() app_dir = artifact_root / "app" @@ -78,34 +254,54 @@ def prepare(import_root: Path, artifact_root: Path, force: bool = False) -> None raise ValueError("import root cannot be inside the artifact root") if app_dir.exists() and not app_dir.is_dir(): raise ValueError(f"app path exists but is not a directory: {app_dir}") - if app_dir.exists() and not force: + if force and refresh: + raise ValueError("--force and --refresh are mutually exclusive") + if refresh and not app_dir.is_dir(): + raise ValueError(f"cannot refresh because {app_dir} does not exist") + if app_dir.exists() and not force and not refresh: raise FileExistsError( - f"{app_dir} already exists; use --force to replace only the source copy" + f"{app_dir} already exists; use --refresh to preserve port edits, or " + "--force to discard and replace the entire source copy" ) if config_dir.exists() and not config_dir.is_dir(): raise ValueError(f"config path exists but is not a directory: {config_dir}") + _reject_symlinks(import_root) + if refresh: + _reject_symlinks(app_dir) + artifact_root.mkdir(parents=True, exist_ok=True) - temporary_app = artifact_root / f".app-{uuid.uuid4().hex}.tmp" + config_dir.mkdir(exist_ok=True) + transaction_id = uuid.uuid4().hex + source_copy = artifact_root / f".source-{transaction_id}.tmp" + temporary_app = artifact_root / f".app-{transaction_id}.tmp" previous_app = artifact_root / f".app-{uuid.uuid4().hex}.previous" + temporary_state = config_dir / f".{STATE_FILENAME}-{transaction_id}.tmp" + state_path = config_dir / STATE_FILENAME installed_new_app = False try: - shutil.copytree( - import_root, - temporary_app, - ignore=_ignore_factory(artifact_root), - copy_function=shutil.copy2, - symlinks=True, - ) + _copy_source(import_root, source_copy, artifact_root) + source_hashes = _file_hashes(source_copy) + if refresh: + previous_source = _load_state(config_dir) + _refresh_app(app_dir, source_copy, temporary_app, previous_source) + shutil.rmtree(source_copy) + else: + source_copy.rename(temporary_app) + _write_state(temporary_state, source_hashes) if app_dir.exists(): app_dir.rename(previous_app) temporary_app.rename(app_dir) installed_new_app = True - config_dir.mkdir(exist_ok=True) + os.replace(temporary_state, state_path) except Exception: + if source_copy.exists(): + shutil.rmtree(source_copy) if temporary_app.exists(): shutil.rmtree(temporary_app) + if temporary_state.exists(): + temporary_state.unlink() if installed_new_app and app_dir.exists(): shutil.rmtree(app_dir) if previous_app.exists(): @@ -133,7 +329,12 @@ def parse_args(argv=None): parser.add_argument( "--force", action="store_true", - help="replace an existing app/ copy; leave config/ unchanged", + help="discard edits and replace an existing app/ copy; leave config/ unchanged", + ) + parser.add_argument( + "--refresh", + action="store_true", + help="update unchanged source files while preserving port-authored edits", ) return parser.parse_args(argv) @@ -141,14 +342,20 @@ def parse_args(argv=None): def main(argv=None) -> int: args = parse_args(argv) try: - prepare(Path(args.import_root), Path(args.artifact_root), args.force) + prepare( + Path(args.import_root), + Path(args.artifact_root), + force=args.force, + refresh=args.refresh, + ) except (OSError, ValueError) as error: sys.stderr.write(f"prepare.py: {error}\n") return 1 artifact_root = Path(args.artifact_root) print(f"Created {artifact_root / 'config'}") - print(f"Copied {Path(args.import_root)} to {artifact_root / 'app'}") + action = "Refreshed" if args.refresh else "Copied" + print(f"{action} {Path(args.import_root)} to {artifact_root / 'app'}") return 0 diff --git a/.claude/skills/porting-to-canyonos/references/example-port.md b/.claude/skills/porting-to-canyonos/references/example-port.md index 326160a..4da2d00 100644 --- a/.claude/skills/porting-to-canyonos/references/example-port.md +++ b/.claude/skills/porting-to-canyonos/references/example-port.md @@ -34,7 +34,7 @@ until the model calls `Done`. is `src/`, not the repository root: ```bash -python /prepare.py src .car +python3 /prepare.py src .car ``` This creates `.car/config/` and copies the contents of `src/` into `.car/app/` diff --git a/.claude/skills/porting-to-canyonos/references/manifest.md b/.claude/skills/porting-to-canyonos/references/manifest.md index 8dc7e07..1a6a9df 100644 --- a/.claude/skills/porting-to-canyonos/references/manifest.md +++ b/.claude/skills/porting-to-canyonos/references/manifest.md @@ -1,13 +1,13 @@ # Writing what goes into `.car/config` -Read this before writing the manifest or an agent declaration. The build phase -of `canyonos deploy` owns yaml syntax; nothing here is syntax. These are the -values that can build successfully and then decide whether a container imports -its own dependencies. +Read this before writing the manifest or an agent declaration. The validator +checks YAML structure and the public artifact contract before an approved +`canyonos deploy`; this reference explains how to derive the values inside it. ## Contents - Who decides each key +- Review configuration through the CanyonOS CLI flow - Agent yaml - Requirements - The manifest, in full @@ -19,8 +19,8 @@ and the copy holds it; asking the developer can only make it worse. A **developer** key is a deployment choice the source does not contain, and deriving it means guessing and presenting the guess as a reading. -SKILL.md step 3 shows the whole manifest and then asks about the second column -only, in one round, carrying these defaults. +The configuration review shows the whole manifest and asks about the second +column only, in one round, carrying these defaults. | Key | Decided by | Default when unanswered | |---|---|---| @@ -49,6 +49,29 @@ the config rather than asking about it: deploy preflight or, worse, provisions something unreachable. Unanswered means the entry stays `local`. +## Review configuration through the CanyonOS CLI flow + +Use the interaction implemented by `canyonos config` before writing +`.car/config/global_controller.yaml`: + +1. Build the complete candidate manifest in memory from derived values and the + defaults above. +2. **View** prints the whole candidate, annotating defaults and source-imposed + constraints such as `replicas: 1` for in-memory state. +3. **Change** asks in one batch only for developer-owned values: provider and + EC2 fields, unconstrained replicas, resources, ports, secret-file location, + and access restrictions. Show each current/default value, apply answers, and + show the result. +4. Write the reviewed candidate and run the validator. + +Prefer running `canyonos config` when an interactive terminal is available; +otherwise reproduce View/Change in conversation. Do not ask for derived values +such as entrypoints or requirements. + +An unattended `canyonos integrate` run must not block on this interaction. Use +and report the displayed defaults. Never invent EC2 infrastructure identifiers: +without them, keep the entry `local`. + ## Agent yaml Declarations go in `.car/config/`, beside the manifest. The build reads every diff --git a/.claude/skills/porting-to-canyonos/references/packaging.md b/.claude/skills/porting-to-canyonos/references/packaging.md index 2b7b727..e87c002 100644 --- a/.claude/skills/porting-to-canyonos/references/packaging.md +++ b/.claude/skills/porting-to-canyonos/references/packaging.md @@ -52,7 +52,7 @@ Reach for the metadata below only when one copy root cannot serve every import Run: ```bash -python /validate.py .car +python3 /validate.py .car ``` Read the `editable_install` capability. If it is unavailable and the original diff --git a/.claude/skills/porting-to-canyonos/references/refresh.md b/.claude/skills/porting-to-canyonos/references/refresh.md new file mode 100644 index 0000000..eaab51b --- /dev/null +++ b/.claude/skills/porting-to-canyonos/references/refresh.md @@ -0,0 +1,38 @@ +# Refreshing an existing port + +Read this when `.car/app` already exists and the application source has changed. + +Run the same preparation command with `--refresh` and the same import root: + +```bash +python3 /prepare.py .car --refresh +``` + +The initial copy records source-file hashes in +`.car/config/.porting-state.json`. Refresh compares three states: + +```text +previous source hash → current source + ↘ current .car/app +``` + +- Only the source changed: update `.car/app`. +- Only `.car/app` changed: preserve the port edit. +- The source added a path unused by the port: add it. +- The source deleted an unmodified path: delete it from `.car/app`. +- Both sides changed the same path differently: make no changes and report all + conflicts. + +Resolve a conflict in `.car/app`, then either make the source match that result +or intentionally start over. The script does not guess a merge because an +adapter and its source often change for different reasons while sharing one +module. + +`--force` is not refresh. It discards the entire `.car/app` tree and replaces it +with a clean source copy while retaining `.car/config`. Use it only when every +adapter and workflow edit in `.car/app` is intentionally disposable. + +After refresh, survey changed imports, dependencies, runtime assets, and service +boundaries again, then run the full validator. A clean file merge is not proof +that the deployment contract still holds. + diff --git a/.claude/skills/porting-to-canyonos/references/source-survey.md b/.claude/skills/porting-to-canyonos/references/source-survey.md new file mode 100644 index 0000000..b44538a --- /dev/null +++ b/.claude/skills/porting-to-canyonos/references/source-survey.md @@ -0,0 +1,29 @@ +# Surveying the copied source + +Read this after `prepare.py` and before choosing service boundaries. Survey +`.car/app`, not a guessed abstraction of the original repository. + +Identify all of the following: + +1. The production entry point and callable input/output. If several + implementations look plausible, trace imports from the documented route, + CLI, or launch path instead of choosing by filename. +2. Framework-owned control flow: graphs, crews, chats, routing, fan-out, + commands, and interrupts. +3. Runtime-injected stores, context, memory, sessions, and callback managers. +4. Sync/async boundaries and objects tied to an event loop. +5. The transitive import graph and the source's pinned runtime distributions. +6. Model provider, credential names, streaming, and optional `llm_proxy` use. +7. Independent work that benefits from separate resource or replica profiles. +8. Whether imports resolve with `.car/app` as `/app`; read `packaging.md` when + they do not. +9. Non-Python runtime files such as prompts, CrewAI YAML, PDFs, templates, + schemas, and corpora. If the runtime cannot sweep all files, report this as a + blocker; do not embed files or rewrite paths to hide it. +10. Whether every Python file on the selected import graph parses. Existing + syntax errors are source defects; report them and obtain approval before + changing even the copied version. + +Run `python3 /validate.py .car` after the survey and after every +change. Missing or malformed required inputs fail closed. If a required runtime +capability is reported unavailable, stop instead of assuming it exists. diff --git a/.claude/skills/porting-to-canyonos/validate.py b/.claude/skills/porting-to-canyonos/validate.py index a8f38c9..0eb9e93 100755 --- a/.claude/skills/porting-to-canyonos/validate.py +++ b/.claude/skills/porting-to-canyonos/validate.py @@ -1,14 +1,13 @@ #!/usr/bin/env python3 -"""Preflight the runtime traps that `ventis build` cannot see. +"""Preflight a CanyonOS port before an approved deployment. -This deliberately does not duplicate build-time validation such as malformed -YAML, missing entrypoints, or config-to-yaml matching. `ventis build` owns those -checks. This script parses Python without importing it and catches failures that -otherwise stay hidden until a container loads an agent, starts a workflow, or -serves its first request. A replica is not evidence: the controller writes -`healthy` to Redis before `_load_agent` runs. +This checks the public `.car` artifact contract first, then parses Python without +importing it to catch failures that would otherwise stay hidden until a +container loads an agent, starts a workflow, or serves its first request. It +fails closed when the required inputs cannot be checked. A replica is not +evidence: the controller writes `healthy` to Redis before `_load_agent` runs. - python validate.py [artifact_root] [-c config/global_controller.yaml] + python3 validate.py [artifact_root] [-c config/global_controller.yaml] [--json] [--strict] `artifact_root` is the `.car` directory: `config/` beside `app/`, the copy of @@ -28,130 +27,43 @@ import os import re import sys -from typing import ClassVar - -try: - import yaml -except ImportError: # pragma: no cover - pyyaml is a CanyonOS Core dependency - sys.stderr.write("validate.py needs pyyaml: pip install pyyaml\n") - raise SystemExit(2) from None +SKILL_DIR = os.path.dirname(os.path.abspath(__file__)) +if SKILL_DIR not in sys.path: + sys.path.insert(0, SKILL_DIR) + +from validation.core import ERROR, INFO, WARN, Report, line_of, load_yaml # noqa: E402 +from validation.manifest import ( # noqa: E402 + check_declaration_bindings, + check_manifest_structure, + check_policy, + check_self_contained_tree, + discover_agent_declarations, +) +from validation.python_source import ( # noqa: E402 + class_methods, + find_class, + parameter_names, + parse_python, + reachable_imports, + required_parameters, + toplevel_import_names, +) +from validation.runtime import ( # noqa: E402 + BASE_AGENT_REQUIREMENTS, + BASE_WORKFLOW_REQUIREMENTS, + CAPABILITY_SOURCE, + IMPORT_TO_DISTRIBUTION, + NAMESPACE_DISTRIBUTIONS, + RUNTIME_FLAT_NAMES, + STDLIB_MODULE_NAMES, + probe_capabilities, +) DEFAULT_CONFIG_PATH = "config/global_controller.yaml" # ventis/cli.py SOURCE_DIR_NAME -- the duplicated application source. SOURCE_DIR_NAME = "app" -# Copied flat into every image over the swept project tree, so a project module -# landing flat under one of these names is overwritten. -# ventis/stub_generator.py generate_docker / generate_workflow_docker. -RUNTIME_FLAT_NAMES = frozenset( - { - "future.py", - "ventis_context.py", - "local_controller.py", - "local_controller_frontend.py", - "redis_client.py", - "grpc_options.py", - "gpu_metrics.py", - "bedrock.py", - "deploy.py", - "session_logging.py", - "workflow_launcher.py", - } -) - -def _base_requirements(): - """What the generator preinstalls, taken from the importable runtime. - - The literals below are a fallback for a machine where `ventis` is not - importable. They are also the only copy of a runtime fact in this file that - nothing checks at run time, and a copied fact rots: the `sweeps_all_files` - probe named a function that never existed and reported `no` for an entire - 45-repository corpus before a porter caught it. Prefer the live values, and - let tests/test_porting_skill_validate.py hold the fallback to them. - """ - agent = [ - "grpcio", - "grpcio-tools", - "redis", - "pyyaml", - "psutil", - "ipdb", - "ipython", - "boto3", - ] - workflow = [*agent, "flask", "sqlalchemy", "psycopg[binary]"] - try: - from ventis import stub_generator - except Exception: # noqa: BLE001 - a broken install must not crash the check - return agent, workflow - return ( - list(getattr(stub_generator, "BASE_AGENT_REQUIREMENTS", agent)), - list(getattr(stub_generator, "BASE_WORKFLOW_REQUIREMENTS", workflow)), - ) - - -BASE_AGENT_REQUIREMENTS, BASE_WORKFLOW_REQUIREMENTS = _base_requirements() -# Import name -> every distribution that provides it. A tuple rather than a -# string because more than one distribution can ship the same import name, and -# reporting a correct declaration as a gap teaches the reader to dismiss W006. -IMPORT_TO_DISTRIBUTION = { - "attr": ("attrs",), - # `import autogen` is shipped by three unrelated distributions: pyautogen - # (Microsoft's original), ag2 (the community continuation), and a package - # literally named autogen. Any of them satisfies the import. - "autogen": ("pyautogen", "ag2", "autogen", "autogen-agentchat"), - "bs4": ("beautifulsoup4",), - "cv2": ("opencv-python",), - "dateutil": ("python-dateutil",), - "dotenv": ("python-dotenv",), - "grpc": ("grpcio",), - "grpc_tools": ("grpcio-tools",), - "jwt": ("pyjwt",), - "PIL": ("pillow",), - "psycopg": ("psycopg",), - "psycopg2": ("psycopg2-binary",), - "pydantic_settings": ("pydantic-settings",), - "sklearn": ("scikit-learn",), - "typing_extensions": ("typing-extensions",), - "yaml": ("pyyaml",), -} -# Import name -> distribution prefix, where the top-level package is shipped by -# a family of distributions rather than one. `import llama_index.llms.openai` -# collapses to `llama_index`, which no correctly scoped requirements list ever -# names: it declares llama-index-core, llama-index-llms-openai and so on. Any -# member of the family satisfies the import. -NAMESPACE_DISTRIBUTIONS = { - "llama_index": "llama-index", -} - -def _stdlib_names(): - """Module names the interpreter provides without any distribution. - - `sys.stdlib_module_names` exists only from 3.10. Below that, derive the set - from the interpreter's own library directory rather than shipping a list - that rots -- without it every `import os` in the copy becomes a W006, and a - check that cries wolf is a check the reader stops reading. - """ - names = getattr(sys, "stdlib_module_names", None) - if names: - return frozenset(names) - found = set(sys.builtin_module_names) - library = os.path.dirname(os.__file__) - try: - entries = os.listdir(library) - except OSError: - return frozenset(found) - for entry in entries: - if entry.endswith(".py"): - found.add(entry[:-3]) - elif "." not in entry and "-" not in entry: - found.add(entry) - return frozenset(found) - - -STDLIB_MODULE_NAMES = _stdlib_names() - SECRET_PATTERNS = [ (re.compile(r"sk-[A-Za-z0-9_-]{20,}"), "an OpenAI-style secret key"), (re.compile(r"AKIA[0-9A-Z]{16}"), "an AWS access key id"), @@ -160,319 +72,6 @@ def _stdlib_names(): ] SECRET_NAME = re.compile(r"(API_KEY|SECRET|TOKEN|PASSWORD|CREDENTIAL)", re.IGNORECASE) -ERROR = "ERROR" -WARN = "WARN" -INFO = "INFO" - - -# ------------------------------------------------------------------ # -# Capabilities # -# ------------------------------------------------------------------ # -# -# Stable labels for behavior detected from the importable runtime. They contain -# no external development metadata. - -CAPABILITY_SOURCE = { - "env_file": "runtime env-file injection", - "editable_install": "editable project installation", - "sweeps_all_files": "full project-file sweep", -} - - -def probe_capabilities(): - """Ask the importable ventis package what it actually supports.""" - caps = dict.fromkeys(CAPABILITY_SOURCE, False) - caps["ventis"] = False - try: - from ventis import stub_generator - except Exception: # noqa: BLE001 - a broken install must not crash the check - return caps - - caps["ventis"] = True - caps["editable_install"] = hasattr(stub_generator, "_install_step") - caps["sweeps_all_files"] = hasattr(stub_generator, "_sweep_project_files") - - import importlib - - for module_name in ("ventis.controller.utils.env_file", "ventis.utils.env_file"): - try: - module = importlib.import_module(module_name) - except Exception: # noqa: BLE001,S112 - the other path is the live one - continue - if hasattr(module, "resolve_env_file"): - caps["env_file"] = True - break - return caps - - -# ------------------------------------------------------------------ # -# YAML with line numbers # -# ------------------------------------------------------------------ # - - -class LineDict(dict): - """A mapping that remembers where it and each of its keys were written.""" - - line = 0 - key_lines: ClassVar[dict] = {} - - -class LineLoader(yaml.SafeLoader): - pass - - -def _construct_mapping(loader, node): - data = LineDict() - yield data - data.update(loader.construct_mapping(node, deep=False)) - data.line = node.start_mark.line + 1 - data.key_lines = { - key.value: key.start_mark.line + 1 - for key, _ in node.value - if isinstance(key, yaml.ScalarNode) - } - - -LineLoader.add_constructor( - yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, _construct_mapping -) - - -def line_of(mapping, key=None): - """The source line of `key` inside `mapping`, or of the mapping itself.""" - if not isinstance(mapping, LineDict): - return 0 - if key is not None: - return mapping.key_lines.get(key, mapping.line) - return mapping.line - - -def load_yaml(path): - """Parse `path`, returning (data, error). Never raises.""" - try: - with open(path, "r", encoding="utf-8") as handle: - return yaml.load(handle, Loader=LineLoader), None - except Exception as exc: # noqa: BLE001 - any parse failure is a finding - return None, str(exc) - - -# ------------------------------------------------------------------ # -# Findings # -# ------------------------------------------------------------------ # - - -class Report: - def __init__(self, project_dir, capabilities): - self.project_dir = project_dir - self.capabilities = capabilities - self.findings = [] - - def add(self, check, level, path, line, summary, mechanism): - self.findings.append( - { - "check": check, - "level": level, - "path": self.rel(path) if path else "", - "line": line or 0, - "summary": summary, - "mechanism": mechanism, - } - ) - - def error(self, check, path, line, summary, mechanism): - self.add(check, ERROR, path, line, summary, mechanism) - - def warn(self, check, path, line, summary, mechanism): - self.add(check, WARN, path, line, summary, mechanism) - - def unavailable(self, check, summary): - self.add(check, INFO, "", 0, summary, "") - - def rel(self, path): - try: - return os.path.relpath(path, self.project_dir) - except ValueError: - return path - - def counts(self): - errors = sum(1 for f in self.findings if f["level"] == ERROR) - warnings = sum(1 for f in self.findings if f["level"] == WARN) - return errors, warnings - - -# ------------------------------------------------------------------ # -# Python source helpers # -# ------------------------------------------------------------------ # - - -def parse_python(path): - """AST for `path`, or (None, error). The port is never imported.""" - try: - with open(path, "r", encoding="utf-8") as handle: - source = handle.read() - except OSError as exc: - return None, str(exc) - try: - return ast.parse(source, filename=path), None - except SyntaxError as exc: - return None, f"{exc.msg} (line {exc.lineno})" - - -def find_class(tree, name): - for node in tree.body: - if isinstance(node, ast.ClassDef) and node.name == name: - return node - return None - - -def class_methods(class_node): - return { - node.name: node - for node in class_node.body - if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) - } - - -def parameter_names(func_node): - """Every parameter a caller can pass by keyword, minus self.""" - args = func_node.args - positional = [a.arg for a in args.posonlyargs + args.args] - if positional and positional[0] in ("self", "cls"): - positional = positional[1:] - return positional + [a.arg for a in args.kwonlyargs] - - -def required_parameters(func_node): - """Parameters with no default, minus self.""" - args = func_node.args - positional = [a.arg for a in args.posonlyargs + args.args] - if positional and positional[0] in ("self", "cls"): - positional = positional[1:] - if args.defaults: - positional = positional[: len(positional) - len(args.defaults)] - kwonly = [ - arg.arg - for arg, default in zip(args.kwonlyargs, args.kw_defaults) - if default is None - ] - return positional + kwonly - - -def toplevel_import_names(tree): - """Top-level package name of every import in the module, with line numbers.""" - names = {} - for node in ast.walk(tree): - if isinstance(node, ast.Import): - for alias in node.names: - names.setdefault(alias.name.split(".")[0], node.lineno) - elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module: - names.setdefault(node.module.split(".")[0], node.lineno) - return names - - -def dotted_import_names(tree): - """Every absolute import in the module as its full dotted path, with lines.""" - names = {} - for node in ast.walk(tree): - if isinstance(node, ast.Import): - for alias in node.names: - names.setdefault(alias.name, node.lineno) - elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module: - names.setdefault(node.module, node.lineno) - return names - - -def _local_module_files(project_dir, dotted): - """Files inside the copy that `import ` executes, outermost first. - - Python runs every package `__init__.py` on the way down before the leaf - module. That is how an image ends up executing code it never names: the - workflow imports `pkg.agent`, `pkg/__init__.py` runs first, and whatever it - imports runs with it. - """ - parts = dotted.split(".") - found = [] - prefix = project_dir - for depth, part in enumerate(parts): - if os.path.isdir(os.path.join(prefix, part)): - init_path = os.path.join(prefix, part, "__init__.py") - if os.path.isfile(init_path): - found.append(init_path) - prefix = os.path.join(prefix, part) - continue - leaf = os.path.join(prefix, part + ".py") - if depth == len(parts) - 1 and os.path.isfile(leaf): - found.append(leaf) - return found - return found - - -def _relative_import_files(project_dir, path, tree): - """Files a module's own `from .sibling import x` imports execute.""" - root = os.path.realpath(project_dir) - found = [] - for node in ast.walk(tree): - if not isinstance(node, ast.ImportFrom) or not node.level: - continue - base = os.path.dirname(path) - for _ in range(node.level - 1): - base = os.path.dirname(base) - resolved = os.path.realpath(base) - if resolved != root and not resolved.startswith(root + os.sep): - continue - target = os.path.join(base, *(node.module.split(".") if node.module else [])) - candidates = [target + ".py", os.path.join(target, "__init__.py")] - candidates += [ - os.path.join(target, alias.name + ".py") for alias in node.names - ] - candidates += [ - os.path.join(target, alias.name, "__init__.py") for alias in node.names - ] - found += [c for c in candidates if os.path.isfile(c)] - return found - - -def reachable_imports(project_dir, root_path, shadowed_paths=()): - """Every third-party import the built image executes, transitively. - - An image runs far more than the file the config names. `tools/parser.py` is - one hop from an entrypoint and its pdfplumber import is invisible to an AST - walk of the entrypoint alone; a package `__init__.py` three hops up drags in - graphql-core. Both surface only as a ModuleNotFoundError deep inside an - import chain at agent load, long after a green build. - - `shadowed_paths` are real source modules replaced in this image before it - runs. In particular, a workflow image receives generated stubs over every - agent entrypoint, so following the overwritten implementations would report - dependencies that image never imports. - - Returns {dotted import name: (file that imports it, line)} for names that do - not resolve inside the copy. - """ - external = {} - seen = set() - shadowed = {os.path.realpath(path) for path in shadowed_paths} - queue = [os.path.realpath(root_path)] - while queue: - path = queue.pop() - if path in shadowed or path in seen or not os.path.isfile(path): - continue - seen.add(path) - tree, _ = parse_python(path) - if tree is None: - continue - for dotted, lineno in dotted_import_names(tree).items(): - local = _local_module_files(project_dir, dotted) - if local: - queue += [os.path.realpath(p) for p in local] - else: - external.setdefault(dotted, (path, lineno)) - queue += [ - os.path.realpath(p) - for p in _relative_import_files(project_dir, path, tree) - ] - return external - # ------------------------------------------------------------------ # # V006-V010 adapter failures hidden by _load_agent # @@ -654,7 +253,7 @@ def check_stub_imports(report, workflow_path, tree, stub_modules): flat, through a package re-export, or from a second copy of the module -- resolves to the real class instead, and the workflow runs the agent in-process with none of the deployment behind it. The class name is another - trap: `ventis build` prints one with a `Stub` suffix that it never writes. + trap: the deploy build prints one with a `Stub` suffix that it never writes. """ for node in ast.walk(tree): if not isinstance(node, ast.ImportFrom) or not node.module: @@ -1135,7 +734,7 @@ def check_secrets(report, port_paths): for number, line in enumerate(lines, start=1): for pattern, description in SECRET_PATTERNS: if pattern.search(line): - report.warn( + report.error( "W003", path, number, @@ -1279,7 +878,7 @@ def check_requirements_coverage( f"it runs {report.rel(where)} on the way there, and that module " "needs it." ) - report.warn( + report.error( "W006", where, lineno, @@ -1308,44 +907,24 @@ def _normalize_distribution(name): # ------------------------------------------------------------------ # -def find_agent_declarations(config_dir): - """Map agent name -> declaration, for every declaration in `config/`. - - Mirrors ventis/cli.py: declarations sit in `config/` beside the manifest, - which -- like `policy.yaml` -- carries no top-level `agent.name` and so - drops out here. - """ - import glob - - declarations = {} - for path in sorted(glob.glob(os.path.join(config_dir, "*.yaml"))): - data, error = load_yaml(path) - if error is not None or not isinstance(data, dict): - continue - agent = data.get("agent") - name = agent.get("name") if isinstance(agent, dict) else None - if isinstance(name, str) and name: - declarations[name] = (path, agent) - return declarations - - def module_path(entrypoint): """Dotted module name an entrypoint has inside the container.""" return os.path.splitext(entrypoint)[0].replace("\\", "/").replace("/", ".") def validate(artifact_dir, config_path, capabilities): - """Inspect only failures hidden behind a successful image build.""" + """Check the public artifact contract and deeper runtime failure modes.""" report = Report(artifact_dir, capabilities) - # The build owns config/YAML syntax and shape validation. We read only enough - # valid structure to locate code for the deeper checks below. config, error = load_yaml(config_path) if error is not None or not isinstance(config, dict): - report.unavailable( - "BUILD", - "runtime preflight skipped because the config cannot be read; " - "ventis build owns and reports this error.", + report.error( + "V001", + config_path, + 0, + f"the global manifest cannot be read: {error or 'expected a YAML mapping'}", + "Validation finishes before an approved `canyonos deploy`, so an " + "unreadable manifest cannot be deferred to deploy.", ) return report @@ -1363,17 +942,17 @@ def validate(artifact_dir, config_path, capabilities): ) return report - agents_by_name = find_agent_declarations(os.path.dirname(config_path)) + check_self_contained_tree(report, artifact_dir) - entries = config.get("agents") - if not isinstance(entries, list): - report.unavailable( - "BUILD", - "runtime preflight skipped because `agents:` is not a list; " - "ventis build owns and reports this error.", - ) + config_dir = os.path.dirname(config_path) + entries = check_manifest_structure(report, config, config_path, source_dir) + if entries is None: return report + agents_by_name = discover_agent_declarations(report, config_dir, config_path) + check_declaration_bindings(report, entries, agents_by_name, config_path) + check_policy(report, config_dir) + entrypoints = [] for entry in entries: if not isinstance(entry, dict) or entry.get("type", "agent") == "workflow": diff --git a/.claude/skills/porting-to-canyonos/validation/__init__.py b/.claude/skills/porting-to-canyonos/validation/__init__.py new file mode 100644 index 0000000..f6152f8 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/__init__.py @@ -0,0 +1,2 @@ +"""Composable validation checks for the CanyonOS porting skill.""" + diff --git a/.claude/skills/porting-to-canyonos/validation/core.py b/.claude/skills/porting-to-canyonos/validation/core.py new file mode 100644 index 0000000..684e5cc --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/core.py @@ -0,0 +1,99 @@ +"""Shared result and YAML primitives for validation checks.""" + +import os +from typing import ClassVar + +try: + import yaml +except ImportError: # pragma: no cover - pyyaml is a CanyonOS dependency + raise RuntimeError("validate.py needs pyyaml: pip install pyyaml") from None + + +ERROR = "ERROR" +WARN = "WARN" +INFO = "INFO" + + +class LineDict(dict): + """A mapping that remembers where it and each of its keys were written.""" + + line = 0 + key_lines: ClassVar[dict] = {} + + +class LineLoader(yaml.SafeLoader): + pass + + +def _construct_mapping(loader, node): + data = LineDict() + yield data + data.update(loader.construct_mapping(node, deep=False)) + data.line = node.start_mark.line + 1 + data.key_lines = { + key.value: key.start_mark.line + 1 + for key, _ in node.value + if isinstance(key, yaml.ScalarNode) + } + + +LineLoader.add_constructor( + yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, _construct_mapping +) + + +def line_of(mapping, key=None): + if not isinstance(mapping, LineDict): + return 0 + if key is not None: + return mapping.key_lines.get(key, mapping.line) + return mapping.line + + +def load_yaml(path): + """Parse path and return ``(data, error)`` without raising.""" + try: + with open(path, "r", encoding="utf-8") as handle: + return yaml.load(handle, Loader=LineLoader), None + except Exception as exc: # noqa: BLE001 - every parse failure is a finding + return None, str(exc) + + +class Report: + def __init__(self, project_dir, capabilities): + self.project_dir = project_dir + self.capabilities = capabilities + self.findings = [] + + def add(self, check, level, path, line, summary, mechanism): + self.findings.append( + { + "check": check, + "level": level, + "path": self.rel(path) if path else "", + "line": line or 0, + "summary": summary, + "mechanism": mechanism, + } + ) + + def error(self, check, path, line, summary, mechanism): + self.add(check, ERROR, path, line, summary, mechanism) + + def warn(self, check, path, line, summary, mechanism): + self.add(check, WARN, path, line, summary, mechanism) + + def unavailable(self, check, summary): + self.add(check, INFO, "", 0, summary, "") + + def rel(self, path): + try: + return os.path.relpath(path, self.project_dir) + except ValueError: + return path + + def counts(self): + errors = sum(1 for finding in self.findings if finding["level"] == ERROR) + warnings = sum(1 for finding in self.findings if finding["level"] == WARN) + return errors, warnings + diff --git a/.claude/skills/porting-to-canyonos/validation/manifest.py b/.claude/skills/porting-to-canyonos/validation/manifest.py new file mode 100644 index 0000000..df40c23 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/manifest.py @@ -0,0 +1,295 @@ +"""Fail-closed checks for the public CanyonOS artifact contract.""" + +import glob +import os + +from validation.core import line_of, load_yaml + +def _safe_relative_python_path(value): + if not isinstance(value, str) or not value.strip(): + return False + normalized = value.replace("\\", "/") + return ( + not normalized.startswith("/") + and ".." not in normalized.split("/") + and normalized.endswith(".py") + ) + + +def check_manifest_structure(report, config, config_path, source_dir): + """Return entries only when deeper checks can traverse them safely.""" + entries = config.get("agents") + if not isinstance(entries, list): + report.error( + "V001", + config_path, + line_of(config, "agents"), + "`agents:` must be a list", + "The CanyonOS manifest cannot be traversed or built without an agents list.", + ) + return None + + valid = True + agent_count = sum( + 1 + for entry in entries + if isinstance(entry, dict) and entry.get("type", "agent") == "agent" + ) + workflow_count = sum( + 1 + for entry in entries + if isinstance(entry, dict) and entry.get("type", "agent") == "workflow" + ) + if agent_count < 1: + report.error( + "V002", + config_path, + line_of(config, "agents"), + "the manifest must contain at least one agent service", + "A CanyonOS port needs a callable service behind its workflow.", + ) + valid = False + if workflow_count != 1: + report.error( + "V002", + config_path, + line_of(config, "agents"), + f"the manifest must contain exactly one workflow service; found {workflow_count}", + "The deployment exposes one `/main` workflow and builds one workflow image.", + ) + valid = False + for index, entry in enumerate(entries): + if not isinstance(entry, dict): + report.error( + "V002", + config_path, + 0, + f"agents[{index}] must be a mapping", + "CanyonOS reads each agents item as a service declaration.", + ) + valid = False + continue + + name = entry.get("name") + if not isinstance(name, str) or not name.strip(): + report.error( + "V002", + config_path, + line_of(entry, "name"), + f"agents[{index}] has no non-empty string `name`", + "Names bind manifest entries, declarations, generated stubs, and images.", + ) + valid = False + else: + earlier = [ + item.get("name") + for item in entries[:index] + if isinstance(item, dict) and isinstance(item.get("name"), str) + ] + collision = next( + (other for other in earlier if other.lower() == name.lower()), None + ) + if collision is not None: + report.error( + "V002", + config_path, + line_of(entry, "name"), + f"`{name}` collides with `{collision}` after lowercase normalization", + "CanyonOS uses lowercase image and target names, so one " + "service overwrites the other.", + ) + valid = False + + service_type = entry.get("type", "agent") + if service_type not in ("agent", "workflow"): + report.error( + "V002", + config_path, + line_of(entry, "type"), + f"`type: {service_type}` is neither `agent` nor `workflow`", + "Only those two service shapes have a CanyonOS build contract.", + ) + valid = False + + provider = entry.get("provider", "local") + if provider not in ("local", "EC2"): + report.error( + "V002", + config_path, + line_of(entry, "provider"), + f"unsupported provider spelling `{provider}`", + "Use lowercase `local` or uppercase `EC2`; runtime provider " + "handling is case-sensitive.", + ) + valid = False + + replicas = entry.get("replicas", 1) + if isinstance(replicas, bool) or not isinstance(replicas, int) or replicas < 1: + report.error( + "V002", + config_path, + line_of(entry, "replicas"), + "`replicas` must be an integer greater than zero", + "CanyonOS creates one placement per replica and cannot deploy " + "an empty or fractional set.", + ) + valid = False + + requirements = entry.get("requirements", []) + if not isinstance(requirements, list) or not all( + isinstance(item, str) and item.strip() for item in requirements + ): + report.error( + "V002", + config_path, + line_of(entry, "requirements"), + "`requirements` must be a list of non-empty strings", + "CanyonOS writes this list into the image requirements file.", + ) + valid = False + + path_key = "workflow_file" if service_type == "workflow" else "entrypoint" + relative = entry.get(path_key) + if not _safe_relative_python_path(relative): + report.error( + "V002", + config_path, + line_of(entry, path_key), + f"`{path_key}` must be a relative .py path contained by `.car/app`", + "Absolute and parent-relative paths escape the self-contained CanyonOS artifact.", + ) + valid = False + elif not os.path.isfile(os.path.join(source_dir, relative)): + report.error( + "V002", + config_path, + line_of(entry, path_key), + f"`{path_key}: {relative}` does not exist in `.car/app`", + "The deploy build cannot create this service without its Python entry file.", + ) + valid = False + + return entries if valid else None + + +def check_self_contained_tree(report, artifact_dir): + """Reject symlinks because `.car` must not depend on outside state.""" + for root, directories, files in os.walk(artifact_dir, followlinks=False): + for name in [*directories, *files]: + path = os.path.join(root, name) + if not os.path.islink(path): + continue + report.error( + "V036", + path, + 0, + "`.car` contains a symbolic link", + "A symlink can escape the artifact or be skipped by the " + "Python-file sweep. Copy the intended file or directory into " + "the artifact explicitly.", + ) + + +def discover_agent_declarations(report, config_dir, config_path): + """Load declarations without silently discarding malformed or duplicate YAML.""" + declarations = {} + for path in sorted(glob.glob(os.path.join(config_dir, "*.yaml"))): + data, error = load_yaml(path) + if error is not None: + report.error( + "V003", + path, + 0, + f"YAML cannot be parsed: {error}", + "CanyonOS reads every YAML file in the config directory during deploy.", + ) + continue + if not isinstance(data, dict): + if os.path.realpath(path) == os.path.realpath(config_path): + report.error( + "V003", + path, + 0, + "the global manifest must be a mapping", + "A scalar or empty manifest has no CanyonOS configuration contract.", + ) + continue + agent = data.get("agent") + if agent is None: + continue + if ( + not isinstance(agent, dict) + or not isinstance(agent.get("name"), str) + or not agent["name"] + ): + report.error( + "V003", + path, + line_of(data, "agent"), + "`agent` must contain a non-empty string `name`", + "The declaration name is the binding used to generate its stub.", + ) + continue + name = agent["name"] + if name in declarations: + report.error( + "V003", + path, + line_of(agent, "name"), + f"duplicate declaration for `{name}`", + "Filename ordering would otherwise choose one declaration silently.", + ) + continue + declarations[name] = (path, agent) + return declarations + + +def check_declaration_bindings(report, entries, declarations, config_path): + """Require a one-to-one binding for every agent service.""" + configured = { + entry["name"] + for entry in entries + if entry.get("type", "agent") != "workflow" + } + for name in sorted(configured - declarations.keys()): + report.error( + "V004", + config_path, + 0, + f"agent `{name}` has no matching declaration in `.car/config`", + f"Without `agent.name: {name}`, CanyonOS cannot generate the service stub.", + ) + for name in sorted(declarations.keys() - configured): + path, _ = declarations[name] + report.warn( + "V004", + path, + 0, + f"declaration `{name}` has no agent service in the manifest", + "It is stale or unused and will not produce a deployable service.", + ) + + +def check_policy(report, config_dir): + path = os.path.join(config_dir, "policy.yaml") + if not os.path.exists(path): + return + data, error = load_yaml(path) + if error is not None or not isinstance(data, dict): + report.error( + "V005", + path, + 0, + "policy.yaml must be a YAML mapping", + error or "CanyonOS reads `rules` from this mapping during deploy.", + ) + return + rules = data.get("rules") + if not isinstance(rules, list) or not rules: + report.error( + "V005", + path, + line_of(data, "rules"), + "policy.yaml must contain a non-empty `rules` list", + "Remove the file for unrestricted access; an empty policy is not a valid restriction.", + ) diff --git a/.claude/skills/porting-to-canyonos/validation/python_source.py b/.claude/skills/porting-to-canyonos/validation/python_source.py new file mode 100644 index 0000000..7c3eefe --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/python_source.py @@ -0,0 +1,152 @@ +"""Static Python-source discovery used by adapter and packaging checks.""" + +import ast +import os + + +def parse_python(path): + """Return ``(AST, None)`` or ``(None, error)`` without importing the file.""" + try: + with open(path, "r", encoding="utf-8") as handle: + source = handle.read() + except OSError as exc: + return None, str(exc) + try: + return ast.parse(source, filename=path), None + except SyntaxError as exc: + return None, f"{exc.msg} (line {exc.lineno})" + + +def find_class(tree, name): + for node in tree.body: + if isinstance(node, ast.ClassDef) and node.name == name: + return node + return None + + +def class_methods(class_node): + return { + node.name: node + for node in class_node.body + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) + } + + +def parameter_names(func_node): + """Return every keyword-callable parameter, excluding ``self``/``cls``.""" + args = func_node.args + positional = [arg.arg for arg in args.posonlyargs + args.args] + if positional and positional[0] in ("self", "cls"): + positional = positional[1:] + return positional + [arg.arg for arg in args.kwonlyargs] + + +def required_parameters(func_node): + """Return parameters without defaults, excluding ``self``/``cls``.""" + args = func_node.args + positional = [arg.arg for arg in args.posonlyargs + args.args] + if positional and positional[0] in ("self", "cls"): + positional = positional[1:] + if args.defaults: + positional = positional[: len(positional) - len(args.defaults)] + kwonly = [ + arg.arg + for arg, default in zip(args.kwonlyargs, args.kw_defaults) + if default is None + ] + return positional + kwonly + + +def toplevel_import_names(tree): + """Return top-level import names and their first line numbers.""" + names = {} + for node in ast.walk(tree): + if isinstance(node, ast.Import): + for alias in node.names: + names.setdefault(alias.name.split(".")[0], node.lineno) + elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module: + names.setdefault(node.module.split(".")[0], node.lineno) + return names + + +def dotted_import_names(tree): + """Return absolute dotted imports and their first line numbers.""" + names = {} + for node in ast.walk(tree): + if isinstance(node, ast.Import): + for alias in node.names: + names.setdefault(alias.name, node.lineno) + elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module: + names.setdefault(node.module, node.lineno) + return names + + +def _local_module_files(project_dir, dotted): + """Return local files executed by importing ``dotted``, outermost first.""" + parts = dotted.split(".") + found = [] + prefix = project_dir + for depth, part in enumerate(parts): + if os.path.isdir(os.path.join(prefix, part)): + init_path = os.path.join(prefix, part, "__init__.py") + if os.path.isfile(init_path): + found.append(init_path) + prefix = os.path.join(prefix, part) + continue + leaf = os.path.join(prefix, part + ".py") + if depth == len(parts) - 1 and os.path.isfile(leaf): + found.append(leaf) + return found + return found + + +def _relative_import_files(project_dir, path, tree): + """Return local files executed by a module's relative imports.""" + root = os.path.realpath(project_dir) + found = [] + for node in ast.walk(tree): + if not isinstance(node, ast.ImportFrom) or not node.level: + continue + base = os.path.dirname(path) + for _ in range(node.level - 1): + base = os.path.dirname(base) + resolved = os.path.realpath(base) + if resolved != root and not resolved.startswith(root + os.sep): + continue + target = os.path.join(base, *(node.module.split(".") if node.module else [])) + candidates = [target + ".py", os.path.join(target, "__init__.py")] + candidates += [ + os.path.join(target, alias.name + ".py") for alias in node.names + ] + candidates += [ + os.path.join(target, alias.name, "__init__.py") for alias in node.names + ] + found += [candidate for candidate in candidates if os.path.isfile(candidate)] + return found + + +def reachable_imports(project_dir, root_path, shadowed_paths=()): + """Return third-party imports reachable from ``root_path`` transitively.""" + external = {} + seen = set() + shadowed = {os.path.realpath(path) for path in shadowed_paths} + queue = [os.path.realpath(root_path)] + while queue: + path = queue.pop() + if path in shadowed or path in seen or not os.path.isfile(path): + continue + seen.add(path) + tree, _ = parse_python(path) + if tree is None: + continue + for dotted, lineno in dotted_import_names(tree).items(): + local = _local_module_files(project_dir, dotted) + if local: + queue += [os.path.realpath(item) for item in local] + else: + external.setdefault(dotted, (path, lineno)) + queue += [ + os.path.realpath(item) + for item in _relative_import_files(project_dir, path, tree) + ] + return external diff --git a/.claude/skills/porting-to-canyonos/validation/runtime.py b/.claude/skills/porting-to-canyonos/validation/runtime.py new file mode 100644 index 0000000..36b1e55 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/runtime.py @@ -0,0 +1,123 @@ +"""Runtime capabilities and dependency facts used by validation checks.""" + +import importlib +import os +import sys + + +RUNTIME_FLAT_NAMES = frozenset( + { + "future.py", + "ventis_context.py", + "local_controller.py", + "local_controller_frontend.py", + "redis_client.py", + "grpc_options.py", + "gpu_metrics.py", + "bedrock.py", + "deploy.py", + "session_logging.py", + "workflow_launcher.py", + } +) + +IMPORT_TO_DISTRIBUTION = { + "attr": ("attrs",), + "autogen": ("pyautogen", "ag2", "autogen", "autogen-agentchat"), + "bs4": ("beautifulsoup4",), + "cv2": ("opencv-python",), + "dateutil": ("python-dateutil",), + "dotenv": ("python-dotenv",), + "grpc": ("grpcio",), + "grpc_tools": ("grpcio-tools",), + "jwt": ("pyjwt",), + "PIL": ("pillow",), + "psycopg": ("psycopg",), + "psycopg2": ("psycopg2-binary",), + "pydantic_settings": ("pydantic-settings",), + "sklearn": ("scikit-learn",), + "typing_extensions": ("typing-extensions",), + "yaml": ("pyyaml",), +} + +NAMESPACE_DISTRIBUTIONS = {"llama_index": "llama-index"} + +CAPABILITY_SOURCE = { + "env_file": "runtime env-file injection", + "editable_install": "editable project installation", + "sweeps_all_files": "full project-file sweep", +} + + +def _base_requirements(): + agent = [ + "grpcio", + "grpcio-tools", + "redis", + "pyyaml", + "psutil", + "ipdb", + "ipython", + "boto3", + ] + workflow = [*agent, "flask", "sqlalchemy", "psycopg[binary]"] + try: + from ventis import stub_generator + except Exception: # noqa: BLE001 - a broken install must not crash validation + return agent, workflow + return ( + list(getattr(stub_generator, "BASE_AGENT_REQUIREMENTS", agent)), + list(getattr(stub_generator, "BASE_WORKFLOW_REQUIREMENTS", workflow)), + ) + + +def _stdlib_names(): + names = getattr(sys, "stdlib_module_names", None) + if names: + return frozenset(names) + found = set(sys.builtin_module_names) + library = os.path.dirname(os.__file__) + try: + entries = os.listdir(library) + except OSError: + return frozenset(found) + for entry in entries: + if entry.endswith(".py"): + found.add(entry[:-3]) + elif "." not in entry and "-" not in entry: + found.add(entry) + return frozenset(found) + + +BASE_AGENT_REQUIREMENTS, BASE_WORKFLOW_REQUIREMENTS = _base_requirements() +STDLIB_MODULE_NAMES = _stdlib_names() + + +def probe_capabilities(): + """Probe the installed compatibility runtime behind the CanyonOS CLI.""" + capabilities = dict.fromkeys(CAPABILITY_SOURCE, False) + capabilities["ventis"] = False + try: + from ventis import stub_generator + except Exception: # noqa: BLE001 - unavailable runtime is reported, not fatal + return capabilities + + capabilities["ventis"] = True + capabilities["editable_install"] = hasattr(stub_generator, "_install_step") + capabilities["sweeps_all_files"] = hasattr( + stub_generator, "_sweep_project_files" + ) + + for module_name in ( + "ventis.controller.utils.env_file", + "ventis.utils.env_file", + ): + try: + module = importlib.import_module(module_name) + except Exception: # noqa: BLE001 - try the other supported location + continue + if hasattr(module, "resolve_env_file"): + capabilities["env_file"] = True + break + return capabilities + From 6f4e8675bad5fd52efbeeb0cf825d84bd83bad46 Mon Sep 17 00:00:00 2001 From: Nick Huo Date: Fri, 4 Sep 2026 13:02:07 -0700 Subject: [PATCH 12/14] Finish splitting the validator, and stop citing a table that no longer exists validate.py still held nineteen of the twenty-five checks after the last split, so `validation/` was a helper package with one very large caller rather than a set of checks. The remaining check functions move out by the section boundaries the file already had: validation/adapter.py V006-V010 adapter faults _load_agent swallows validation/workflow.py V016-V018, V023 the workflow and how it reaches an agent validation/entrypoint.py V019, V020, V033-V035 traps set by which module the entrypoint names validation/packaging.py V030-V031 credentials and import roots validation/dependencies.py W003, W006 credentials and imports a successful build does not reject `module_path` and the two import resolvers join python_source.py, which is where the other static-source helpers already live; both resolvers were private and are now shared by packaging.py and dependencies.py, so they lose the underscore. validate.py keeps argparse, the driver, and report printing: 1153 lines to 311. This is a pure move. All twenty-four relocated definitions are AST-identical to their originals, and ten fixtures covering every check code that lives outside manifest.py -- V002, V006-V010, V016-V020, V023, V030-V035, W003, W006 -- produce byte-identical `--json` output before and after. Four references still cited hard-rule labels by number: references/adapter.md M18 references/llm-proxy.md M18, M21 references/manifest.md M22, M23 The M1-M32 table went away when SKILL.md gained `## Source-integrity boundary`, which restates the four rules static analysis cannot prove and drops the twenty-eight the validator now checks itself. The rules survived; the labels did not, so each citation named a table a reader could not find. They now name the boundary. Also drops nine dead `# noqa: E402` directives and sorts the new modules' imports, which nets the skill's scripts down from twelve ruff findings to seven. The seven that remain are in prepare.py and runtime.py and predate this change. (cherry picked from commit 7cfbece8cb84a2bb591aa88d346a4742a95fb540) --- .../porting-to-canyonos/references/adapter.md | 5 +- .../references/llm-proxy.md | 6 +- .../references/manifest.md | 11 +- .../skills/porting-to-canyonos/validate.py | 876 +----------------- .../validation/__init__.py | 1 - .../porting-to-canyonos/validation/adapter.py | 175 ++++ .../porting-to-canyonos/validation/core.py | 1 - .../validation/dependencies.py | 207 +++++ .../validation/entrypoint.py | 141 +++ .../validation/manifest.py | 5 +- .../validation/packaging.py | 108 +++ .../validation/python_source.py | 33 +- .../porting-to-canyonos/validation/runtime.py | 5 +- .../validation/workflow.py | 213 +++++ 14 files changed, 906 insertions(+), 881 deletions(-) create mode 100644 .claude/skills/porting-to-canyonos/validation/adapter.py create mode 100644 .claude/skills/porting-to-canyonos/validation/dependencies.py create mode 100644 .claude/skills/porting-to-canyonos/validation/entrypoint.py create mode 100644 .claude/skills/porting-to-canyonos/validation/packaging.py create mode 100644 .claude/skills/porting-to-canyonos/validation/workflow.py diff --git a/.claude/skills/porting-to-canyonos/references/adapter.md b/.claude/skills/porting-to-canyonos/references/adapter.md index 00e8d37..606e4ba 100644 --- a/.claude/skills/porting-to-canyonos/references/adapter.md +++ b/.claude/skills/porting-to-canyonos/references/adapter.md @@ -45,8 +45,9 @@ another. 5. **Does module-level code perform a real run?** A script ending in `result = crew.kickoff(...)` / `print(result)` fires that run whenever the module loads, before a request exists. Delete the - invocation and keep the construction: M18 protects prompts, tools, schemas - and model calls, not a script's own main body. + invocation and keep the construction. SKILL.md's source-integrity boundary + protects prompts, tools, schemas, model calls and node bodies -- not a + script's own main body. ## Bridging async diff --git a/.claude/skills/porting-to-canyonos/references/llm-proxy.md b/.claude/skills/porting-to-canyonos/references/llm-proxy.md index f660672..d2047bb 100644 --- a/.claude/skills/porting-to-canyonos/references/llm-proxy.md +++ b/.claude/skills/porting-to-canyonos/references/llm-proxy.md @@ -57,9 +57,9 @@ credentials in the separate proxy process, not in the port's `env_file`. Some sources build the HTTP call themselves -- `urllib.request` against a module constant like `API = "https://api.openai.com/v1/responses"` -- and read no -base-URL variable at all. Editing that constant is a source edit M18 and M21 -forbid, so the `env_file` is inert and the container can only ever reach the -real provider. Report this as a proxy blocker and stop. Do not hand the +base-URL variable at all. Editing that constant swaps the source provider's +endpoint, which SKILL.md's source-integrity boundary forbids, so the `env_file` +is inert and the container can only ever reach the real provider. Report this as a proxy blocker and stop. Do not hand the container a real upstream credential instead. Detect it before deploying: grep the source for the provider hostname. A literal diff --git a/.claude/skills/porting-to-canyonos/references/manifest.md b/.claude/skills/porting-to-canyonos/references/manifest.md index 1a6a9df..e90188b 100644 --- a/.claude/skills/porting-to-canyonos/references/manifest.md +++ b/.claude/skills/porting-to-canyonos/references/manifest.md @@ -102,9 +102,9 @@ you wrote: In a peer image the entrypoint is a stub, but its package `__init__` and its siblings are real, so that image still installs what they import. 3. Omit distributions reachable only from source files no image imports, such as - a Gradio or Streamlit UI beside the agent. M22 forbids reclassifying a - declared dependency, not declining to ship an unreachable one; name what you - left out in the report. + a Gradio or Streamlit UI beside the agent. The source-integrity boundary + forbids reclassifying a declared dependency, not declining to ship an + unreachable one; name what you left out in the report. `validate.py` walks the same graph and reports what is missing as W006. @@ -121,8 +121,9 @@ today: - **The source predates a known SDK break**: pin contemporaneous with its last commit. A 2023 AutoGen script passing `request_timeout=` needs `pyautogen==0.1.14`, which depends on `openai<1`, not `autogen==0.7.5`, which - floors on `openai>=1.58` where that kwarg is `timeout`. M23 forbids rewriting - the source call, so the pin has to absorb the difference. Compare the source's + floors on `openai>=1.58` where that kwarg is `timeout`. The source-integrity + boundary forbids rewriting that call, so the pin has to absorb the + difference. Compare the source's commit date against the pin's release date whenever the source hardcodes SDK kwargs. diff --git a/.claude/skills/porting-to-canyonos/validate.py b/.claude/skills/porting-to-canyonos/validate.py index 0eb9e93..61788e8 100755 --- a/.claude/skills/porting-to-canyonos/validate.py +++ b/.claude/skills/porting-to-canyonos/validate.py @@ -21,885 +21,48 @@ """ import argparse -import ast -import builtins import json import os -import re import sys SKILL_DIR = os.path.dirname(os.path.abspath(__file__)) if SKILL_DIR not in sys.path: sys.path.insert(0, SKILL_DIR) -from validation.core import ERROR, INFO, WARN, Report, line_of, load_yaml # noqa: E402 -from validation.manifest import ( # noqa: E402 +from validation.adapter import check_adapter +from validation.core import ERROR, INFO, WARN, Report, load_yaml +from validation.dependencies import ( + check_requirements_coverage, + check_secrets, +) +from validation.entrypoint import ( + check_entrypoint_module, + check_flat_collisions, +) +from validation.manifest import ( check_declaration_bindings, check_manifest_structure, check_policy, check_self_contained_tree, discover_agent_declarations, ) -from validation.python_source import ( # noqa: E402 - class_methods, - find_class, - parameter_names, - parse_python, - reachable_imports, - required_parameters, - toplevel_import_names, -) -from validation.runtime import ( # noqa: E402 +from validation.packaging import check_env_file, check_import_root +from validation.python_source import module_path +from validation.runtime import ( BASE_AGENT_REQUIREMENTS, BASE_WORKFLOW_REQUIREMENTS, CAPABILITY_SOURCE, - IMPORT_TO_DISTRIBUTION, - NAMESPACE_DISTRIBUTIONS, - RUNTIME_FLAT_NAMES, - STDLIB_MODULE_NAMES, probe_capabilities, ) +from validation.workflow import check_workflow DEFAULT_CONFIG_PATH = "config/global_controller.yaml" # ventis/cli.py SOURCE_DIR_NAME -- the duplicated application source. SOURCE_DIR_NAME = "app" -SECRET_PATTERNS = [ - (re.compile(r"sk-[A-Za-z0-9_-]{20,}"), "an OpenAI-style secret key"), - (re.compile(r"AKIA[0-9A-Z]{16}"), "an AWS access key id"), - (re.compile(r"gh[pousr]_[A-Za-z0-9]{20,}"), "a GitHub token"), - (re.compile(r"AIza[0-9A-Za-z_-]{30,}"), "a Google API key"), -] -SECRET_NAME = re.compile(r"(API_KEY|SECRET|TOKEN|PASSWORD|CREDENTIAL)", re.IGNORECASE) - - -# ------------------------------------------------------------------ # -# V006-V010 adapter failures hidden by _load_agent # -# ------------------------------------------------------------------ # - -BUILTIN_TYPE_NAMES = frozenset( - name - for name in dir(builtins) - if isinstance(getattr(builtins, name), type) and not name[0].isupper() -) - - -def check_adapter(report, agent_yaml_path, agent_block, entry, project_dir): - """V006 V007 V008 V009 V010.""" - name = agent_block["name"] - functions = agent_block.get("functions") or [] - check_argument_types(report, agent_yaml_path, functions) - - entrypoint = entry.get("entrypoint") - if not entrypoint: - return - entrypoint_path = os.path.join(project_dir, entrypoint) - if not os.path.isfile(entrypoint_path): - return - - tree, error = parse_python(entrypoint_path) - if tree is None: - report.error( - "V006", - entrypoint_path, - 0, - f"the entrypoint does not parse: {error}", - "_load_agent exec_module's it and swallows the exception; the first " - "request answers 'No agent loaded'.", - ) - return - - class_node = find_class(tree, name) - if class_node is None: - classes = [n.name for n in tree.body if isinstance(n, ast.ClassDef)] - found = ", ".join(classes) if classes else "no classes at all" - report.error( - "V006", - entrypoint_path, - 1, - f"no class named `{name}` at module level (found: {found})", - "_load_agent does getattr(module, VENTIS_AGENT_NAME) and swallows " - "the AttributeError. The class name must equal agent.name exactly.", - ) - return - - methods = class_methods(class_node) - check_constructor(report, entrypoint_path, name, methods) - - for func in functions: - if not isinstance(func, dict) or not isinstance(func.get("name"), str): - continue - check_method(report, entrypoint_path, agent_yaml_path, name, func, methods) - - -def check_argument_types(report, agent_yaml_path, functions): - """V010 -- the type string is pasted into an ast.Name, never checked.""" - for func in functions: - if not isinstance(func, dict): - continue - for arg in func.get("arguments") or []: - if not isinstance(arg, dict) or "type" not in arg: - continue - declared = arg.get("type") - if not isinstance(declared, str): - continue # stub generation reports malformed type values - if declared in BUILTIN_TYPE_NAMES: - continue - report.error( - "V010", - agent_yaml_path, - line_of(arg, "type"), - f"`type: {declared}` is not a builtin", - "stub_generator pastes it verbatim into the generated " - "annotation, and the stub module imports only Future and " - "inspect. Anything else raises NameError when the stub is " - "imported -- after a green build. Use str int float bool dict " - "list.", - ) - - -def check_constructor(report, entrypoint_path, name, methods): - """V007 -- _load_agent calls agent_class() with no arguments.""" - init = methods.get("__init__") - if init is None: - return - required = required_parameters(init) - if required: - report.error( - "V007", - entrypoint_path, - init.lineno, - f"`{name}.__init__` requires {', '.join(required)}", - "_load_agent calls agent_class() with no arguments; the TypeError is " - "swallowed and the first request answers 'No agent loaded'. Read " - "configuration from the environment inside __init__ instead.", - ) - - -def check_method(report, entrypoint_path, agent_yaml_path, class_name, func, methods): - """V008 V009.""" - func_name = func["name"] - method = methods.get(func_name) - if method is None: - report.error( - "V008", - entrypoint_path, - 0, - f"`{class_name}` has no method `{func_name}`", - "The yaml declares it, so callers get a stub for it; the controller " - f"then answers \"Agent {class_name} has no method '{func_name}'\".", - ) - return - - # V009 -- nothing on the execution path awaits. - if isinstance(method, ast.AsyncFunctionDef): - report.error( - "V009", - entrypoint_path, - method.lineno, - f"`{class_name}.{func_name}` is `async def`", - "The executor calls method(**args) with no await, so Redis receives " - "''. Keep the signature synchronous and call " - "asyncio.run(...) inside the body.", - ) - - # V008 -- the controller calls method(**args) with the yaml's names. - declared = [ - arg["name"] - for arg in func.get("arguments") or [] - if isinstance(arg, dict) and isinstance(arg.get("name"), str) - ] - actual = parameter_names(method) - required = required_parameters(method) - - missing = [arg for arg in declared if arg not in actual] - if missing: - report.error( - "V008", - entrypoint_path, - method.lineno, - f"`{class_name}.{func_name}` has no parameter " - f"{', '.join(repr(m) for m in missing)}, but the yaml declares it", - "LocalController does method(**args) with the yaml's argument names. " - "A mismatch is TypeError: unexpected keyword argument, at request " - f"time. See {os.path.basename(agent_yaml_path)}.", - ) - - unfilled = [arg for arg in required if arg not in declared] - if unfilled: - report.error( - "V008", - entrypoint_path, - method.lineno, - f"`{class_name}.{func_name}` requires {', '.join(unfilled)}, which " - "the yaml does not declare", - "Only declared arguments are ever sent, and the generated stub gives " - "none of them a default. Declare them in the yaml or default them " - "in the signature.", - ) - - - -# ------------------------------------------------------------------ # -# V016-V018 the workflow # -# ------------------------------------------------------------------ # - - -def check_stub_imports(report, workflow_path, tree, stub_modules): - """V023 -- the workflow must import each agent from its own entrypoint module. - - The build writes a stub over exactly one path: the agent's `entrypoint` - inside the source copy. An import that reaches the class any other way -- - flat, through a package re-export, or from a second copy of the module -- - resolves to the real class instead, and the workflow runs the agent - in-process with none of the deployment behind it. The class name is another - trap: the deploy build prints one with a `Stub` suffix that it never writes. - """ - for node in ast.walk(tree): - if not isinstance(node, ast.ImportFrom) or not node.module: - continue - for alias in node.names: - name = alias.name - base = name.removesuffix("Stub") - expected = stub_modules.get(base) - if expected is None: - continue - if name.endswith("Stub"): - report.error( - "V023", workflow_path, node.lineno, - f"`{name}` is the name the build prints, not the class it " - f"writes", - "generate_stub sets class_name = agent_config['name'] and " - "then recomputes it with a 'Stub' suffix for the log line " - "only. The message names a class that does not exist; the " - f"class is `{base}`.", - ) - elif node.module != expected: - report.error( - "V023", workflow_path, node.lineno, - f"`from {node.module} import {name}` -- the stub for {name} " - f"is written to {expected.replace('.', '/')}.py", - "The build replaces the module at the agent's entrypoint " - "and nothing else, so this import reaches the real class " - "and runs the agent in this process instead of over gRPC. " - f"Import it from `{expected}`, where the source already " - "keeps it.", - ) - - -def check_workflow(report, workflow_path, stub_modules=None): - """V016 V017 V018 V023.""" - tree, error = parse_python(workflow_path) - if tree is None: - report.error("V016", workflow_path, 0, f"does not parse: {error}", "") - return - - main = None - for node in tree.body: - if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and ( - node.name == "main" - ): - main = node - break - if main is None: - defined = [ - n.name - for n in tree.body - if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef)) - ] - found = ", ".join(defined) if defined else "no top-level functions" - report.error( - "V016", - workflow_path, - 1, - f"no top-level function named `main` (found: {found})", - "CanyonOS Core serves POST /, but the deployment platform's " - "test endpoint posts to a hardcoded /main. A differently named " - "workflow builds, deploys and stays unreachable -- 404, container " - "healthy.", - ) - else: - check_main_signature(report, workflow_path, main) - - if stub_modules: - check_stub_imports(report, workflow_path, tree, stub_modules) - - # V016 -- deploy() is what starts Flask. - if not any( - isinstance(node, ast.Call) - and isinstance(node.func, ast.Name) - and node.func.id == "deploy" - for node in ast.walk(tree) - ): - report.error( - "V016", - workflow_path, - 1, - "the workflow never calls `deploy(...)`", - "workflow_launcher.py exec's this file and nothing else starts the " - "HTTP server; the container comes up serving nothing.", - ) - - check_main_guard(report, workflow_path, tree) - check_fused_fanout(report, workflow_path, tree) - - -def check_main_signature(report, workflow_path, main): - """V016 -- the platform sends exactly {"query": ...}.""" - if isinstance(main, ast.AsyncFunctionDef): - report.error( - "V016", - workflow_path, - main.lineno, - "`main` is `async def`", - "deploy() calls workflow_fn(**kwargs) on a Flask worker thread with " - "no await; the response body would be a coroutine repr.", - ) - params = parameter_names(main) - if not params: - report.error( - "V016", - workflow_path, - main.lineno, - "`main` takes no arguments", - 'The platform posts {"query": "..."} and deploy() splats the ' - "body in as kwargs -- TypeError on every request.", - ) - return - if params[0] != "query": - report.error( - "V016", - workflow_path, - main.lineno, - f"`main`'s first parameter is `{params[0]}`, not `query`", - "The platform's body schema is strictly validated as " - "{query: string}; any other key is rejected with 400 in the control " - "plane, before the request reaches the host.", - ) - extra = [p for p in required_parameters(main) if p != "query"] - if extra: - report.error( - "V016", - workflow_path, - main.lineno, - f"`main` requires {', '.join(extra)} beyond `query`", - "Only `query` is ever sent, so every other parameter needs a " - "default or the call raises on every request. Pack richer input " - "into `query`.", - ) - - -def check_main_guard(report, workflow_path, tree): - """V017 -- the workflow is exec'd, so __name__ == "__main__".""" - for node in tree.body: - if not isinstance(node, ast.If): - continue - test = node.test - if ( - isinstance(test, ast.Compare) - and isinstance(test.left, ast.Name) - and test.left.id == "__name__" - and any( - isinstance(c, ast.Constant) and c.value == "__main__" - for c in test.comparators - ) - ): - report.error( - "V017", - workflow_path, - node.lineno, - '`if __name__ == "__main__":` block in the workflow', - "workflow_launcher.py runs exec(open().read()), so " - "__name__ IS '__main__' here and this block executes in " - "production, at container start.", - ) - - -def check_fused_fanout(report, workflow_path, tree): - """V018 -- .value() blocks, so dispatching and resolving in one - comprehension runs the fan-out one call at a time.""" - comprehensions = (ast.ListComp, ast.SetComp, ast.GeneratorExp) - for node in ast.walk(tree): - if not isinstance(node, comprehensions + (ast.DictComp,)): - continue - elements = ( - [node.key, node.value] if isinstance(node, ast.DictComp) else [node.elt] - ) - for element in elements: - for inner in ast.walk(element): - if ( - isinstance(inner, ast.Call) - and isinstance(inner.func, ast.Attribute) - and inner.func.attr == "value" - and isinstance(inner.func.value, ast.Call) - ): - report.error( - "V018", - workflow_path, - node.lineno, - "one comprehension both dispatches a call and resolves " - "it with .value()", - ".value() blocks, so each call completes before the " - "next is dispatched. It does not error -- the fan-out " - "is just silently serial, and with it the reason to be " - "on CanyonOS Core. Dispatch every call first, then resolve: " - "futures = [a.work(i) for i in items] then " - "[f.value() for f in futures].", - ) - return - - -# ------------------------------------------------------------------ # -# V019-V020 what the copy order overwrites # -# ------------------------------------------------------------------ # - - -def check_flat_collisions(report, source_dir, entrypoints): - """V019 V020 -- later copies land on earlier ones at the context root.""" - for entry in sorted(os.listdir(source_dir)): - path = os.path.join(source_dir, entry) - if not os.path.isfile(path) or not entry.endswith(".py"): - continue - - # V019 -- the shared runtime is copied flat, after the project sweep. - if entry in RUNTIME_FLAT_NAMES: - report.error( - "V019", - path, - 1, - f"a project module named `{entry}` sits at the root of the " - "source copy", - "The shared CanyonOS Core runtime is copied flat into the image after " - "the project sweep, so this file is overwritten by CanyonOS Core's own " - f"{entry}. Rename it or move it into a package directory.", - ) - - # V020 -- one module cannot be the entrypoint of two agents. - owners = {} - for name, entrypoint in entrypoints: - owners.setdefault(entrypoint, []).append(name) - for entrypoint, names in sorted(owners.items()): - if len(names) < 2: - continue - report.error( - "V020", - os.path.join(source_dir, entrypoint), - 1, - f"{' and '.join(sorted(names))} both declare `{entrypoint}` as their " - "entrypoint", - "Each agent's stub is written over its own entrypoint, so the two " - "land on one path and the last one built wins. Every caller then " - "reaches whichever agent that was. Give each agent its own module.", - ) - - -# ------------------------------------------------------------------ # -# V030-V031 capability-gated rules # -# ------------------------------------------------------------------ # - - -def check_env_file(report, config, config_path, artifact_dir): - """V030 -- gated on detected env-file injection support.""" - declared = config.get("env_file") - supported = report.capabilities.get("env_file") - - if not supported: - if declared: - report.error( - "V030", - config_path, - line_of(config, "env_file"), - f"`env_file: {declared}` is set, but this CanyonOS Core never reads it", - "No resolve_env_file in the importable ventis package, so the " - "key is silently dropped and the container answers a provider " - "credential error on the first request. This port requires the " - "`env_file` runtime capability.", - ) - else: - report.unavailable( - "V030", - "env_file is not supported by the importable `ventis` runtime. " - "Credentials have no declared path into a container on this tree.", - ) - return - - if not declared: - report.warn( - "V030", - config_path, - line_of(config), - "no `env_file:` in the config", - "Only runtime-managed VENTIS_* variables are guaranteed without it. " - "If the source reads credentials from the environment, the first " - "request fails on a provider error.", - ) - return - - # Path existence and readability are deploy-preflight checks. Do not - # duplicate them here. - - -def check_import_root(report, source_dir, entrypoint_paths): - """V031 -- gated on detected editable-install support.""" - supported = report.capabilities.get("editable_install") - has_metadata = any( - os.path.isfile(os.path.join(source_dir, name)) - for name in ("pyproject.toml", "setup.py", "setup.cfg") - ) - - non_flat = [] - for path in entrypoint_paths: - tree, _ = parse_python(path) - if tree is None: - continue - for name, lineno in toplevel_import_names(tree).items(): - if f"{name}.py" in RUNTIME_FLAT_NAMES: - continue - if _resolves_flat(source_dir, name): - continue - location = _resolves_nested(source_dir, name) - if location: - non_flat.append((path, lineno, name, location)) - - if not supported: - report.unavailable( - "V031", - "the editable install (`-e .`) is not supported by the importable " - "`ventis` runtime. Only names rooted at /app import inside a container.", - ) - for path, lineno, name, location in non_flat: - report.error( - "V031", - path, - lineno, - f"`import {name}` resolves to {location}, which is not at the " - "root of the source copy", - "sys.path[0] is /app and this CanyonOS Core runs no editable install, " - "so only modules swept to the root import. The adapter raises " - "ModuleNotFoundError inside _load_agent and the first request " - "answers 'No agent loaded'.", - ) - return - - if non_flat and not has_metadata: - for path, lineno, name, location in non_flat: - report.error( - "V031", - path, - lineno, - f"`import {name}` resolves to {location}, and the source copy's " - "root has no packaging metadata", - "A pyproject.toml, setup.py or setup.cfg at the root of the " - "source copy is what adds `-e .`; metadata nested deeper in the " - "tree is ignored. Add minimal root metadata pointing at the " - "existing package directory. Without it the install is skipped " - "silently.", - ) - - -# ------------------------------------------------------------------ # -# V033-V035 traps set by where the entrypoint sits # -# ------------------------------------------------------------------ # - - -def check_entrypoint_module(report, source_dir, name, entrypoint): - """V033 V034 V035. - - Two runtime facts collide here. The build writes this agent's stub over - `entrypoint` in every image except this agent's own, and the controller - loads the real file by path rather than by import. Each breaks a module - layout that is correct everywhere else in Python. - """ - path = os.path.join(source_dir, entrypoint) - if not os.path.isfile(path): - return - - segments = os.path.splitext(entrypoint)[0].replace("\\", "/").split("/") - invalid = [part for part in segments if not part.isidentifier()] - if invalid: - report.error( - "V034", - path, - 0, - f"`{invalid[0]}` in the entrypoint path is not a Python identifier", - "The controller loads the entrypoint by file path, so this file runs " - "-- but the workflow has to import the class from " - f"`{module_path(entrypoint)}` (V023), and that is a SyntaxError, not " - "an ImportError. Rename the file inside the copy, or point " - "`entrypoint` at a normally-named sibling that loads this file by " - "path and re-exposes the class.", - ) - - tree, _ = parse_python(path) - if tree is not None: - for node in ast.walk(tree): - if isinstance(node, ast.ImportFrom) and node.level: - spelling = "." * node.level + (node.module or "") - report.error( - "V035", - path, - node.lineno, - f"the entrypoint's own `from {spelling} import ...` is relative", - "_load_agent loads this file with spec_from_file_location(" - "VENTIS_AGENT_FILE.replace('.py', ''), path). That name keeps " - "the entrypoint's directory separator, so it has no parent " - "package and __package__ is empty: every relative import in " - "this file raises 'attempted relative import with no known " - "parent package' at agent load, behind 'No agent loaded'. " - "Make this file's own top-level imports absolute; modules it " - "imports may keep theirs.", - ) - break - - directory = os.path.dirname(entrypoint) - if not directory: - return - init_path = os.path.join(source_dir, directory, "__init__.py") - if not os.path.isfile(init_path): - return - module = os.path.splitext(os.path.basename(entrypoint))[0] - package = directory.replace("\\", "/").replace("/", ".") - init_tree, _ = parse_python(init_path) - if init_tree is None: - return - for node in ast.walk(init_tree): - if not isinstance(node, ast.ImportFrom): - continue - target = node.module or "" - hit = target == module if node.level else target in ( - module, - f"{package}.{module}", - ) - if not hit and node.level and not node.module: - hit = any(alias.name == module for alias in node.names) - if not hit: - continue - report.error( - "V033", - init_path, - node.lineno, - f"`{package}/__init__.py` re-exports from `{module}`, the entrypoint " - f"for {name}", - "Python runs a package's __init__.py before any of its submodules, " - "and in every image except this agent's own the module at the " - "entrypoint is the generated stub, which defines the agent class and " - f"nothing else. Any peer image that imports anything from `{package}` " - "-- the workflow importing the agent class included -- re-runs this " - "re-export against the stub and dies at container startup with " - f"ImportError. Point `entrypoint` at a module `{package}/__init__.py` " - "does not re-export from; add one that imports the real module if " - "every existing module is re-exported.", - ) - break - - -def _resolves_flat(project_dir, name): - """Whether Python can resolve `name` with /app as its import root. - - A directory does not need __init__.py: PEP 420 namespace packages resolve - from sys.path just like regular packages. - """ - return os.path.isfile(os.path.join(project_dir, f"{name}.py")) or os.path.isdir( - os.path.join(project_dir, name) - ) - - -def _resolves_nested(project_dir, name): - """Where below /app `name` lives but cannot resolve as a top-level name.""" - for root, dirs, files in os.walk(project_dir): - dirs[:] = [d for d in dirs if not d.startswith(".") and d != "__pycache__"] - if root == project_dir: - continue - if f"{name}.py" in files: - return os.path.relpath(os.path.join(root, f"{name}.py"), project_dir) - if name in dirs: - return os.path.relpath(os.path.join(root, name), project_dir) - return None - - -# ------------------------------------------------------------------ # -# W003, W006 secrets and imports a green build does not reject # -# ------------------------------------------------------------------ # - - -def check_secrets(report, port_paths): - """W003 -- env_file is the way in; nothing else is.""" - for path in port_paths: - try: - with open(path, "r", encoding="utf-8") as handle: - lines = handle.read().splitlines() - except OSError: - continue - for number, line in enumerate(lines, start=1): - for pattern, description in SECRET_PATTERNS: - if pattern.search(line): - report.error( - "W003", - path, - number, - f"this line looks like {description}", - "Never put a secret in the source tree or the build " - "context. The build sweeps the project into every " - "image.", - ) - break - - tree, _ = parse_python(path) - if tree is None: - continue - for node in ast.walk(tree): - if not isinstance(node, ast.Assign): - continue - if not ( - isinstance(node.value, ast.Constant) - and isinstance(node.value.value, str) - and node.value.value.strip() - ): - continue - for target in node.targets: - if isinstance(target, ast.Name) and SECRET_NAME.search(target.id): - report.warn( - "W003", - path, - node.lineno, - f"`{target.id}` is assigned a literal string", - "Read it from the environment instead; the build sweeps " - "this file into every image.", - ) - - -def _pyproject_dependencies(project_dir): - """What `-e .` installs alongside `requirements:`, or None if unreadable. - - None and the empty set mean different things here: empty means the project - declares no dependencies, None means we could not find out -- a setup.py, or - a tomllib this interpreter does not have. The caller must not treat the - second as the first, or it warns about imports the install would satisfy. - """ - path = os.path.join(project_dir, "pyproject.toml") - if not os.path.isfile(path): - return None - try: - import tomllib - except ImportError: # < 3.11 - return None - try: - with open(path, "rb") as handle: - data = tomllib.load(handle) - except Exception: # noqa: BLE001 - malformed metadata is uv's error to give - return None - deps = (data.get("project") or {}).get("dependencies") - if not isinstance(deps, list): - return None - return {_normalize_distribution(d) for d in deps if isinstance(d, str)} - - -def check_requirements_coverage( - report, - project_dir, - entry, - root_path, - config_path, - base_requirements, - shadowed_paths=(), -): - """W006 -- an import the container cannot satisfy. - - Walks the whole import graph the image executes from `root_path`, not just - that one file: a distribution reached through a local module or a package - __init__ is exactly as missing, and exactly as invisible until the container - starts. - """ - declared = { - _normalize_distribution(item) - for item in (entry.get("requirements") or []) - if isinstance(item, str) - } - - # Where the editable install exists, `-e .` resolves the project's own - # [project.dependencies] in the same pass as `requirements:`. Warning about - # those is a false positive, and a false warning about a dependency is worse - # than none: it teaches the reader to dismiss this check. - editable = report.capabilities.get("editable_install") - metadata = any( - os.path.isfile(os.path.join(project_dir, name)) - for name in ("pyproject.toml", "setup.py", "setup.cfg") - ) - unreadable_metadata = False - if editable and metadata: - project_deps = _pyproject_dependencies(project_dir) - if project_deps is None: - unreadable_metadata = True - else: - declared |= project_deps - base = {_normalize_distribution(item) for item in base_requirements} - satisfied = base | declared - - external = reachable_imports(project_dir, root_path, shadowed_paths) - for dotted, (where, lineno) in sorted(external.items()): - name = dotted.split(".")[0] - if name in STDLIB_MODULE_NAMES or name == "ventis": - continue - # Provided by the image itself: the shared runtime is copied flat over - # the swept tree. A stub is not listed here -- it replaces a module the - # source copy already carries, so the tree checks below cover it. - if f"{name}.py" in RUNTIME_FLAT_NAMES: - continue - if _resolves_flat(project_dir, name) or _resolves_nested(project_dir, name): - continue - prefix = NAMESPACE_DISTRIBUTIONS.get(name) - if prefix and any( - item == prefix or item.startswith(prefix + "-") for item in declared - ): - continue - if _candidate_distributions(name) & satisfied: - continue - if unreadable_metadata: - mechanism = ( - "The container installs the base list, `requirements:`, and -- " - "since this project declares packaging metadata -- whatever " - "`-e .` resolves from it. That metadata could not be read here, " - f"so if it already requires `{name}` this line is noise; " - "otherwise it is a ModuleNotFoundError inside _load_agent and " - "'No agent loaded' on the first request." - ) - else: - mechanism = ( - "The container installs the base list plus `requirements:` and " - "nothing else, so this is a ModuleNotFoundError inside " - "_load_agent and 'No agent loaded' on the first request. If the " - f"distribution is named something other than `{name}`, declare " - f"that name in {report.rel(config_path)}." - ) - if os.path.realpath(where) != os.path.realpath(root_path): - mechanism += ( - f" This image never names `{name}` in {report.rel(root_path)}; " - f"it runs {report.rel(where)} on the way there, and that module " - "needs it." - ) - report.error( - "W006", - where, - lineno, - f"`import {dotted}` is in neither the runtime's base list nor " - f"{entry.get('name') or 'this entry'}'s `requirements:`", - mechanism, - ) - - -def _candidate_distributions(name): - """Every distribution name that would satisfy `import `.""" - return { - _normalize_distribution(item) - for item in IMPORT_TO_DISTRIBUTION.get(name, (name,)) - } - - -def _normalize_distribution(name): - return re.split(r"[<>=!\[;\s]", name.strip().lower(), maxsplit=1)[0].replace( - "_", "-" - ) +# Path existence and readability are deploy-preflight checks. Do not +# duplicate them here. # ------------------------------------------------------------------ # @@ -907,11 +70,6 @@ def _normalize_distribution(name): # ------------------------------------------------------------------ # -def module_path(entrypoint): - """Dotted module name an entrypoint has inside the container.""" - return os.path.splitext(entrypoint)[0].replace("\\", "/").replace("/", ".") - - def validate(artifact_dir, config_path, capabilities): """Check the public artifact contract and deeper runtime failure modes.""" report = Report(artifact_dir, capabilities) diff --git a/.claude/skills/porting-to-canyonos/validation/__init__.py b/.claude/skills/porting-to-canyonos/validation/__init__.py index f6152f8..aa7a1a3 100644 --- a/.claude/skills/porting-to-canyonos/validation/__init__.py +++ b/.claude/skills/porting-to-canyonos/validation/__init__.py @@ -1,2 +1 @@ """Composable validation checks for the CanyonOS porting skill.""" - diff --git a/.claude/skills/porting-to-canyonos/validation/adapter.py b/.claude/skills/porting-to-canyonos/validation/adapter.py new file mode 100644 index 0000000..445ff10 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/adapter.py @@ -0,0 +1,175 @@ +"""V006-V010 -- adapter faults the controller swallows inside _load_agent.""" + +import ast +import builtins +import os + +from validation.core import line_of +from validation.python_source import ( + class_methods, + find_class, + parameter_names, + parse_python, + required_parameters, +) + +BUILTIN_TYPE_NAMES = frozenset( + name + for name in dir(builtins) + if isinstance(getattr(builtins, name), type) and not name[0].isupper() +) + + +def check_adapter(report, agent_yaml_path, agent_block, entry, project_dir): + """V006 V007 V008 V009 V010.""" + name = agent_block["name"] + functions = agent_block.get("functions") or [] + check_argument_types(report, agent_yaml_path, functions) + + entrypoint = entry.get("entrypoint") + if not entrypoint: + return + entrypoint_path = os.path.join(project_dir, entrypoint) + if not os.path.isfile(entrypoint_path): + return + + tree, error = parse_python(entrypoint_path) + if tree is None: + report.error( + "V006", + entrypoint_path, + 0, + f"the entrypoint does not parse: {error}", + "_load_agent exec_module's it and swallows the exception; the first " + "request answers 'No agent loaded'.", + ) + return + + class_node = find_class(tree, name) + if class_node is None: + classes = [n.name for n in tree.body if isinstance(n, ast.ClassDef)] + found = ", ".join(classes) if classes else "no classes at all" + report.error( + "V006", + entrypoint_path, + 1, + f"no class named `{name}` at module level (found: {found})", + "_load_agent does getattr(module, VENTIS_AGENT_NAME) and swallows " + "the AttributeError. The class name must equal agent.name exactly.", + ) + return + + methods = class_methods(class_node) + check_constructor(report, entrypoint_path, name, methods) + + for func in functions: + if not isinstance(func, dict) or not isinstance(func.get("name"), str): + continue + check_method(report, entrypoint_path, agent_yaml_path, name, func, methods) + + +def check_argument_types(report, agent_yaml_path, functions): + """V010 -- the type string is pasted into an ast.Name, never checked.""" + for func in functions: + if not isinstance(func, dict): + continue + for arg in func.get("arguments") or []: + if not isinstance(arg, dict) or "type" not in arg: + continue + declared = arg.get("type") + if not isinstance(declared, str): + continue # stub generation reports malformed type values + if declared in BUILTIN_TYPE_NAMES: + continue + report.error( + "V010", + agent_yaml_path, + line_of(arg, "type"), + f"`type: {declared}` is not a builtin", + "stub_generator pastes it verbatim into the generated " + "annotation, and the stub module imports only Future and " + "inspect. Anything else raises NameError when the stub is " + "imported -- after a green build. Use str int float bool dict " + "list.", + ) + + +def check_constructor(report, entrypoint_path, name, methods): + """V007 -- _load_agent calls agent_class() with no arguments.""" + init = methods.get("__init__") + if init is None: + return + required = required_parameters(init) + if required: + report.error( + "V007", + entrypoint_path, + init.lineno, + f"`{name}.__init__` requires {', '.join(required)}", + "_load_agent calls agent_class() with no arguments; the TypeError is " + "swallowed and the first request answers 'No agent loaded'. Read " + "configuration from the environment inside __init__ instead.", + ) + + +def check_method(report, entrypoint_path, agent_yaml_path, class_name, func, methods): + """V008 V009.""" + func_name = func["name"] + method = methods.get(func_name) + if method is None: + report.error( + "V008", + entrypoint_path, + 0, + f"`{class_name}` has no method `{func_name}`", + "The yaml declares it, so callers get a stub for it; the controller " + f"then answers \"Agent {class_name} has no method '{func_name}'\".", + ) + return + + # V009 -- nothing on the execution path awaits. + if isinstance(method, ast.AsyncFunctionDef): + report.error( + "V009", + entrypoint_path, + method.lineno, + f"`{class_name}.{func_name}` is `async def`", + "The executor calls method(**args) with no await, so Redis receives " + "''. Keep the signature synchronous and call " + "asyncio.run(...) inside the body.", + ) + + # V008 -- the controller calls method(**args) with the yaml's names. + declared = [ + arg["name"] + for arg in func.get("arguments") or [] + if isinstance(arg, dict) and isinstance(arg.get("name"), str) + ] + actual = parameter_names(method) + required = required_parameters(method) + + missing = [arg for arg in declared if arg not in actual] + if missing: + report.error( + "V008", + entrypoint_path, + method.lineno, + f"`{class_name}.{func_name}` has no parameter " + f"{', '.join(repr(m) for m in missing)}, but the yaml declares it", + "LocalController does method(**args) with the yaml's argument names. " + "A mismatch is TypeError: unexpected keyword argument, at request " + f"time. See {os.path.basename(agent_yaml_path)}.", + ) + + unfilled = [arg for arg in required if arg not in declared] + if unfilled: + report.error( + "V008", + entrypoint_path, + method.lineno, + f"`{class_name}.{func_name}` requires {', '.join(unfilled)}, which " + "the yaml does not declare", + "Only declared arguments are ever sent, and the generated stub gives " + "none of them a default. Declare them in the yaml or default them " + "in the signature.", + ) diff --git a/.claude/skills/porting-to-canyonos/validation/core.py b/.claude/skills/porting-to-canyonos/validation/core.py index 684e5cc..490b949 100644 --- a/.claude/skills/porting-to-canyonos/validation/core.py +++ b/.claude/skills/porting-to-canyonos/validation/core.py @@ -96,4 +96,3 @@ def counts(self): errors = sum(1 for finding in self.findings if finding["level"] == ERROR) warnings = sum(1 for finding in self.findings if finding["level"] == WARN) return errors, warnings - diff --git a/.claude/skills/porting-to-canyonos/validation/dependencies.py b/.claude/skills/porting-to-canyonos/validation/dependencies.py new file mode 100644 index 0000000..4060319 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/dependencies.py @@ -0,0 +1,207 @@ +"""W003, W006 -- credentials and imports a successful build does not reject.""" + +import ast +import os +import re + +from validation.python_source import ( + parse_python, + reachable_imports, + resolves_flat, + resolves_nested, +) +from validation.runtime import ( + IMPORT_TO_DISTRIBUTION, + NAMESPACE_DISTRIBUTIONS, + RUNTIME_FLAT_NAMES, + STDLIB_MODULE_NAMES, +) + +SECRET_PATTERNS = [ + (re.compile(r"sk-[A-Za-z0-9_-]{20,}"), "an OpenAI-style secret key"), + (re.compile(r"AKIA[0-9A-Z]{16}"), "an AWS access key id"), + (re.compile(r"gh[pousr]_[A-Za-z0-9]{20,}"), "a GitHub token"), + (re.compile(r"AIza[0-9A-Za-z_-]{30,}"), "a Google API key"), +] + + +SECRET_NAME = re.compile(r"(API_KEY|SECRET|TOKEN|PASSWORD|CREDENTIAL)", re.IGNORECASE) + + +def check_secrets(report, port_paths): + """W003 -- env_file is the way in; nothing else is.""" + for path in port_paths: + try: + with open(path, "r", encoding="utf-8") as handle: + lines = handle.read().splitlines() + except OSError: + continue + for number, line in enumerate(lines, start=1): + for pattern, description in SECRET_PATTERNS: + if pattern.search(line): + report.error( + "W003", + path, + number, + f"this line looks like {description}", + "Never put a secret in the source tree or the build " + "context. The build sweeps the project into every " + "image.", + ) + break + + tree, _ = parse_python(path) + if tree is None: + continue + for node in ast.walk(tree): + if not isinstance(node, ast.Assign): + continue + if not ( + isinstance(node.value, ast.Constant) + and isinstance(node.value.value, str) + and node.value.value.strip() + ): + continue + for target in node.targets: + if isinstance(target, ast.Name) and SECRET_NAME.search(target.id): + report.warn( + "W003", + path, + node.lineno, + f"`{target.id}` is assigned a literal string", + "Read it from the environment instead; the build sweeps " + "this file into every image.", + ) + + +def _pyproject_dependencies(project_dir): + """What `-e .` installs alongside `requirements:`, or None if unreadable. + + None and the empty set mean different things here: empty means the project + declares no dependencies, None means we could not find out -- a setup.py, or + a tomllib this interpreter does not have. The caller must not treat the + second as the first, or it warns about imports the install would satisfy. + """ + path = os.path.join(project_dir, "pyproject.toml") + if not os.path.isfile(path): + return None + try: + import tomllib + except ImportError: # < 3.11 + return None + try: + with open(path, "rb") as handle: + data = tomllib.load(handle) + except Exception: # noqa: BLE001 - malformed metadata is uv's error to give + return None + deps = (data.get("project") or {}).get("dependencies") + if not isinstance(deps, list): + return None + return {_normalize_distribution(d) for d in deps if isinstance(d, str)} + + +def check_requirements_coverage( + report, + project_dir, + entry, + root_path, + config_path, + base_requirements, + shadowed_paths=(), +): + """W006 -- an import the container cannot satisfy. + + Walks the whole import graph the image executes from `root_path`, not just + that one file: a distribution reached through a local module or a package + __init__ is exactly as missing, and exactly as invisible until the container + starts. + """ + declared = { + _normalize_distribution(item) + for item in (entry.get("requirements") or []) + if isinstance(item, str) + } + + # Where the editable install exists, `-e .` resolves the project's own + # [project.dependencies] in the same pass as `requirements:`. Warning about + # those is a false positive, and a false warning about a dependency is worse + # than none: it teaches the reader to dismiss this check. + editable = report.capabilities.get("editable_install") + metadata = any( + os.path.isfile(os.path.join(project_dir, name)) + for name in ("pyproject.toml", "setup.py", "setup.cfg") + ) + unreadable_metadata = False + if editable and metadata: + project_deps = _pyproject_dependencies(project_dir) + if project_deps is None: + unreadable_metadata = True + else: + declared |= project_deps + base = {_normalize_distribution(item) for item in base_requirements} + satisfied = base | declared + + external = reachable_imports(project_dir, root_path, shadowed_paths) + for dotted, (where, lineno) in sorted(external.items()): + name = dotted.split(".")[0] + if name in STDLIB_MODULE_NAMES or name == "ventis": + continue + # Provided by the image itself: the shared runtime is copied flat over + # the swept tree. A stub is not listed here -- it replaces a module the + # source copy already carries, so the tree checks below cover it. + if f"{name}.py" in RUNTIME_FLAT_NAMES: + continue + if resolves_flat(project_dir, name) or resolves_nested(project_dir, name): + continue + prefix = NAMESPACE_DISTRIBUTIONS.get(name) + if prefix and any( + item == prefix or item.startswith(prefix + "-") for item in declared + ): + continue + if _candidate_distributions(name) & satisfied: + continue + if unreadable_metadata: + mechanism = ( + "The container installs the base list, `requirements:`, and -- " + "since this project declares packaging metadata -- whatever " + "`-e .` resolves from it. That metadata could not be read here, " + f"so if it already requires `{name}` this line is noise; " + "otherwise it is a ModuleNotFoundError inside _load_agent and " + "'No agent loaded' on the first request." + ) + else: + mechanism = ( + "The container installs the base list plus `requirements:` and " + "nothing else, so this is a ModuleNotFoundError inside " + "_load_agent and 'No agent loaded' on the first request. If the " + f"distribution is named something other than `{name}`, declare " + f"that name in {report.rel(config_path)}." + ) + if os.path.realpath(where) != os.path.realpath(root_path): + mechanism += ( + f" This image never names `{name}` in {report.rel(root_path)}; " + f"it runs {report.rel(where)} on the way there, and that module " + "needs it." + ) + report.error( + "W006", + where, + lineno, + f"`import {dotted}` is in neither the runtime's base list nor " + f"{entry.get('name') or 'this entry'}'s `requirements:`", + mechanism, + ) + + +def _candidate_distributions(name): + """Every distribution name that would satisfy `import `.""" + return { + _normalize_distribution(item) + for item in IMPORT_TO_DISTRIBUTION.get(name, (name,)) + } + + +def _normalize_distribution(name): + return re.split(r"[<>=!\[;\s]", name.strip().lower(), maxsplit=1)[0].replace( + "_", "-" + ) diff --git a/.claude/skills/porting-to-canyonos/validation/entrypoint.py b/.claude/skills/porting-to-canyonos/validation/entrypoint.py new file mode 100644 index 0000000..902c3f8 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/entrypoint.py @@ -0,0 +1,141 @@ +"""V019, V020, V033-V035 -- traps set by which module the entrypoint names.""" + +import ast +import os + +from validation.python_source import module_path, parse_python +from validation.runtime import RUNTIME_FLAT_NAMES + + +def check_flat_collisions(report, source_dir, entrypoints): + """V019 V020 -- later copies land on earlier ones at the context root.""" + for entry in sorted(os.listdir(source_dir)): + path = os.path.join(source_dir, entry) + if not os.path.isfile(path) or not entry.endswith(".py"): + continue + + # V019 -- the shared runtime is copied flat, after the project sweep. + if entry in RUNTIME_FLAT_NAMES: + report.error( + "V019", + path, + 1, + f"a project module named `{entry}` sits at the root of the source copy", + "The shared CanyonOS Core runtime is copied flat into the image after " + "the project sweep, so this file is overwritten by CanyonOS Core's own " + f"{entry}. Rename it or move it into a package directory.", + ) + + # V020 -- one module cannot be the entrypoint of two agents. + owners = {} + for name, entrypoint in entrypoints: + owners.setdefault(entrypoint, []).append(name) + for entrypoint, names in sorted(owners.items()): + if len(names) < 2: + continue + report.error( + "V020", + os.path.join(source_dir, entrypoint), + 1, + f"{' and '.join(sorted(names))} both declare `{entrypoint}` as their " + "entrypoint", + "Each agent's stub is written over its own entrypoint, so the two " + "land on one path and the last one built wins. Every caller then " + "reaches whichever agent that was. Give each agent its own module.", + ) + + +def check_entrypoint_module(report, source_dir, name, entrypoint): + """V033 V034 V035. + + Two runtime facts collide here. The build writes this agent's stub over + `entrypoint` in every image except this agent's own, and the controller + loads the real file by path rather than by import. Each breaks a module + layout that is correct everywhere else in Python. + """ + path = os.path.join(source_dir, entrypoint) + if not os.path.isfile(path): + return + + segments = os.path.splitext(entrypoint)[0].replace("\\", "/").split("/") + invalid = [part for part in segments if not part.isidentifier()] + if invalid: + report.error( + "V034", + path, + 0, + f"`{invalid[0]}` in the entrypoint path is not a Python identifier", + "The controller loads the entrypoint by file path, so this file runs " + "-- but the workflow has to import the class from " + f"`{module_path(entrypoint)}` (V023), and that is a SyntaxError, not " + "an ImportError. Rename the file inside the copy, or point " + "`entrypoint` at a normally-named sibling that loads this file by " + "path and re-exposes the class.", + ) + + tree, _ = parse_python(path) + if tree is not None: + for node in ast.walk(tree): + if isinstance(node, ast.ImportFrom) and node.level: + spelling = "." * node.level + (node.module or "") + report.error( + "V035", + path, + node.lineno, + f"the entrypoint's own `from {spelling} import ...` is relative", + "_load_agent loads this file with spec_from_file_location(" + "VENTIS_AGENT_FILE.replace('.py', ''), path). That name keeps " + "the entrypoint's directory separator, so it has no parent " + "package and __package__ is empty: every relative import in " + "this file raises 'attempted relative import with no known " + "parent package' at agent load, behind 'No agent loaded'. " + "Make this file's own top-level imports absolute; modules it " + "imports may keep theirs.", + ) + break + + directory = os.path.dirname(entrypoint) + if not directory: + return + init_path = os.path.join(source_dir, directory, "__init__.py") + if not os.path.isfile(init_path): + return + module = os.path.splitext(os.path.basename(entrypoint))[0] + package = directory.replace("\\", "/").replace("/", ".") + init_tree, _ = parse_python(init_path) + if init_tree is None: + return + for node in ast.walk(init_tree): + if not isinstance(node, ast.ImportFrom): + continue + target = node.module or "" + hit = ( + target == module + if node.level + else target + in ( + module, + f"{package}.{module}", + ) + ) + if not hit and node.level and not node.module: + hit = any(alias.name == module for alias in node.names) + if not hit: + continue + report.error( + "V033", + init_path, + node.lineno, + f"`{package}/__init__.py` re-exports from `{module}`, the entrypoint " + f"for {name}", + "Python runs a package's __init__.py before any of its submodules, " + "and in every image except this agent's own the module at the " + "entrypoint is the generated stub, which defines the agent class and " + f"nothing else. Any peer image that imports anything from `{package}` " + "-- the workflow importing the agent class included -- re-runs this " + "re-export against the stub and dies at container startup with " + f"ImportError. Point `entrypoint` at a module `{package}/__init__.py` " + "does not re-export from; add one that imports the real module if " + "every existing module is re-exported.", + ) + break diff --git a/.claude/skills/porting-to-canyonos/validation/manifest.py b/.claude/skills/porting-to-canyonos/validation/manifest.py index df40c23..a69b7d4 100644 --- a/.claude/skills/porting-to-canyonos/validation/manifest.py +++ b/.claude/skills/porting-to-canyonos/validation/manifest.py @@ -5,6 +5,7 @@ from validation.core import line_of, load_yaml + def _safe_relative_python_path(value): if not isinstance(value, str) or not value.strip(): return False @@ -247,9 +248,7 @@ def discover_agent_declarations(report, config_dir, config_path): def check_declaration_bindings(report, entries, declarations, config_path): """Require a one-to-one binding for every agent service.""" configured = { - entry["name"] - for entry in entries - if entry.get("type", "agent") != "workflow" + entry["name"] for entry in entries if entry.get("type", "agent") != "workflow" } for name in sorted(configured - declarations.keys()): report.error( diff --git a/.claude/skills/porting-to-canyonos/validation/packaging.py b/.claude/skills/porting-to-canyonos/validation/packaging.py new file mode 100644 index 0000000..d0682fc --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/packaging.py @@ -0,0 +1,108 @@ +"""V030-V031 -- capability-gated rules about credentials and import roots.""" + +import os + +from validation.core import line_of +from validation.python_source import ( + parse_python, + resolves_flat, + resolves_nested, + toplevel_import_names, +) +from validation.runtime import RUNTIME_FLAT_NAMES + + +def check_env_file(report, config, config_path, artifact_dir): + """V030 -- gated on detected env-file injection support.""" + declared = config.get("env_file") + supported = report.capabilities.get("env_file") + + if not supported: + if declared: + report.error( + "V030", + config_path, + line_of(config, "env_file"), + f"`env_file: {declared}` is set, but this CanyonOS Core never reads it", + "No resolve_env_file in the importable ventis package, so the " + "key is silently dropped and the container answers a provider " + "credential error on the first request. This port requires the " + "`env_file` runtime capability.", + ) + else: + report.unavailable( + "V030", + "env_file is not supported by the importable `ventis` runtime. " + "Credentials have no declared path into a container on this tree.", + ) + return + + if not declared: + report.warn( + "V030", + config_path, + line_of(config), + "no `env_file:` in the config", + "Only runtime-managed VENTIS_* variables are guaranteed without it. " + "If the source reads credentials from the environment, the first " + "request fails on a provider error.", + ) + return + + +def check_import_root(report, source_dir, entrypoint_paths): + """V031 -- gated on detected editable-install support.""" + supported = report.capabilities.get("editable_install") + has_metadata = any( + os.path.isfile(os.path.join(source_dir, name)) + for name in ("pyproject.toml", "setup.py", "setup.cfg") + ) + + non_flat = [] + for path in entrypoint_paths: + tree, _ = parse_python(path) + if tree is None: + continue + for name, lineno in toplevel_import_names(tree).items(): + if f"{name}.py" in RUNTIME_FLAT_NAMES: + continue + if resolves_flat(source_dir, name): + continue + location = resolves_nested(source_dir, name) + if location: + non_flat.append((path, lineno, name, location)) + + if not supported: + report.unavailable( + "V031", + "the editable install (`-e .`) is not supported by the importable " + "`ventis` runtime. Only names rooted at /app import inside a container.", + ) + for path, lineno, name, location in non_flat: + report.error( + "V031", + path, + lineno, + f"`import {name}` resolves to {location}, which is not at the " + "root of the source copy", + "sys.path[0] is /app and this CanyonOS Core runs no editable install, " + "so only modules swept to the root import. The adapter raises " + "ModuleNotFoundError inside _load_agent and the first request " + "answers 'No agent loaded'.", + ) + return + + if non_flat and not has_metadata: + for path, lineno, name, location in non_flat: + report.error( + "V031", + path, + lineno, + f"`import {name}` resolves to {location}, and the source copy's " + "root has no packaging metadata", + "A pyproject.toml, setup.py or setup.cfg at the root of the " + "source copy is what adds `-e .`; metadata nested deeper in the " + "tree is ignored. Add minimal root metadata pointing at the " + "existing package directory. Without it the install is skipped " + "silently.", + ) diff --git a/.claude/skills/porting-to-canyonos/validation/python_source.py b/.claude/skills/porting-to-canyonos/validation/python_source.py index 7c3eefe..50df1e6 100644 --- a/.claude/skills/porting-to-canyonos/validation/python_source.py +++ b/.claude/skills/porting-to-canyonos/validation/python_source.py @@ -115,9 +115,7 @@ def _relative_import_files(project_dir, path, tree): continue target = os.path.join(base, *(node.module.split(".") if node.module else [])) candidates = [target + ".py", os.path.join(target, "__init__.py")] - candidates += [ - os.path.join(target, alias.name + ".py") for alias in node.names - ] + candidates += [os.path.join(target, alias.name + ".py") for alias in node.names] candidates += [ os.path.join(target, alias.name, "__init__.py") for alias in node.names ] @@ -150,3 +148,32 @@ def reachable_imports(project_dir, root_path, shadowed_paths=()): for item in _relative_import_files(project_dir, path, tree) ] return external + + +def module_path(entrypoint): + """Dotted module name an entrypoint has inside the container.""" + return os.path.splitext(entrypoint)[0].replace("\\", "/").replace("/", ".") + + +def resolves_flat(project_dir, name): + """Whether Python can resolve `name` with /app as its import root. + + A directory does not need __init__.py: PEP 420 namespace packages resolve + from sys.path just like regular packages. + """ + return os.path.isfile(os.path.join(project_dir, f"{name}.py")) or os.path.isdir( + os.path.join(project_dir, name) + ) + + +def resolves_nested(project_dir, name): + """Where below /app `name` lives but cannot resolve as a top-level name.""" + for root, dirs, files in os.walk(project_dir): + dirs[:] = [d for d in dirs if not d.startswith(".") and d != "__pycache__"] + if root == project_dir: + continue + if f"{name}.py" in files: + return os.path.relpath(os.path.join(root, f"{name}.py"), project_dir) + if name in dirs: + return os.path.relpath(os.path.join(root, name), project_dir) + return None diff --git a/.claude/skills/porting-to-canyonos/validation/runtime.py b/.claude/skills/porting-to-canyonos/validation/runtime.py index 36b1e55..4ec438a 100644 --- a/.claude/skills/porting-to-canyonos/validation/runtime.py +++ b/.claude/skills/porting-to-canyonos/validation/runtime.py @@ -104,9 +104,7 @@ def probe_capabilities(): capabilities["ventis"] = True capabilities["editable_install"] = hasattr(stub_generator, "_install_step") - capabilities["sweeps_all_files"] = hasattr( - stub_generator, "_sweep_project_files" - ) + capabilities["sweeps_all_files"] = hasattr(stub_generator, "_sweep_project_files") for module_name in ( "ventis.controller.utils.env_file", @@ -120,4 +118,3 @@ def probe_capabilities(): capabilities["env_file"] = True break return capabilities - diff --git a/.claude/skills/porting-to-canyonos/validation/workflow.py b/.claude/skills/porting-to-canyonos/validation/workflow.py new file mode 100644 index 0000000..613c4ce --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/workflow.py @@ -0,0 +1,213 @@ +"""V016-V018, V023 -- the workflow module and how it reaches an agent.""" + +import ast + +from validation.python_source import parameter_names, parse_python, required_parameters + + +def check_stub_imports(report, workflow_path, tree, stub_modules): + """V023 -- the workflow must import each agent from its own entrypoint module. + + The build writes a stub over exactly one path: the agent's `entrypoint` + inside the source copy. An import that reaches the class any other way -- + flat, through a package re-export, or from a second copy of the module -- + resolves to the real class instead, and the workflow runs the agent + in-process with none of the deployment behind it. The class name is another + trap: the deploy build prints one with a `Stub` suffix that it never writes. + """ + for node in ast.walk(tree): + if not isinstance(node, ast.ImportFrom) or not node.module: + continue + for alias in node.names: + name = alias.name + base = name.removesuffix("Stub") + expected = stub_modules.get(base) + if expected is None: + continue + if name.endswith("Stub"): + report.error( + "V023", + workflow_path, + node.lineno, + f"`{name}` is the name the build prints, not the class it writes", + "generate_stub sets class_name = agent_config['name'] and " + "then recomputes it with a 'Stub' suffix for the log line " + "only. The message names a class that does not exist; the " + f"class is `{base}`.", + ) + elif node.module != expected: + report.error( + "V023", + workflow_path, + node.lineno, + f"`from {node.module} import {name}` -- the stub for {name} " + f"is written to {expected.replace('.', '/')}.py", + "The build replaces the module at the agent's entrypoint " + "and nothing else, so this import reaches the real class " + "and runs the agent in this process instead of over gRPC. " + f"Import it from `{expected}`, where the source already " + "keeps it.", + ) + + +def check_workflow(report, workflow_path, stub_modules=None): + """V016 V017 V018 V023.""" + tree, error = parse_python(workflow_path) + if tree is None: + report.error("V016", workflow_path, 0, f"does not parse: {error}", "") + return + + main = None + for node in tree.body: + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and ( + node.name == "main" + ): + main = node + break + + if main is None: + defined = [ + n.name + for n in tree.body + if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef)) + ] + found = ", ".join(defined) if defined else "no top-level functions" + report.error( + "V016", + workflow_path, + 1, + f"no top-level function named `main` (found: {found})", + "CanyonOS Core serves POST /, but the deployment platform's " + "test endpoint posts to a hardcoded /main. A differently named " + "workflow builds, deploys and stays unreachable -- 404, container " + "healthy.", + ) + else: + check_main_signature(report, workflow_path, main) + + if stub_modules: + check_stub_imports(report, workflow_path, tree, stub_modules) + + # V016 -- deploy() is what starts Flask. + if not any( + isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "deploy" + for node in ast.walk(tree) + ): + report.error( + "V016", + workflow_path, + 1, + "the workflow never calls `deploy(...)`", + "workflow_launcher.py exec's this file and nothing else starts the " + "HTTP server; the container comes up serving nothing.", + ) + + check_main_guard(report, workflow_path, tree) + check_fused_fanout(report, workflow_path, tree) + + +def check_main_signature(report, workflow_path, main): + """V016 -- the platform sends exactly {"query": ...}.""" + if isinstance(main, ast.AsyncFunctionDef): + report.error( + "V016", + workflow_path, + main.lineno, + "`main` is `async def`", + "deploy() calls workflow_fn(**kwargs) on a Flask worker thread with " + "no await; the response body would be a coroutine repr.", + ) + params = parameter_names(main) + if not params: + report.error( + "V016", + workflow_path, + main.lineno, + "`main` takes no arguments", + 'The platform posts {"query": "..."} and deploy() splats the ' + "body in as kwargs -- TypeError on every request.", + ) + return + if params[0] != "query": + report.error( + "V016", + workflow_path, + main.lineno, + f"`main`'s first parameter is `{params[0]}`, not `query`", + "The platform's body schema is strictly validated as " + "{query: string}; any other key is rejected with 400 in the control " + "plane, before the request reaches the host.", + ) + extra = [p for p in required_parameters(main) if p != "query"] + if extra: + report.error( + "V016", + workflow_path, + main.lineno, + f"`main` requires {', '.join(extra)} beyond `query`", + "Only `query` is ever sent, so every other parameter needs a " + "default or the call raises on every request. Pack richer input " + "into `query`.", + ) + + +def check_main_guard(report, workflow_path, tree): + """V017 -- the workflow is exec'd, so __name__ == "__main__".""" + for node in tree.body: + if not isinstance(node, ast.If): + continue + test = node.test + if ( + isinstance(test, ast.Compare) + and isinstance(test.left, ast.Name) + and test.left.id == "__name__" + and any( + isinstance(c, ast.Constant) and c.value == "__main__" + for c in test.comparators + ) + ): + report.error( + "V017", + workflow_path, + node.lineno, + '`if __name__ == "__main__":` block in the workflow', + "workflow_launcher.py runs exec(open().read()), so " + "__name__ IS '__main__' here and this block executes in " + "production, at container start.", + ) + + +def check_fused_fanout(report, workflow_path, tree): + """V018 -- .value() blocks, so dispatching and resolving in one + comprehension runs the fan-out one call at a time.""" + comprehensions = (ast.ListComp, ast.SetComp, ast.GeneratorExp) + for node in ast.walk(tree): + if not isinstance(node, comprehensions + (ast.DictComp,)): + continue + elements = ( + [node.key, node.value] if isinstance(node, ast.DictComp) else [node.elt] + ) + for element in elements: + for inner in ast.walk(element): + if ( + isinstance(inner, ast.Call) + and isinstance(inner.func, ast.Attribute) + and inner.func.attr == "value" + and isinstance(inner.func.value, ast.Call) + ): + report.error( + "V018", + workflow_path, + node.lineno, + "one comprehension both dispatches a call and resolves " + "it with .value()", + ".value() blocks, so each call completes before the " + "next is dispatched. It does not error -- the fan-out " + "is just silently serial, and with it the reason to be " + "on CanyonOS Core. Dispatch every call first, then resolve: " + "futures = [a.work(i) for i in items] then " + "[f.value() for f in futures].", + ) + return From ea945af6804c023ab0b7a972fcc17fddac27bf8e Mon Sep 17 00:00:00 2001 From: Nick Huo Date: Tue, 8 Sep 2026 12:08:50 -0700 Subject: [PATCH 13/14] small updates --- .claude/skills/porting-to-canyonos/SKILL.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/.claude/skills/porting-to-canyonos/SKILL.md b/.claude/skills/porting-to-canyonos/SKILL.md index 1dcfb8f..f994984 100644 --- a/.claude/skills/porting-to-canyonos/SKILL.md +++ b/.claude/skills/porting-to-canyonos/SKILL.md @@ -93,10 +93,11 @@ never inside it: `.car` has exactly two authored directories: `config/`, which holds every declaration Canyon owns, and `app/`, the copy that becomes `/app` in every -container. The container keeps the directory structure the application already -had. Write adapters into that copy, in the module the code they wrap already -lives in -- not into new `agents/` and `workflow/` directories. `canyonos` -commands run from the application root and read `.car` below it. +container. Running `prepare.py` (step 1) creates both; do not assemble them by +hand. The container keeps the directory structure the application already had. +Write adapters into that copy, in the module the code they wrap already lives +in -- not into new `agents/` and `workflow/` directories. `canyonos` commands +run from the application root and read `.car` below it. Nothing under `.car` points back out at the application source, and nothing in the application source points at `.car`. Deleting `.car` returns the project to From 04214661373601403faafabd9e948555caae90ee Mon Sep 17 00:00:00 2001 From: Nick Huo Date: Tue, 8 Sep 2026 15:17:20 -0700 Subject: [PATCH 14/14] update skill --- .claude/skills/porting-to-canyonos/SKILL.md | 271 +++--------------- .../porting-to-canyonos/references/adapter.md | 47 ++- .../porting-to-canyonos/references/ec2.md | 11 +- .../references/example-port.md | 132 --------- .../references/llm-proxy.md | 14 +- .../references/manifest.md | 54 ++-- .../references/packaging.md | 137 --------- .../references/preparation.md | 263 +++++++++++++++++ .../porting-to-canyonos/references/refresh.md | 38 --- .../references/runtime-contract.md | 24 +- .../references/source-survey.md | 122 ++++++-- .../references/troubleshooting.md | 13 +- .../references/validation-and-deploy.md | 70 +++++ .../skills/porting-to-canyonos/validate.py | 8 +- .../validation/manifest.py | 18 -- 15 files changed, 583 insertions(+), 639 deletions(-) delete mode 100644 .claude/skills/porting-to-canyonos/references/example-port.md delete mode 100644 .claude/skills/porting-to-canyonos/references/packaging.md create mode 100644 .claude/skills/porting-to-canyonos/references/preparation.md delete mode 100644 .claude/skills/porting-to-canyonos/references/refresh.md create mode 100644 .claude/skills/porting-to-canyonos/references/validation-and-deploy.md diff --git a/.claude/skills/porting-to-canyonos/SKILL.md b/.claude/skills/porting-to-canyonos/SKILL.md index f994984..5c23ad6 100644 --- a/.claude/skills/porting-to-canyonos/SKILL.md +++ b/.claude/skills/porting-to-canyonos/SKILL.md @@ -1,259 +1,68 @@ --- name: porting-to-canyonos -description: Ports existing LangChain, LangGraph, CrewAI, AutoGen, and hand-rolled Python agent projects onto CanyonOS Core, whose CLI is `canyonos` and whose artifacts live in a `.car` directory. Writes `.car/config`, copies source into `.car/app`, writes adapters and the workflow, and validates the port. Stops after validation and asks before running `canyonos deploy`, which performs both build and deployment. Use when converting, migrating, adapting, packaging, validating, or deploying an existing agent or multi-agent project onto CanyonOS Core, or when a `.car` port fails validation, build, load, or deployment. +description: Port existing Python agents—including LangChain, LangGraph, CrewAI, AutoGen, and custom implementations—to CanyonOS Core. Use for migrations, `.car` packaging, adapter and workflow generation, validation, deployment, or diagnosing build and runtime failures. Creates and validates a self-contained `.car`, then requires explicit approval before `canyonos deploy`. --- -# Port an agent project to CanyonOS Core +# Port an agent project to CanyonOS -Requires Python, Docker, and the `canyonos` CLI. `prepare.py` uses only the -Python standard library; `validate.py` needs Python 3 and `pyyaml`. +Requires Python, Docker, and the `canyonos` CLI. `prepare.py` uses the Python +standard library; `validate.py` also requires `pyyaml`. -CanyonOS Core is the product name and `canyonos` is its user-facing CLI. The -internal compatibility Python package, environment variables, and Docker -resources retain the `ventis`, `VENTIS_*`, and `ventis-*` names. These are -protocol identifiers, not CLI instructions or branding strings. Do not rename -them, and do not tell users to run the obsolete `ventis` CLI. +## Progress -## Port checklist +Copy this checklist into the response and update it while working: -Copy this into your response and check items off as you go. Every step below -maps to one line here. - -``` +```text Port progress: -- [ ] 1. Choose the import root; run prepare.py to create .car/config and .car/app +- [ ] 1. Prepare `.car` - [ ] 2. Survey the copy and choose service boundaries -- [ ] 3. Read adapter.md + manifest.md; write declarations, adapters, workflow; - use the `canyonos config` flow to review deployment choices, then write config -- [ ] 4. validate.py exits 0; report readiness and stop +- [ ] 3. Write adapters, workflow, declarations, and reviewed configuration +- [ ] 4. Gap validation exits 0; report readiness and stop ``` -Do not skip step 4. The porting workflow ends when validation exits 0: -report the files created, warnings and unresolved runtime blockers, then stop. -Never build or deploy as an implicit continuation of the port. - -## References - -Every reference is linked from here and read whole when its trigger fires. What -differs between the groups is the *kind* of trigger. - -**Before you write.** Triggered by the step, not by a symptom: a porter cannot -look up a rule whose violation builds green and fails in a container. Neither is -optional. - -- [references/adapter.md](references/adapter.md) -- choosing the entrypoint, - bridging async, session state. Read before writing into `.car/app`. -- [references/manifest.md](references/manifest.md) -- the agent yaml, the - complete manifest, and how to build a `requirements` list. Read before writing - into `.car/config`. +## 1. Prepare `.car` -**When the target has this shape.** Triggered by a fact about the source or the -deployment, all three knowable at step 1. +Read [references/preparation.md](references/preparation.md) in full. Choose the +import root from actual imports and use `prepare.py`; do not assemble or refresh +`.car` manually. After preparation, edit only `.car`. -- [references/packaging.md](references/packaging.md) -- read when a source - import does not resolve from `/app`, the source is nested, packaging metadata - is involved, or the source reads non-Python files at runtime. -- [references/source-survey.md](references/source-survey.md) -- read after - preparing the copy and before choosing service boundaries. -- [references/refresh.md](references/refresh.md) -- read when `.car/app` already - exists and the source has changed; preserve port edits while refreshing it. -- [references/llm-proxy.md](references/llm-proxy.md) -- read when the target - includes `llm_proxy`. -- [references/ec2.md](references/ec2.md) -- read when any config entry uses - `provider: EC2`. +## 2. Survey and design -**After something failed.** Triggered by a symptom. +Read [references/source-survey.md](references/source-survey.md) in full. Produce +its survey record and choose the smallest useful service map before writing +runtime code or configuration. -- [references/troubleshooting.md](references/troubleshooting.md) -- read after an - explicitly approved deploy fails during build, startup, or a request; - symptom-to-cause tables. -- [references/runtime-contract.md](references/runtime-contract.md) -- read when - a validator finding needs explanation or the runtime mechanism is unclear. +## 3. Implement the port -**For orientation.** - -- [references/example-port.md](references/example-port.md) -- one LangGraph port - end to end: the decisions, the files, and the evidence that closed it. - -## Goal: a self-contained `.car`, and a source tree that never learns about it - -The port lives entirely inside `.car/`, next to the application source and -never inside it: - -```text -.car/config/global_controller.yaml deployment manifest -.car/config/policy.yaml optional access restriction -.car/config/.yaml one callable surface per service -.car/app/ a copy of the application source -.car/app//.py the adapter, written where the code it wraps lives -.car/app//_workflow.py HTTP entry point; calls deploy() -.car/app/pyproject.toml conditional nested-import scaffolding -/ the developer's tree, untouched and unaware -``` - -`.car` has exactly two authored directories: `config/`, which holds every -declaration Canyon owns, and `app/`, the copy that becomes `/app` in every -container. Running `prepare.py` (step 1) creates both; do not assemble them by -hand. The container keeps the directory structure the application already had. -Write adapters into that copy, in the module the code they wrap already lives -in -- not into new `agents/` and `workflow/` directories. `canyonos` commands -run from the application root and read `.car` below it. - -Nothing under `.car` points back out at the application source, and nothing in -the application source points at `.car`. Deleting `.car` returns the project to -exactly where it started; regenerating it touches no file the developer owns. - -The file count follows the deployment. A multi-agent port has one yaml/adapter -pair per service that is worth deploying separately. If a source class in the -copy already satisfies the runtime contract, point its config entry at that -file and do not write an adapter beside it. - -Everything the source already owns—prompts, tools, schemas, parsing, retries, -model clients, and node bodies—is imported from where the copy keeps it. The -port re-expresses only the CanyonOS Core boundary and framework-owned -orchestration. - -## 1. Prepare the artifact tree, then survey it - -**Choose the source's import root, which is not always its repository root.** -`/app` is the copy, and without the editable-install capability it is the only -entry on `sys.path`, so a source under `src/` that imports `from tools import -...` needs the *contents* of `src/` at `.car/app/`. Read the source's own -imports, not its directory names, to decide. Getting it wrong builds green and -answers `No agent loaded` on the first request; -[references/packaging.md](references/packaging.md) works the case through. - -Once the import root is known, use the skill's preparation script rather than -assembling `.car` with ad hoc copy commands: - -```bash -python3 /prepare.py .car -``` - -The script creates `.car/config/` and copies the import root's **contents** into -`.car/app/`, preserving its structure. It excludes version-control data, -`.car`, virtualenvs, caches, build outputs, bytecode, and credential-bearing -`.env*` files while retaining `.env.example`, `.env.sample`, and -`.env.template`. It rejects symbolic links because they either escape the -self-contained artifact or are skipped by the runtime source sweep. - -If `.car/app` already exists, read [references/refresh.md](references/refresh.md) -and use `--refresh`. It updates source-owned files while preserving port edits, -and stops atomically when both sides changed one path. Use `--force` only to -discard every edit in `.car/app`; it leaves `.car/config/` unchanged. - -Choosing the import root remains a porter decision; the script standardizes -only directory creation and copying. After it runs, every edit is inside -`.car`. The application source outside it is read-only for the rest of the port --- `git status` at the end shows `.car/` and nothing else. - -Read [references/source-survey.md](references/source-survey.md), survey the -copy, and run the validator. The survey determines the source facts used in the -next two steps; do not infer them from framework conventions. - -## 2. Choose service boundaries - -Start with one service. Split only when it creates independent parallel work or -a distinct resource/replica profile. - -- Keep a ReAct loop together; every turn needs shared message history. -- Hoist supervisor task lists and `Send`-style fan-out into the workflow. -- Do not create a one-replica service with no distinct resource profile merely - to mirror every source graph node. - -Rewrite framework-owned edges as ordinary Python **only where they cross a -service boundary you chose**. A graph whose nodes all land in one agent has no -boundary to express: keep `graph.compile().invoke(...)` and wrap it. Rewriting -it anyway restates control flow the source already had working and buys no -deployment. Import the connected node functions unchanged wherever you do -rewrite. - -Construct runtime-injected service objects from source configuration; do not -invent models, dimensions, stores, or defaults silently. Report any choice the -source does not specify. A service object that holds state across requests -- a -vector store, a memory, a checkpointer built in `__init__` -- makes -`replicas: 1` a correctness requirement rather than a sizing choice, because -the controller picks a replica per call and the others cannot see that state. -Say so in the report; do not leave it implied. - -## 3. Write declarations and adapters - -Read [references/adapter.md](references/adapter.md) before writing an adapter, -and [references/manifest.md](references/manifest.md) before writing -`.car/config`. Neither is optional and neither is triggered by a symptom: every -rule in them builds green and fails inside a container. - -For each service, keep these names aligned: +Read [references/adapter.md](references/adapter.md) before changing `.car/app` +and [references/manifest.md](references/manifest.md) before changing +`.car/config`. Keep this binding exact: ```text config entry name == yaml agent.name == entrypoint class name ``` -Write a no-argument, synchronous adapter class at the entrypoint selected by -`adapter.md`. Import source-owned behavior instead of duplicating it. Expose -`main(query: str)` in the workflow, import every service from its exact -entrypoint module, and call `deploy(main, port=...)` at module scope. - -For parallel remote calls, dispatch before resolving: - -```python -futures = [agent.work(item=item) for item in items] -results = [json.loads(future.value()) for future in futures] -``` - -Do not fuse dispatch and `.value()` in one comprehension. Do not add a main -guard; the workflow executes as `__main__` in production. - -Build declarations and per-image requirements from the copied import graph. -Then use the View/Change flow in `manifest.md` (and `canyonos config` when -interactive) to review developer-owned deployment choices. Write only the -reviewed candidate and rerun validation. - -## Source-integrity boundary - -The validator owns mechanical runtime rules; do not duplicate its check list in -the prompt. The porter owns the rules static analysis cannot prove: - -- Never edit outside `.car` or copy source-owned prompts, tools, schemas, model - calls, and node bodies into an adapter. -- Never swap the source provider, invent runtime configuration, or silently - move, drop, or reclassify a dependency. -- Rewrite framework control flow only where it crosses a service boundary; - preserve it inside a service. -- Never hardcode or bake a real credential into `.car`. - -When a source defect or unsupported runtime capability requires breaking one of -these boundaries, report the blocker and obtain approval for that specific -change. Do not broaden approval to unrelated source edits. - -## 4. Validate and stop - -Run static preflight from the application root: - -```bash -python3 /validate.py .car -``` - -Fix every ERROR and re-run until it exits 0. Warnings and capability -limitations are not permission to hide risk: list each one in the handoff and -say whether it blocks this source. Confirm that `git status` outside `.car` -shows no change to a file the developer owns. +Use the View/Change configuration flow in `manifest.md`; prefer +`canyonos config` in an interactive terminal. Preserve the source-integrity +boundary defined in `preparation.md`. -At that point, report that the `.car` port is validated and stop. Ask the user a -direct yes/no question before taking the next step: +Read these only when triggered: -> Validation passed. Run `canyonos deploy` now? This will build images and start -> the deployment. +- [references/llm-proxy.md](references/llm-proxy.md) when the target uses + `llm_proxy`. +- [references/ec2.md](references/ec2.md) when any entry uses `provider: EC2`. -Do not run a standalone build first. `canyonos deploy` owns both build and -deployment, and must run only after explicit user approval. Silence, an -unattended run, or the original request to "port" is not approval. +## 4. Gap validation and stop -If the user approves, run from the application root: +Read +[references/validation-and-deploy.md](references/validation-and-deploy.md). +Validate only authored contracts that CanyonOS tooling does not strongly +guarantee. Fix every reported error, hand off warnings and blockers, then stop. +Do not run `canyonos deploy` without explicit user approval. -```bash -canyonos deploy -``` +## Diagnose an approved deployment -Do not add build, probe, deployment-debugging, or cleanup work to this skill's -porting flow. +After an explicitly approved deployment fails, start with +[references/troubleshooting.md](references/troubleshooting.md). Read +[references/runtime-contract.md](references/runtime-contract.md) only when a +runtime mechanism or validator finding needs explanation. diff --git a/.claude/skills/porting-to-canyonos/references/adapter.md b/.claude/skills/porting-to-canyonos/references/adapter.md index 606e4ba..da4417e 100644 --- a/.claude/skills/porting-to-canyonos/references/adapter.md +++ b/.claude/skills/porting-to-canyonos/references/adapter.md @@ -1,15 +1,48 @@ -# Writing what goes into `.car/app` +# Implement runtime code in `.car/app` -Read this before writing any adapter. Every rule here can pass static build -checks and fail only when a container loads -- which is why the trigger is the -step, not a symptom. +**When:** after selecting service boundaries, before writing an adapter or +workflow. + +**Output:** one loadable adapter per service and one workflow exposing +`main(query: str)`. + +Use this order: + +1. Choose a safe entrypoint module for each service. +2. Write a no-argument synchronous adapter around source-owned behavior. +3. Bridge async or session state only when the source requires it. +4. Write the workflow and preserve parallel dispatch. + +Complete `manifest.md`, then validate only the authored contracts that CanyonOS +does not already guarantee. + +These rules are required even when static checks pass; violations often appear +only when a container loads. ## Contents +- Adapter and workflow shape - Choosing the entrypoint - Bridging async - Multi-turn and session state +## Adapter and workflow shape + +Write a no-argument, synchronous adapter class at the selected entrypoint. +Import source-owned behavior instead of duplicating it. The workflow exposes +`main(query: str)`, imports every service from its exact entrypoint module, and +calls `deploy(main, port=...)` at module scope. Do not add a main guard: the +workflow executes as `__main__` in production. + +For parallel remote calls, dispatch all work before resolving any result: + +```python +futures = [agent.work(item=item) for item in items] +results = [json.loads(future.value()) for future in futures] +``` + +Combining dispatch and `.value()` in one comprehension serializes the work. + ## Choosing the entrypoint The entrypoint is the one module in the copy the build destroys: each agent's @@ -45,9 +78,9 @@ another. 5. **Does module-level code perform a real run?** A script ending in `result = crew.kickoff(...)` / `print(result)` fires that run whenever the module loads, before a request exists. Delete the - invocation and keep the construction. SKILL.md's source-integrity boundary - protects prompts, tools, schemas, model calls and node bodies -- not a - script's own main body. + invocation and keep the construction. The source-integrity boundary in + `preparation.md` protects prompts, tools, schemas, model calls, and node + bodies—not a script's own main body. ## Bridging async diff --git a/.claude/skills/porting-to-canyonos/references/ec2.md b/.claude/skills/porting-to-canyonos/references/ec2.md index 486049d..6dbdfe2 100644 --- a/.claude/skills/porting-to-canyonos/references/ec2.md +++ b/.claude/skills/porting-to-canyonos/references/ec2.md @@ -1,6 +1,9 @@ -# EC2 deployment +# Configure EC2 deployment -Read this only when at least one config entry uses `provider: EC2`. +**When:** at least one config entry uses `provider: EC2`. + +**Output:** developer-supplied EC2 settings, reachable service addresses, and a +safe remote cleanup plan. ## Configuration @@ -9,8 +12,8 @@ top-level `ec2` block supplies the runtime's required infrastructure and SSH settings. Read the target checkout's deploy preflight and EC2 runtime before writing the block; do not copy values from an example environment. -These identifiers come from the developer, in step 3's config round -- they are -the one part of the manifest with no safe default. If the round produces no +These identifiers come from the developer during the configuration review; +they are the one part of the manifest with no safe default. If the round produces no answer, leave the entry `provider: local` and report that EC2 was requested but not configured. Never fill the block from an example, a previous port, or another entry in the same manifest. diff --git a/.claude/skills/porting-to-canyonos/references/example-port.md b/.claude/skills/porting-to-canyonos/references/example-port.md deleted file mode 100644 index 4da2d00..0000000 --- a/.claude/skills/porting-to-canyonos/references/example-port.md +++ /dev/null @@ -1,132 +0,0 @@ -# One port, end to end - -A LangGraph email assistant, ported and validated, then deployed with explicit -approval. Read this for the shape of the decisions; the rules themselves are in -SKILL.md. - -## Contents - -- The source -- Decision 1: where to root the copy -- Decision 2: one agent, two methods -- The files -- The evidence - -## The source - -A single-file LangGraph app under `src/`, plus its own packages: - -```text -src/email_assistant.py triage_router + a ReAct loop (llm_call/tool_node) -src/prompts.py src/schemas.py src/utils.py src/tools/ -pyproject.toml package-dir = {"" = "src"} -.env OPENAI_API_KEY -``` - -Two graphs. Outer: `START -> triage_router -> (END | response_agent)`, routed by -a `Command(goto=...)`. Inner: `llm_call -> should_continue -> tool_node`, looping -until the model calls `Done`. - -## Decision 1: where to root the copy - -`email_assistant.py` imports `from tools import ...` and `from prompts import -...`, and `pyproject.toml` says `package-dir = {"" = "src"}`. So the import root -is `src/`, not the repository root: - -```bash -python3 /prepare.py src .car -``` - -This creates `.car/config/` and copies the contents of `src/` into `.car/app/` -with the standard source and credential exclusions. - -Copying the repository root instead puts those modules at `/app/src/tools` while -`/app` is the only entry on `sys.path`. The build stays green, the replica -reports healthy, and the first request answers `No agent loaded` with -`No module named 'tools'` in the container log. The validator reports this as -V031 before any of that happens. - -`pyproject.toml` was left out of the copy on purpose: this runtime runs no -editable install, and its `package-dir = {"" = "src"}` is false of a copy that -is already rooted at `src/`. - -## Decision 2: one agent, two methods - -The outer graph is framework control flow, so it became an `if` in the workflow. -The inner ReAct loop stayed inside one agent method: every turn needs the whole -message history, so splitting `llm_call` from `tool_node` would push a growing -message list through Redis for no parallelism. - -The adapter is appended to the bottom of the copied `email_assistant.py`, so it -calls `triage_router`, `llm_call`, `should_continue` and `tool_node` as -module-level names. No prompt, tool, schema or model call is restated. - -```python -class EmailAgent: - def __init__(self): - self.recursion_limit = int(os.environ.get("VENTIS_RECURSION_LIMIT", "25")) - - def triage(self, email_input: dict) -> dict: - command = triage_router({"email_input": email_input, "messages": []}) - update = command.update or {} - return {"goto": command.goto, **update} - - def respond(self, messages: list) -> dict: - state = {"messages": add_messages([], messages)} - for _ in range(self.recursion_limit): - state["messages"] = add_messages(state["messages"], llm_call(state)["messages"]) - if should_continue(state) != "Action": - return {"messages": messages_to_dict(state["messages"])} - state["messages"] = add_messages(state["messages"], tool_node(state)["messages"]) - raise RuntimeError(f"agent did not call Done within {self.recursion_limit} turns") -``` - -`Command` and LangChain message objects are framework types, so they are -unpacked and serialized before they cross the boundary. - -## The files - -```text -.car/config/global_controller.yaml EmailAgent + Workflow, env_file: .env -.car/config/email_agent.yaml triage(email_input: dict), respond(messages: list) -.car/app/email_assistant.py source + the adapter above -.car/app/email_workflow.py the outer graph as an if; deploy(main, port=8080) -.car/app/prompts.py schemas.py utils.py tools/ untouched copies -``` - -`entrypoint: email_assistant.py` and `workflow_file: email_workflow.py`, both -relative to `.car/app`. The workflow imports the agent from its entrypoint -- -`from email_assistant import EmailAgent` -- which is the one module the build -replaces with a stub. - -The platform sends `{query: string}` only, so the four email fields ride inside -`query` as JSON and the workflow unpacks them. The adapter returns a dict; the -runtime encodes it once for transport, so the workflow decodes the Future once -and returns an ordinary dict without another `json.dumps`: - -```python -def main(query: str) -> dict: - email = json.loads(query) - triage = json.loads(agent.triage(email_input=email).value()) - if triage["goto"] == "END": - return triage - return json.loads(agent.respond(messages=triage["messages"]).value()) -``` - -`GET /status/` hands that result back under `result`. - -## The evidence - -```text -validate.py .car 0 errors; porting workflow stopped -user approval yes, run deployment -canyonos deploy build complete; 2 replicas ready -POST /main 202 {"request_id": ...} -GET /status/ status: error, 401 from OpenAI -``` - -The last line is the interesting one. The agent log showed the request arriving -over gRPC (`route_to: :8000`, `function: triage`), the agent loading, and the -source's own model call returning 401 on an expired key. A source-level failure -behind a working boundary still closes the port: record it as such rather than -calling the port broken. diff --git a/.claude/skills/porting-to-canyonos/references/llm-proxy.md b/.claude/skills/porting-to-canyonos/references/llm-proxy.md index d2047bb..66d76db 100644 --- a/.claude/skills/porting-to-canyonos/references/llm-proxy.md +++ b/.claude/skills/porting-to-canyonos/references/llm-proxy.md @@ -1,7 +1,10 @@ -# LLM proxy integration +# Route model calls through `llm_proxy` -Read this only when the target checkout contains `llm_proxy` or the deployment -explicitly routes model SDKs through it. +**When:** the target contains `llm_proxy` or deployment explicitly routes model +SDKs through it. + +**Output:** provider-preserving proxy environment settings, verified routing, +and a clear blocker for unsupported call shapes. ## Contents @@ -58,8 +61,9 @@ credentials in the separate proxy process, not in the port's `env_file`. Some sources build the HTTP call themselves -- `urllib.request` against a module constant like `API = "https://api.openai.com/v1/responses"` -- and read no base-URL variable at all. Editing that constant swaps the source provider's -endpoint, which SKILL.md's source-integrity boundary forbids, so the `env_file` -is inert and the container can only ever reach the real provider. Report this as a proxy blocker and stop. Do not hand the +endpoint, which the source-integrity boundary in +[preparation.md](preparation.md) forbids, so the `env_file` is inert and the container can only ever reach the +real provider. Report this as a proxy blocker and stop. Do not hand the container a real upstream credential instead. Detect it before deploying: grep the source for the provider hostname. A literal diff --git a/.claude/skills/porting-to-canyonos/references/manifest.md b/.claude/skills/porting-to-canyonos/references/manifest.md index e90188b..0eb00ac 100644 --- a/.claude/skills/porting-to-canyonos/references/manifest.md +++ b/.claude/skills/porting-to-canyonos/references/manifest.md @@ -1,18 +1,29 @@ -# Writing what goes into `.car/config` +# Configure `.car/config` -Read this before writing the manifest or an agent declaration. The validator -checks YAML structure and the public artifact contract before an approved -`canyonos deploy`; this reference explains how to derive the values inside it. +**When:** before writing an agent declaration or the deployment manifest. + +**Output:** one declaration per service and one reviewed +`global_controller.yaml`, with requirements derived separately for each image. + +Work in this order: + +1. Derive names, entrypoints, workflow path, and requirements from the service + map and copied import graph. +2. Write each agent declaration. +3. Build the complete manifest candidate with documented defaults. +4. Review developer-owned choices through the View/Change flow. +5. Write the reviewed candidate; validate only cross-file contracts not already + guaranteed by the config flow. ## Contents -- Who decides each key -- Review configuration through the CanyonOS CLI flow -- Agent yaml -- Requirements -- The manifest, in full +- Ownership of configuration keys +- Configuration review +- Agent declarations +- Per-image requirements +- Complete manifest shape -## Who decides each key +## Ownership of configuration keys Two kinds of key share one file. A **derived** key has exactly one right answer and the copy holds it; asking the developer can only make it worse. A @@ -24,7 +35,7 @@ column only, in one round, carrying these defaults. | Key | Decided by | Default when unanswered | |---|---|---| -| `name`, `entrypoint`, `workflow_file`, `type` | derived — service boundaries, step 2 | — | +| `name`, `entrypoint`, `workflow_file`, `type` | derived — selected service map | — | | `requirements` | derived — the entry's import graph | — | | `database` | neither; omit it always (see below) | absent | | `provider` | developer | `local` | @@ -41,15 +52,15 @@ Two entries in that table are not free choices, and saying so is part of showing the config rather than asking about it: - **`replicas` stops being a choice once a service holds cross-request state.** - Where the step-2 survey found such state, SKILL.md already fixes `replicas: 1` - as a correctness requirement, so `1` is derived: report it as a constraint and - do not offer to raise it. + When the survey finds such state, the service-boundary section in + `source-survey.md` fixes `replicas: 1` as a correctness requirement. Report it as a constraint; do not offer to + raise it. - **EC2 identifiers are wrong to invent.** ec2.md forbids copying them from an example environment, and a wrong AMI, subnet, or security group fails at deploy preflight or, worse, provisions something unreachable. Unanswered means the entry stays `local`. -## Review configuration through the CanyonOS CLI flow +## Configuration review Use the interaction implemented by `canyonos config` before writing `.car/config/global_controller.yaml`: @@ -62,7 +73,8 @@ Use the interaction implemented by `canyonos config` before writing EC2 fields, unconstrained replicas, resources, ports, secret-file location, and access restrictions. Show each current/default value, apply answers, and show the result. -4. Write the reviewed candidate and run the validator. +4. Write the reviewed candidate. Defer gap validation until runtime code and + configuration are both complete. Prefer running `canyonos config` when an interactive terminal is available; otherwise reproduce View/Change in conversation. Do not ask for derived values @@ -72,7 +84,7 @@ An unattended `canyonos integrate` run must not block on this interaction. Use and report the displayed defaults. Never invent EC2 infrastructure identifiers: without them, keep the entry `local`. -## Agent yaml +## Agent declarations Declarations go in `.car/config/`, beside the manifest. The build reads every `*.yaml` there and keeps the ones with a top-level `agent.name`, so the @@ -83,7 +95,7 @@ Use one yaml per deployed service. Argument types are bare builtins only: required by the generated stub. `returns.type` is documentation; use `dict` or `list` to signal that workflow callers must `json.loads` the returned string. -## Requirements +## Per-image requirements Each image installs the runtime's base list plus that entry's `requirements:` and nothing else. The source's own `requirements.txt` is never installed -- the @@ -134,7 +146,7 @@ transitive pin still requires) is a port blocker; one command finds it instead of one build-fail/pin/rebuild cycle per attempt. Report it and stop rather than upgrading the source out of the problem. -## The manifest, in full +## Complete manifest shape `.car/config/global_controller.yaml` in full -- every key the runtime reads, and no others: @@ -176,5 +188,5 @@ env_file: .env # relative to the application root, not .car Omit `database`. Without it every metrics poll logs `Could not parse SQLAlchemy URL from given URL string`, once per replica every `poll_interval` seconds -- expected noise, not a failure, and not a reason to add the key. Adding it drops -a sqlite file at the application root, outside `.car`, which the step-8 -`git status` check then fails on. +a sqlite file at the application root, outside `.car`, which the final source- +integrity check then reports. diff --git a/.claude/skills/porting-to-canyonos/references/packaging.md b/.claude/skills/porting-to-canyonos/references/packaging.md deleted file mode 100644 index e87c002..0000000 --- a/.claude/skills/porting-to-canyonos/references/packaging.md +++ /dev/null @@ -1,137 +0,0 @@ -# Packaging and import roots - -Read this reference when an adapter imports nested source code, the source uses a -`src/` layout, V031 reports an import-root problem, or the source reads -non-Python files at runtime. - -## Contents - -- What `/app` can import -- Re-root the copy before reaching for metadata -- Detect support, do not infer it from release history -- Root metadata is the trigger -- Dependencies in nested metadata -- Runtime data and configuration files -- Validation boundary - -## What `/app` can import - -CanyonOS Core copies `.car/app/` into the image with its paths intact and -starts Python at `/app`, so `/app` is that copy. Without an editable install, -Python resolves names rooted there: - -- `/app/tools.py` as `import tools` -- `/app/pkg/__init__.py` as `import pkg` -- `/app/src/agents/...` as `import src.agents`, including PEP 420 namespace - directories without `__init__.py` - -It does not resolve `/app/source/pkg` as `import pkg`; `/app/source` must become -an import root first. - -## Re-root the copy before reaching for metadata - -`.car/app/` is a copy Canyon owns, so the cheapest fix is usually to root it -where the source already imports from. A project laid out as - -```text -repo/src/email_assistant.py imports `tools`, `prompts`, `utils` -repo/src/tools/ -repo/pyproject.toml -``` - -has `src/` as its import root. Copy `src/`'s contents to `.car/app/` and every -one of those imports resolves from `/app` with no metadata, no editable install -and no `sys.path` hack. `entrypoint` and `workflow_file` then name modules -relative to that root, and the workflow imports the agent the same way. - -Reach for the metadata below only when one copy root cannot serve every import --- for instance when the source imports both `tools` and `src.tools`. - -## Detect support, do not infer it from release history - -Run: - -```bash -python3 /validate.py .car -``` - -Read the `editable_install` capability. If it is unavailable and the original -import cannot resolve from `/app`, report a runtime capability blocker and stop. -Do not add a `sys.path` hack or relocate source files. - -## Root metadata is the trigger - -When editable install is supported, only packaging metadata at the **root of -the copy** triggers `pip install -e .`: - -```text -.car/app/pyproject.toml detected -.car/app/source/pyproject.toml ignored as an install trigger -``` - -If the application keeps its metadata deeper in the tree, that copy stays where -it is. Add minimal scaffolding at `.car/app/` that points package discovery at -the existing package: - -```toml -[build-system] -requires = ["setuptools>=64"] -build-backend = "setuptools.build_meta" - -[project] -name = "canyonos-port" -version = "0.0.0" -dependencies = [] - -[tool.setuptools.packages.find] -where = ["source/src"] -include = ["pkg*"] -namespaces = true -``` - -Set `where` and `include` from the actual tree and original import spelling. Do -not reference a README or license from this wrapper metadata; file sweeps differ -by runtime capability and a missing referenced file makes the image build fail. - -## Dependencies in nested metadata - -A nested `pyproject.toml` is not installed merely because its Python files are -copied. Keep the source declaration unchanged and repeat its runtime -distributions in each relevant config entry's `requirements` list. This is -compatibility scaffolding, not permission to drop, move, or reclassify declared -dependencies. - -If the application's own metadata already sits at the root of the copy, do not -create a wrapper. Its project dependencies participate in the same resolver as -config requirements. -Report declared-but-unused toolchain dependencies and their image cost; let the -owner decide whether source metadata should change. - -## Runtime data and configuration files - -`prepare.py` copies non-Python files into `.car/app`, but that does not prove the -runtime's image sweep carries them into a container. Inventory every file opened -by the selected import graph: prompt templates, JSON schemas, PDFs, local -corpora, certificates, and framework configuration such as CrewAI -`agents.yaml` and `tasks.yaml`. - -Run `validate.py` and read its `sweeps_all_files` capability: - -- When available, retain each asset at the same path relative to the chosen - import root. Check any path derived from the original repository root or - process working directory; the container starts from `/app`. -- When unavailable, a required non-Python asset is a runtime blocker. Report it - and stop after validation. Do not conceal the gap by base64-encoding the file - into Python, changing a hardcoded path, or duplicating framework config into - adapter code; those changes restate source-owned data and behavior. - -Do not treat successful construction as evidence that configuration loaded. -Frameworks such as CrewAI may warn about a missing yaml and create an empty -configuration, then fail only when the first agent or task is accessed. Inspect -those decorators and file references statically during the survey. - -## Validation boundary - -The build phase of `canyonos deploy` owns packaging syntax and installation -errors. `validate.py` checks only whether adapter imports appear to require a -nested root that the runtime will not expose. diff --git a/.claude/skills/porting-to-canyonos/references/preparation.md b/.claude/skills/porting-to-canyonos/references/preparation.md new file mode 100644 index 0000000..c49eb8e --- /dev/null +++ b/.claude/skills/porting-to-canyonos/references/preparation.md @@ -0,0 +1,263 @@ +# Prepare the `.car` artifact + +**When:** before creating or refreshing `.car`, or when imports and runtime +assets need packaging decisions. + +**Output:** a self-contained `.car/config` and `.car/app`, with the original +application tree untouched and existing imports preserved. + +Use this order: + +1. Read **Artifact model and preparation** and choose the import root from actual + imports. +2. Run `prepare.py`; do not recreate guarantees already enforced by the script. +3. Use **Import roots, metadata, and runtime assets** only when direct `/app` + imports are insufficient or the source opens non-Python files. +4. Use **Refresh an existing source copy** only when `.car/app` already exists. + +## Artifact model and preparation + +### Product and runtime names + +CanyonOS Core is the product name and `canyonos` is its user-facing CLI. The +internal compatibility Python package, environment variables, and Docker +resources retain the `ventis`, `VENTIS_*`, and `ventis-*` names. These are +protocol identifiers, not CLI instructions or branding strings. Do not rename +them, and do not tell users to run the obsolete `ventis` CLI. + +### Artifact boundary + +The port lives entirely inside `.car/`, next to the application source: + +```text +.car/config/global_controller.yaml deployment manifest +.car/config/policy.yaml optional access restriction +.car/config/.yaml one callable surface per service +.car/app/ a copy of the application source +.car/app//.py adapter beside the code it wraps +.car/app//_workflow.py HTTP entry point; calls deploy() +.car/app/pyproject.toml conditional nested-import scaffolding +/ untouched and unaware of the port +``` + +`.car` has exactly two authored directories: `config/`, which holds every +Canyon-owned declaration, and `app/`, which becomes `/app` in every container. +Nothing under `.car` points back into the original source, and nothing in the +original source points at `.car`. Deleting `.car` must restore the project to +its pre-port state. + +Preserve the source's directory structure. Put adapters in the copied module +whose behavior they wrap unless the entrypoint rules require a sibling module; +do not invent generic `agents/` or `workflow/` directories. `canyonos` commands +run from the application root and read `.car` below it. + +The file count follows the deployment: one yaml/adapter pair per independently +deployed service. If a copied source class already satisfies the runtime +contract, point its declaration at that class and do not add an adapter. + +### Prepare the copy + +Choose the source's **import root**, not automatically its repository root. +Without editable-install support, `/app` is the only source entry on +`sys.path`. For example, source under `src/` that says `from tools import ...` +needs the contents of `src/` copied directly into `.car/app/`. Decide from the +source's imports. Continue to **Import roots, metadata, and runtime assets** below when the +copy has those concerns. + +Create the artifact with the skill script rather than ad hoc copy commands: + +```bash +python3 /prepare.py .car +``` + +The script creates `.car/config/` and copies the import root's **contents** to +`.car/app/`. It excludes VCS data, `.car`, virtual environments, caches, build +outputs, bytecode, and credential-bearing `.env*` files while retaining +`.env.example`, `.env.sample`, and `.env.template`. It rejects symbolic links: +they can escape the artifact and may be skipped by runtime source sweeps. + +If `.car/app` already exists, follow **Refresh an existing source copy** below. Use +`--force` only when every edit in `.car/app` may be discarded; it leaves +`.car/config/` unchanged. + +After preparation, edit only `.car` and survey the copy using +[source-survey.md](source-survey.md). Do not add validation for preparation +postconditions that `prepare.py` already guarantees. + +### Source-integrity boundary + +The gap validator owns authored runtime contracts that CanyonOS tooling does +not strongly guarantee. The porter owns constraints static analysis cannot +prove: + +- Never edit outside `.car`, or duplicate source-owned prompts, tools, schemas, + model calls, parsing, retries, and node bodies in an adapter. +- Never swap providers, invent runtime configuration, or silently move, drop, + or reclassify a dependency. +- Rewrite framework control flow only where it crosses a chosen service + boundary; preserve it inside a service. +- Never hardcode or bake a real credential into `.car`. + +When a source defect or unsupported runtime capability requires crossing one of +these boundaries, report the blocker and obtain approval for that specific +change. Do not broaden that approval to unrelated source edits. + +## Import roots, metadata, and runtime assets + +### What `/app` can import + +CanyonOS Core copies `.car/app/` into the image with its paths intact and +starts Python at `/app`, so `/app` is that copy. Without an editable install, +Python resolves names rooted there: + +- `/app/tools.py` as `import tools` +- `/app/pkg/__init__.py` as `import pkg` +- `/app/src/agents/...` as `import src.agents`, including PEP 420 namespace + directories without `__init__.py` + +It does not resolve `/app/source/pkg` as `import pkg`; `/app/source` must become +an import root first. + +### Re-root the copy before reaching for metadata + +`.car/app/` is a copy Canyon owns, so the cheapest fix is usually to root it +where the source already imports from. A project laid out as + +```text +repo/src/email_assistant.py imports `tools`, `prompts`, `utils` +repo/src/tools/ +repo/pyproject.toml +``` + +has `src/` as its import root. Copy `src/`'s contents to `.car/app/` and every +one of those imports resolves from `/app` with no metadata, no editable install +and no `sys.path` hack. `entrypoint` and `workflow_file` then name modules +relative to that root, and the workflow imports the agent the same way. + +Reach for the metadata below only when one copy root cannot serve every import +-- for instance when the source imports both `tools` and `src.tools`. + +### Detect support, do not infer it from release history + +Run: + +```bash +python3 /validate.py .car +``` + +Read the `editable_install` capability. If it is unavailable and the original +import cannot resolve from `/app`, report a runtime capability blocker and stop. +Do not add a `sys.path` hack or relocate source files. + +### Root metadata is the trigger + +When editable install is supported, only packaging metadata at the **root of +the copy** triggers `pip install -e .`: + +```text +.car/app/pyproject.toml detected +.car/app/source/pyproject.toml ignored as an install trigger +``` + +If the application keeps its metadata deeper in the tree, that copy stays where +it is. Add minimal scaffolding at `.car/app/` that points package discovery at +the existing package: + +```toml +[build-system] +requires = ["setuptools>=64"] +build-backend = "setuptools.build_meta" + +[project] +name = "canyonos-port" +version = "0.0.0" +dependencies = [] + +[tool.setuptools.packages.find] +where = ["source/src"] +include = ["pkg*"] +namespaces = true +``` + +Set `where` and `include` from the actual tree and original import spelling. Do +not reference a README or license from this wrapper metadata; file sweeps differ +by runtime capability and a missing referenced file makes the image build fail. + +### Dependencies in nested metadata + +A nested `pyproject.toml` is not installed merely because its Python files are +copied. Keep the source declaration unchanged and repeat its runtime +distributions in each relevant config entry's `requirements` list. This is +compatibility scaffolding, not permission to drop, move, or reclassify declared +dependencies. + +If the application's own metadata already sits at the root of the copy, do not +create a wrapper. Its project dependencies participate in the same resolver as +config requirements. +Report declared-but-unused toolchain dependencies and their image cost; let the +owner decide whether source metadata should change. + +### Runtime data and configuration files + +`prepare.py` copies non-Python files into `.car/app`, but that does not prove the +runtime's image sweep carries them into a container. Inventory every file opened +by the selected import graph: prompt templates, JSON schemas, PDFs, local +corpora, certificates, and framework configuration such as CrewAI +`agents.yaml` and `tasks.yaml`. + +Run `validate.py` and read its `sweeps_all_files` capability: + +- When available, retain each asset at the same path relative to the chosen + import root. Check any path derived from the original repository root or + process working directory; the container starts from `/app`. +- When unavailable, a required non-Python asset is a runtime blocker. Report it + and stop after validation. Do not conceal the gap by base64-encoding the file + into Python, changing a hardcoded path, or duplicating framework config into + adapter code; those changes restate source-owned data and behavior. + +Do not treat successful construction as evidence that configuration loaded. +Frameworks such as CrewAI may warn about a missing yaml and create an empty +configuration, then fail only when the first agent or task is accessed. Inspect +those decorators and file references statically during the survey. + +### Validation boundary + +The build phase of `canyonos deploy` owns packaging syntax and installation +errors. `validate.py` checks only whether adapter imports appear to require a +nested root that the runtime will not expose. + +## Refresh an existing source copy + +Run the same preparation command with `--refresh` and the same import root: + +```bash +python3 /prepare.py .car --refresh +``` + +The initial copy records source-file hashes in +`.car/config/.porting-state.json`. Refresh compares three states: + +```text +previous source hash → current source + ↘ current .car/app +``` + +- Only the source changed: update `.car/app`. +- Only `.car/app` changed: preserve the port edit. +- The source added a path unused by the port: add it. +- The source deleted an unmodified path: delete it from `.car/app`. +- Both sides changed the same path differently: make no changes and report all + conflicts. + +Resolve a conflict in `.car/app`, then either make the source match that result +or intentionally start over. The script does not guess a merge because an +adapter and its source often change for different reasons while sharing one +module. + +`--force` is not refresh. It discards the entire `.car/app` tree and replaces it +with a clean source copy while retaining `.car/config`. Use it only when every +adapter and workflow edit in `.car/app` is intentionally disposable. + +After refresh, survey changed imports, dependencies, runtime assets, and service +boundaries again. Run gap validation for affected authored contracts, but do not +revalidate merge mechanics already guaranteed by the successful atomic refresh. diff --git a/.claude/skills/porting-to-canyonos/references/refresh.md b/.claude/skills/porting-to-canyonos/references/refresh.md deleted file mode 100644 index eaab51b..0000000 --- a/.claude/skills/porting-to-canyonos/references/refresh.md +++ /dev/null @@ -1,38 +0,0 @@ -# Refreshing an existing port - -Read this when `.car/app` already exists and the application source has changed. - -Run the same preparation command with `--refresh` and the same import root: - -```bash -python3 /prepare.py .car --refresh -``` - -The initial copy records source-file hashes in -`.car/config/.porting-state.json`. Refresh compares three states: - -```text -previous source hash → current source - ↘ current .car/app -``` - -- Only the source changed: update `.car/app`. -- Only `.car/app` changed: preserve the port edit. -- The source added a path unused by the port: add it. -- The source deleted an unmodified path: delete it from `.car/app`. -- Both sides changed the same path differently: make no changes and report all - conflicts. - -Resolve a conflict in `.car/app`, then either make the source match that result -or intentionally start over. The script does not guess a merge because an -adapter and its source often change for different reasons while sharing one -module. - -`--force` is not refresh. It discards the entire `.car/app` tree and replaces it -with a clean source copy while retaining `.car/config`. Use it only when every -adapter and workflow edit in `.car/app` is intentionally disposable. - -After refresh, survey changed imports, dependencies, runtime assets, and service -boundaries again, then run the full validator. A clean file merge is not proof -that the deployment contract still holds. - diff --git a/.claude/skills/porting-to-canyonos/references/runtime-contract.md b/.claude/skills/porting-to-canyonos/references/runtime-contract.md index 9d96aff..49d26d1 100644 --- a/.claude/skills/porting-to-canyonos/references/runtime-contract.md +++ b/.claude/skills/porting-to-canyonos/references/runtime-contract.md @@ -1,13 +1,19 @@ -# CanyonOS Core runtime contract +# Explain the CanyonOS Core runtime contract -The product is CanyonOS Core and its user-facing CLI is `canyonos`. Compatibility -identifiers remain `ventis` for the internal Python package, `VENTIS_*` for -runtime variables, and `ventis-*` for Docker resources. +**When:** a validator finding needs explanation, runtime behavior is unclear, +or an approved deployment fails in a way the troubleshooting table attributes +to the runtime. -Read this reference when implementing an adapter or explaining a validator -finding. Runtime-dependent behavior is expressed as capabilities; run -`validate.py` against the target environment instead of inferring support from -release history. +**Purpose:** explain discovery, generated stubs, loading, execution, image +assembly, and cleanup. This is diagnostic background; implementation rules live +in `adapter.md`, `manifest.md`, and `preparation.md`. + +The product is CanyonOS Core and its user-facing CLI is `canyonos`. +Compatibility identifiers remain `ventis` for the internal Python package, +`VENTIS_*` for runtime variables, and `ventis-*` for Docker resources. +Runtime-dependent behavior is expressed as capabilities; run `validate.py` +against the target environment instead of inferring support from release +history. ## Contents @@ -173,7 +179,7 @@ entrypoint, so the second lands on the first and every caller reaches whichever was built last. The validator checks both collisions. File sweep and editable-install behavior are runtime capabilities. For nested -imports, follow [packaging.md](packaging.md). +imports, follow [preparation.md](preparation.md#import-roots-metadata-and-runtime-assets). ## Dependencies and protobuf diff --git a/.claude/skills/porting-to-canyonos/references/source-survey.md b/.claude/skills/porting-to-canyonos/references/source-survey.md index b44538a..d63485e 100644 --- a/.claude/skills/porting-to-canyonos/references/source-survey.md +++ b/.claude/skills/porting-to-canyonos/references/source-survey.md @@ -1,29 +1,93 @@ -# Surveying the copied source - -Read this after `prepare.py` and before choosing service boundaries. Survey -`.car/app`, not a guessed abstraction of the original repository. - -Identify all of the following: - -1. The production entry point and callable input/output. If several - implementations look plausible, trace imports from the documented route, - CLI, or launch path instead of choosing by filename. -2. Framework-owned control flow: graphs, crews, chats, routing, fan-out, - commands, and interrupts. -3. Runtime-injected stores, context, memory, sessions, and callback managers. -4. Sync/async boundaries and objects tied to an event loop. -5. The transitive import graph and the source's pinned runtime distributions. -6. Model provider, credential names, streaming, and optional `llm_proxy` use. -7. Independent work that benefits from separate resource or replica profiles. -8. Whether imports resolve with `.car/app` as `/app`; read `packaging.md` when - they do not. -9. Non-Python runtime files such as prompts, CrewAI YAML, PDFs, templates, - schemas, and corpora. If the runtime cannot sweep all files, report this as a - blocker; do not embed files or rewrite paths to hide it. -10. Whether every Python file on the selected import graph parses. Existing - syntax errors are source defects; report them and obtain approval before - changing even the copied version. - -Run `python3 /validate.py .car` after the survey and after every -change. Missing or malformed required inputs fail closed. If a required runtime -capability is reported unavailable, stop instead of assuming it exists. +# Survey the copied source + +**When:** after `prepare.py`, before writing adapters or configuration. + +**Inspect:** `.car/app`, not a framework-based guess about the original tree. + +**Output:** a short survey record and the smallest useful service map. Do not +start adapter or config work until each section is resolved or marked as a +blocker. + +## 1. Public behavior + +Record: + +- the production entrypoint and callable input/output; +- the documented route, CLI, or launch path that proves this is the entrypoint; +- prompts, tools, schemas, parsing, retries, model clients, and node bodies that + remain source-owned. + +If multiple implementations look plausible, trace imports from the documented +launch path instead of choosing by filename. + +## 2. Control flow and state + +Record: + +- framework-owned graphs, crews, chats, routing, fan-out, commands, and + interrupts; +- independent work that may justify separate resources or replicas; +- injected stores, context, memory, sessions, checkpointers, and callback + managers; +- sync/async boundaries and objects tied to an event loop. + +This evidence feeds the service-boundary decision below and `adapter.md`. + +## 3. Imports and runtime inputs + +Record: + +- the transitive import graph from the selected entrypoint; +- source lockfiles and pinned runtime distributions; +- whether every import resolves with `.car/app` mounted as `/app`; +- model providers, credential variable names, streaming calls, and optional + `llm_proxy` use; +- non-Python files opened at runtime: prompts, framework YAML, PDFs, templates, + schemas, certificates, and corpora. + +Use the import-root and runtime-assets sections of `preparation.md` when imports +do not resolve, packaging metadata matters, or runtime code reads non-Python +files. + +## 4. Capability questions + +Record only capabilities that affect this source, such as editable installation, +full-file sweeping, or environment-file injection. Inspect the installed runtime +or its capability probe; do not run validation merely to reconfirm properties +already guaranteed by `prepare.py`. + +An existing syntax error on the selected import graph is a source defect; +obtain approval before changing even the copied version. The final gap validator +checks authored runtime code and cross-file bindings after the port is complete. +If a required runtime capability is unavailable, report a blocker instead of +assuming support. + +## 5. Choose service boundaries + +**Output:** the smallest useful service map, including which framework edges +cross services and which services require `replicas: 1`. + +Start with one service. Split only when doing so creates independently parallel +work or a genuinely distinct resource or replica profile. + +- Keep a ReAct loop together; each turn needs shared message history. +- Hoist supervisor task lists and `Send`-style fan-out into the workflow. +- Do not create a one-replica service with no distinct resource profile merely + to mirror every source graph node. + +Rewrite framework-owned edges as ordinary Python **only where they cross a +service boundary**. If every graph node stays in one service, preserve +`graph.compile().invoke(...)` and wrap it. Rewriting internal edges restates +working source behavior without creating a deployment benefit. Where an edge +must move into the workflow, import the connected source node functions +unchanged. + +Construct runtime-injected service objects from source configuration. Never +invent models, embedding dimensions, stores, or defaults silently; report any +choice the source does not specify. + +A service object that holds state across requests—for example a vector store, +memory, or checkpointer created in `__init__`—requires `replicas: 1` for +correctness. The controller can route each call to a different replica, and +those replicas cannot see one another's in-memory state. Record this as a +constraint in the configuration review and handoff, not as a sizing preference. diff --git a/.claude/skills/porting-to-canyonos/references/troubleshooting.md b/.claude/skills/porting-to-canyonos/references/troubleshooting.md index 213163c..ef2faa4 100644 --- a/.claude/skills/porting-to-canyonos/references/troubleshooting.md +++ b/.claude/skills/porting-to-canyonos/references/troubleshooting.md @@ -1,8 +1,11 @@ -# Troubleshooting +# Diagnose an approved deployment failure -Read this after an explicitly approved `canyonos deploy` fails during build, -startup, or a request. For mechanisms, read -[runtime-contract.md](runtime-contract.md). For proxy or remote-host failures, +**When:** an explicitly approved `canyonos deploy` fails during build, startup, +or a request. Do not use this table to extend the pre-deployment porting flow. + +**How:** find the symptom, verify the likely cause, then follow the linked +canonical reference. For runtime mechanisms read +[runtime-contract.md](runtime-contract.md); for proxy or remote-host failures read [llm-proxy.md](llm-proxy.md) or [ec2.md](ec2.md). ## Build or deploy stops early @@ -28,7 +31,7 @@ read [llm-proxy.md](llm-proxy.md) or [ec2.md](ec2.md). | First request says `No agent loaded` | Entrypoint import, class lookup, or constructor failed and the controller swallowed the exception; inspect container logs | | Replica is healthy but serves nothing | Health publication does not prove successful agent loading | | Missing credentials while loading | Env injection is unavailable/misconfigured or the source reads another variable | -| Source module is missing | `.car/app` is not rooted at the source's import root, so the original import does not resolve from `/app`; read [packaging.md](packaging.md) | +| Source module is missing | `.car/app` is not rooted at the source's import root, so the original import does not resolve from `/app`; read [preparation.md](preparation.md#import-roots-metadata-and-runtime-assets) | | Third-party module is missing | Distribution is absent from source metadata and config requirements | | Stub import raises `NameError` | yaml argument type is not a bare builtin | | An agent runs in the workflow process instead of a container | The workflow reached the class by a path other than the agent's `entrypoint`, so it got the real module rather than the stub | diff --git a/.claude/skills/porting-to-canyonos/references/validation-and-deploy.md b/.claude/skills/porting-to-canyonos/references/validation-and-deploy.md new file mode 100644 index 0000000..9f5f515 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/references/validation-and-deploy.md @@ -0,0 +1,70 @@ +# Validate, hand off, and optionally deploy + +**When:** after runtime code and configuration are complete. + +**Output:** a clean validation report and a stopped porting workflow. Deployment +is a separate action that requires explicit approval. + +## Validation scope + +Validate only contracts that authored port files can violate and CanyonOS does +not fail closed on—for example declaration/adapter bindings, generated-stub +imports, workflow call shape, per-image dependency coverage, and +capability-dependent behavior. + +Do not add duplicate checks for postconditions strongly guaranteed by code: + +- a successful `prepare.py` run already creates the artifact directories, + rejects source symlinks, applies exclusions, and installs the copy atomically; +- `canyonos config` owns the structure of configuration it generates; +- deploy preflight owns checks that already fail before resources are changed. + +Treat required inputs such as a readable manifest and `.car/app` as validator +preconditions, not independent port rules. If a guarantee changes in CanyonOS, +change the owning code or capability probe rather than maintaining a parallel +rule in prose and validation. + +## Run the gap validator + +From the application root, run: + +```bash +python3 /validate.py .car +``` + +Fix every `ERROR` and rerun until the command exits 0. Do not hide warnings or +capability limitations: list each in the handoff and state whether it blocks +this source. Confirm with `git status` that no developer-owned file outside +`.car` changed. + +Report: + +- that the `.car` port validated; +- files created; +- validator warnings; +- unresolved runtime blockers; +- intentionally omitted unreachable dependencies or source surfaces. + +Then stop and ask exactly one direct approval question: + +> Validation passed. Run `canyonos deploy` now? This will build images and start +> the deployment. + +Do not treat silence, an unattended run, or the original request to “port” as +approval. + +## Deploy only after approval + +If the user explicitly approves, run from the application root: + +```bash +canyonos deploy +``` + +Do not run a standalone build first: `canyonos deploy` performs both build and +deployment. Do not add probing, deployment debugging, or cleanup to the porting +flow. + +If an approved deploy fails during build, startup, or a request, read +`troubleshooting.md`. Read `runtime-contract.md` when a validator finding or +runtime mechanism needs explanation. diff --git a/.claude/skills/porting-to-canyonos/validate.py b/.claude/skills/porting-to-canyonos/validate.py index 61788e8..91631f7 100755 --- a/.claude/skills/porting-to-canyonos/validate.py +++ b/.claude/skills/porting-to-canyonos/validate.py @@ -43,7 +43,6 @@ check_declaration_bindings, check_manifest_structure, check_policy, - check_self_contained_tree, discover_agent_declarations, ) from validation.packaging import check_env_file, check_import_root @@ -100,7 +99,10 @@ def validate(artifact_dir, config_path, capabilities): ) return report - check_self_contained_tree(report, artifact_dir) + # `prepare.py` already rejects symlinks and creates the artifact layout + # atomically. Do not duplicate postcondition checks that the preparation + # code strongly guarantees; validation focuses on authored cross-file and + # runtime contracts. config_dir = os.path.dirname(config_path) entries = check_manifest_structure(report, config, config_path, source_dir) @@ -255,7 +257,7 @@ def print_report(report, artifact_root): def main(argv=None): parser = argparse.ArgumentParser( - description="Check a CanyonOS Core port against the rules in SKILL.md." + description="Check authored CanyonOS port contracts not guaranteed by tooling." ) parser.add_argument( "artifact_root", diff --git a/.claude/skills/porting-to-canyonos/validation/manifest.py b/.claude/skills/porting-to-canyonos/validation/manifest.py index a69b7d4..dab75a2 100644 --- a/.claude/skills/porting-to-canyonos/validation/manifest.py +++ b/.claude/skills/porting-to-canyonos/validation/manifest.py @@ -173,24 +173,6 @@ def check_manifest_structure(report, config, config_path, source_dir): return entries if valid else None -def check_self_contained_tree(report, artifact_dir): - """Reject symlinks because `.car` must not depend on outside state.""" - for root, directories, files in os.walk(artifact_dir, followlinks=False): - for name in [*directories, *files]: - path = os.path.join(root, name) - if not os.path.islink(path): - continue - report.error( - "V036", - path, - 0, - "`.car` contains a symbolic link", - "A symlink can escape the artifact or be skipped by the " - "Python-file sweep. Copy the intended file or directory into " - "the artifact explicitly.", - ) - - def discover_agent_declarations(report, config_dir, config_path): """Load declarations without silently discarding malformed or duplicate YAML.""" declarations = {}