diff --git a/.claude/skills/porting-to-canyonos-core/SKILL.md b/.claude/skills/porting-to-canyonos-core/SKILL.md deleted file mode 100644 index fe6dd49..0000000 --- a/.claude/skills/porting-to-canyonos-core/SKILL.md +++ /dev/null @@ -1,240 +0,0 @@ ---- -name: porting-to-canyonos-core -description: Ports existing LangChain, LangGraph, CrewAI, AutoGen, and hand-rolled Python agent projects to CanyonOS Core. Use when converting, migrating, adapting, packaging, building, or deploying an existing agent or multi-agent project onto CanyonOS Core. -compatibility: Requires Python, Docker, and the `ventis` compatibility CLI. Runtime identifiers remain `ventis`, `VENTIS_*`, and `ventis-*`. ---- - -# Port an agent project to CanyonOS Core - -CanyonOS Core is the product name. Its compatibility executable and Python -package remain `ventis`; environment variables and Docker resources retain the -`VENTIS_*` and `ventis-*` prefixes. These are protocol identifiers, not branding -strings. Do not rename them. - -## Load references only when needed - -- Read [references/packaging.md](references/packaging.md) when a source import - does not resolve from `/app`, the source is nested, or packaging metadata is - involved. -- Read [references/llm-proxy.md](references/llm-proxy.md) only when the target - includes `llm_proxy`. -- Read [references/ec2.md](references/ec2.md) only when any config entry uses - `provider: EC2`. -- Read [references/troubleshooting.md](references/troubleshooting.md) after a - failed build, image probe, deploy, or request. -- Read [references/runtime-contract.md](references/runtime-contract.md) when a - validator finding needs explanation or the runtime mechanism is unclear. - -## Goal: thin scaffolding beside untouched source - -```text -agents/.yaml one callable surface per service -agents/.py one thin adapter per service, when needed -workflow/_workflow.py HTTP entry point; calls deploy() -config/global_controller.yaml deployment manifest -config/policy.yaml optional access restriction -pyproject.toml conditional nested-import scaffolding - unchanged -``` - -The file count follows the deployment. A multi-agent port has one yaml/adapter -pair per service that is worth deploying separately. If a source class already -satisfies the runtime contract, point its config entry at that file and do not -copy it into an adapter. - -Everything the source already owns—prompts, tools, schemas, parsing, retries, -model clients, and node bodies—is imported. The port re-expresses only the -CanyonOS Core boundary and framework-owned orchestration. - -The port root is the existing repository root and the directory from which -`ventis build` runs. Write scaffolding there beside existing directories. If the -repository already uses `src/`, leave it in place and put `agents/`, `workflow/`, -and `config/` beside it. Never move or copy the repository into a new `src/` -directory, and never create an outer wrapper merely for the port. - -## 1. Survey before writing - -Identify: - -1. The source entry point and callable input/output. -2. Framework-owned control flow (`StateGraph`, `Crew`, `GroupChat`, routing, - `Send`, `Command`, interrupts). -3. Runtime-injected services nodes read: stores, context, memory, sessions, or - callback managers. -4. Sync versus async boundaries. -5. Imports and declared runtime dependencies. -6. Model provider, credential names, streaming use, and optional `llm_proxy`. -7. Whether independent work fans out and benefits from separate replicas. -8. Whether source imports resolve from the project root that becomes `/app`. - -Run the validator once now. Its header detects capabilities directly from the -importable runtime rather than external development metadata: - -```bash -python /validate.py . -``` - -If config or agent yaml is malformed, the validator defers to `ventis build`. -Capability-gated findings say which runtime behavior is available. - -## 2. Choose service boundaries - -Start with one service. Split only when it creates independent parallel work or -a distinct resource/replica profile. - -- Keep a ReAct loop together; every turn needs shared message history. -- Hoist supervisor task lists and `Send`-style fan-out into the workflow. -- Do not create a one-replica service with no distinct resource profile merely - to mirror every source graph node. - -Rewrite framework-owned edges as ordinary Python. Import the connected node -functions unchanged. Construct runtime-injected service objects from source -configuration; do not invent models, dimensions, stores, or defaults silently. -Report any choice the source does not specify. - -## 3. Write declarations and adapters - -### Agent yaml - -Use one yaml per deployed service. Argument types are bare builtins only: -`str`, `int`, `float`, `bool`, `dict`, or `list`. Every declared argument is -required by the generated stub. `returns.type` is documentation; use `dict` or -`list` to signal that workflow callers must `json.loads` the returned string. - -### Adapter - -The entrypoint exposes a module-level class named exactly `agent.name`. It -constructs with no arguments and its declared methods are synchronous. Read -configuration from the environment in `__init__`. Bridge source coroutines -inside a synchronous method with `asyncio.run(...)`. Serialize framework objects -with their own JSON-safe serializer before returning. - -Do not duplicate source prompts, tools, schemas, or model calls. Keep the source -provider and SDK. - -### Workflow - -Expose `main(query: str)` and call `deploy(main, port=...)` at module scope. -Import generated stubs by yaml basename and agent class name: - -```python -from deploy import deploy -from agents. import -``` - -The deployment platform sends `{query: string}` to `/main`. Pack richer input -inside `query`; any additional workflow parameter has a default. - -Dispatch every remote call before resolving any future: - -```python -futures = [agent.work(item=item) for item in items] -results = [json.loads(future.value()) for future in futures] -``` - -Do not fuse dispatch and `.value()` in one comprehension; that silently -serializes fan-out. Do not add an `if __name__ == "__main__":` block: the -workflow is executed with `__name__ == "__main__"` in production. - -### Config - -For each service, keep these names aligned: - -```text -config entry name == yaml agent.name == entrypoint class name -``` - -Use lowercase `provider: local`; `replicas` is an integer; `requirements` is a -list of distribution-name strings. Put `env_file` at config top level when the -runtime capability is available. Omit `policy.yaml` unless access must be -restricted; if present, give it a non-empty `rules` list. - -## Hard rules - -Capitalized **MUST** and **NEVER** are reserved for port-breaking or -source-integrity rules. The owner column states where each is decided. - -| ID | Rule | Owner | -|---|---|---| -| M1 | Entrypoint MUST define a class named exactly `agent.name` | V006 | -| M2 | The class MUST construct with no arguments | V007 | -| M3 | yaml argument names MUST match Python parameter names | V008 | -| M4 | yaml argument types MUST be bare builtins | V010 | -| M5 | Declared adapter methods MUST be synchronous | V009 | -| M6 | Config names MUST match yaml agent names | build | -| M7 | Config names MUST not collide after lowercase normalization | build output | -| M8 | Local provider MUST be lowercase `local` | deploy preflight | -| M9 | `replicas` MUST be an integer | deploy preflight | -| M10 | `requirements` MUST be a list of strings | build | -| M11 | Workflow MUST expose `main(query)`; extra parameters MUST default | V016 | -| M12 | Workflow MUST NEVER contain a main guard | V017 | -| M13 | Fan-out MUST dispatch all calls before resolving any | V018 | -| M14 | Project modules MUST not take runtime or generated-stub flat names | V019/V020 | -| M15 | Workflow MUST import stubs from `agents.` | V023 | -| M16 | Policy MUST be absent or contain a non-empty `rules` list | deploy preflight | -| M17 | EC2 entries MUST satisfy the EC2 deployment contract | deploy preflight | -| M18 | NEVER copy source prompts, tools, schemas, or model calls | review | -| M19 | NEVER hardcode or bake a real credential into an image | W003 | -| M20 | NEVER edit or vendor the source tree | `git status` | -| M21 | NEVER swap the source LLM provider | review | -| M22 | NEVER silently move, drop, or reclassify source dependencies | review | -| M23 | Framework control flow MUST be rewritten; source node logic MUST be imported | review | -| M24 | A non-resolving source import MUST have usable root packaging metadata when editable install is supported | V031 | - -## 4. Validate, build, and probe - -Run static preflight, then let the build own build-time validation: - -```bash -python /validate.py . -ventis build -c config/global_controller.yaml -``` - -A green build never imports the adapter. Probe each agent image in this order: - -```bash -# Runtime startup path - docker run --rm ventis- \ - python -c "import local_controller" - -# Agent load path; include --env-file when configured - docker run --rm --env-file ventis- \ - python -c "import importlib.util,sys; \ -s=importlib.util.spec_from_file_location('m','.py'); \ -m=importlib.util.module_from_spec(s);sys.modules['m']=m;s.loader.exec_module(m); \ -m.();print('ok')" -``` - -Also probe the workflow image with `python -c "import local_controller"`; it has -its own dependency resolve and generated-stub imports. - -Then deploy, send a representative request, and poll its status: - -```bash -ventis deploy -c config/global_controller.yaml -curl -X POST http://localhost:8080/main \ - -H 'Content-Type: application/json' -d '{"query":""}' -curl http://localhost:8080/status/ -``` - -A successful outer request with a source-level failure still proves the port -reached and returned the source behavior. Record the distinction. - -## 5. Clean up - -After collecting evidence, stop foreground deploy with Ctrl+C and wait for -controller cleanup. Remove exact leftovers if startup crashed. Then remove build -products and exact images from this config: - -```bash -ventis clean -docker image rm ventis- \ - ventis- - -test ! -e stubs && test ! -e grpc_stubs && test ! -e docker_container -docker ps -a --format '{{.Names}}' -``` - -`ventis clean` removes only `stubs/`, `grpc_stubs/`, and `docker_container/`; it -does not remove containers or images. Keep port scaffolding, untouched source, -and requested logs or reports. diff --git a/.claude/skills/porting-to-canyonos-core/references/ec2.md b/.claude/skills/porting-to-canyonos-core/references/ec2.md deleted file mode 100644 index e06daa0..0000000 --- a/.claude/skills/porting-to-canyonos-core/references/ec2.md +++ /dev/null @@ -1,41 +0,0 @@ -# EC2 deployment - -Read this only when at least one config entry uses `provider: EC2`. - -## Configuration - -Use `provider: EC2` and declare `instance_type` on every EC2 service entry. The -top-level `ec2` block supplies the runtime's required infrastructure and SSH -settings. Read the target checkout's deploy preflight and EC2 runtime before -writing the block; do not copy values from an example environment. - -Typical required categories are: - -- AMI and instance type -- region and subnet -- security groups -- SSH user and credentials accepted by the runtime - -`ventis deploy` owns basic EC2 config validation. A preflight pass is not proof -that provisioning, SSH, image transfer, or remote container startup works. - -## Networking - -A remote container's `host.docker.internal` names its own EC2 Docker host. It -does not name the local controller machine. Databases, model proxies, and other -services must use addresses reachable from every selected host. - -The environment file may be copied temporarily to a remote host by runtimes that -expose the `env_file` capability. Confirm behavior from the capability probe and -target runtime rather than assuming local Docker semantics. - -## Probes and cleanup - -Run the same runtime and adapter probes against the exact image before remote -deployment. After deploy, verify the remote container logs; controller health -can be green even when agent loading failed. - -Stop foreground deploy normally so the controller can terminate recorded EC2 -instances. If provisioning or startup fails before an instance is recorded, -inspect the cloud provider directly and remove exact leaked resources. Never use -a broad cleanup command against unrelated instances. diff --git a/.claude/skills/porting-to-canyonos-core/references/llm-proxy.md b/.claude/skills/porting-to-canyonos-core/references/llm-proxy.md deleted file mode 100644 index f726bd7..0000000 --- a/.claude/skills/porting-to-canyonos-core/references/llm-proxy.md +++ /dev/null @@ -1,64 +0,0 @@ -# LLM proxy integration - -Read this only when the target checkout contains `llm_proxy` or the deployment -explicitly routes model SDKs through it. - -## Preserve provider protocols - -The proxy redirects provider endpoints; it does not convert providers. Keep the -source SDK, model ID, request body, and response parsing unchanged. - -Configure only the provider variables the source uses: - -```dotenv -OPENAI_BASE_URL=http://host.docker.internal:8081/openai/v1 -ANTHROPIC_BASE_URL=http://host.docker.internal:8081/anthropic -AWS_ENDPOINT_URL_BEDROCK_RUNTIME=http://host.docker.internal:8081/bedrock -``` - -Some SDKs refuse to initialize without caller credentials. Give agent containers -non-secret placeholders only when required. Keep real OpenAI, Anthropic, or AWS -credentials in the separate proxy process, not in the port's `env_file`. - -## Start locally - -The proxy defaults conflict with a typical deployment: host loopback is not -reachable from a container, and port 8080 is normally used by the workflow API. -Use a non-loopback bind and a different port: - -```bash -PROXY_HOST=0.0.0.0 PROXY_PORT=8081 python -m llm_proxy -curl http://127.0.0.1:8081/healthz -``` - -Local CanyonOS Core containers resolve `host.docker.internal` through their -Docker host mapping. On EC2 that name resolves to each EC2 Docker host, not the -machine running `ventis deploy`. Distributed deployments need a reachable proxy -address or one proxy on each host. - -## Supported call shape - -The implementation buffers complete requests and responses: - -- OpenAI and Anthropic non-streaming HTTP calls are forwarded. -- Bedrock `invoke` is reissued through the proxy's boto3 client. -- OpenAI/Anthropic streaming is unsupported. -- Bedrock `invoke-with-response-stream`, `converse`, and `converse-stream` are - unsupported. - -Survey the source before selecting the proxy. Do not silently disable streaming; -report the unsupported behavior and stop. - -## Credential behavior - -- The OpenAI adapter removes caller authorization and inserts the proxy key. -- The Anthropic adapter removes caller key headers and inserts the proxy key. -- Botocore still signs requests sent to a custom endpoint, so a caller may need - placeholder AWS credentials even though the proxy reissues upstream with its - own identity. -- `/healthz` proves provider registration and Flask availability, not upstream - credential validity. - -OpenAI and Anthropic upstream HTTP errors pass through. Proxy exceptions return -JSON 502 with `error: proxy_error`. Bedrock `ClientError` bodies are reconstructed -with the upstream status and are not byte-for-byte passthrough. diff --git a/.claude/skills/porting-to-canyonos-core/references/packaging.md b/.claude/skills/porting-to-canyonos-core/references/packaging.md deleted file mode 100644 index 8520c4a..0000000 --- a/.claude/skills/porting-to-canyonos-core/references/packaging.md +++ /dev/null @@ -1,81 +0,0 @@ -# Packaging and import roots - -Read this reference when an adapter imports nested source code, the source uses a -`src/` layout, or V031 reports an import-root problem. - -## What `/app` can import - -CanyonOS Core preserves project-relative paths in the image and starts Python at -`/app`. Without an editable install, Python resolves names rooted there: - -- `/app/tools.py` as `import tools` -- `/app/pkg/__init__.py` as `import pkg` -- `/app/src/agents/...` as `import src.agents`, including PEP 420 namespace - directories without `__init__.py` - -It does not resolve `/app/source/pkg` as `import pkg`; `/app/source` must become -an import root first. - -## Detect support, do not infer it from release history - -Run: - -```bash -python /validate.py . -``` - -Read the `editable_install` capability. If it is unavailable and the original -import cannot resolve from `/app`, report a runtime capability blocker and stop. -Do not add a `sys.path` hack or relocate source files. - -## Root metadata is the trigger - -When editable install is supported, only packaging metadata at the **port root** -triggers `pip install -e .`: - -```text -port-root/pyproject.toml detected -port-root/source/pyproject.toml ignored as an install trigger -``` - -A nested source repository may remain untouched. Add minimal root scaffolding -that points package discovery at the existing source package: - -```toml -[build-system] -requires = ["setuptools>=64"] -build-backend = "setuptools.build_meta" - -[project] -name = "canyonos-port" -version = "0.0.0" -dependencies = [] - -[tool.setuptools.packages.find] -where = ["source/src"] -include = ["pkg*"] -namespaces = true -``` - -Set `where` and `include` from the actual tree and original import spelling. Do -not reference a README or license from this wrapper metadata; file sweeps differ -by runtime capability and a missing referenced file makes the image build fail. - -## Dependencies in nested metadata - -A nested `pyproject.toml` is not installed merely because its Python files are -copied. Keep the source declaration unchanged and repeat its runtime -distributions in each relevant config entry's `requirements` list. This is -compatibility scaffolding, not permission to drop, move, or reclassify declared -dependencies. - -If source metadata is already at the port root, do not create a wrapper. Its -project dependencies participate in the same resolver as config requirements. -Report declared-but-unused toolchain dependencies and their image cost; let the -owner decide whether source metadata should change. - -## Validation boundary - -`ventis build` owns packaging syntax and installation errors. `validate.py` -checks only whether adapter imports appear to require a nested root that the -runtime will not expose. diff --git a/.claude/skills/porting-to-canyonos-core/references/runtime-contract.md b/.claude/skills/porting-to-canyonos-core/references/runtime-contract.md deleted file mode 100644 index 94f7401..0000000 --- a/.claude/skills/porting-to-canyonos-core/references/runtime-contract.md +++ /dev/null @@ -1,187 +0,0 @@ -# CanyonOS Core runtime contract - -The product is CanyonOS Core. Compatibility identifiers remain `ventis` for the -CLI and Python package, `VENTIS_*` for runtime variables, and `ventis-*` for -Docker resources. - -Read this reference when implementing an adapter or explaining a validator -finding. Runtime-dependent behavior is expressed as capabilities; run -`validate.py` against the target environment instead of inferring support from -release history. - -## Project root and discovery - -`ventis build` uses the current working directory as the project root. - -| Input | Discovery | -|---|---| -| `agents/*.yaml` | direct yaml glob under the project root | -| `config/global_controller.yaml` | default config, overridable with `-c` | -| workflow | `workflow_file` on a `type: workflow` entry | -| policy | `policy.yaml` beside the selected config file | -| generated files | `stubs/`, `grpc_stubs/`, `docker_container/` | - -The config name, yaml `agent.name`, and entrypoint class name form one binding: - -```text -config entry name == yaml agent.name == entrypoint class name -``` - -A missing match may skip an image while the command continues, so inspect build -output and generated image tags. - -## Agent yaml and generated stubs - -The consumed yaml shape is: - -```yaml -agent: - name: ExampleAgent - functions: - - name: work - arguments: - - name: query - type: str - returns: - type: dict -``` - -Argument annotations are generated from bare names without adding imports. Use -builtins. Generated methods have no defaults, so every declared argument is -required at the stub call site. `returns` does not control runtime conversion; -it documents whether workflow code should parse the returned string. - -Stub destinations differ by runtime capability and entrypoint layout. Workflow -code in this port convention imports the generated class from -`agents.`. The validator checks that import against declarations -before the workflow image starts. - -## Agent loading and execution - -The local controller effectively performs: - -```python -module = load(entrypoint) -agent_class = getattr(module, configured_name) -agent = agent_class() -result = getattr(agent, method_name)(**args) -``` - -Consequences: - -- The class is module-level and named exactly as configured. -- Construction takes no arguments. -- Declared methods accept yaml argument names as keyword arguments. -- Methods are synchronous; this path does not await a coroutine. -- Dicts and lists are JSON-encoded before entering Redis; other results become - strings. -- A remote Future's `.value()` returns text, not the original Python object. - -Agent import and construction exceptions are caught by the controller. A failed -agent may still advertise healthy because health is written independently of -successful agent loading. That is why image probes import both the runtime and -the entrypoint explicitly. - -## Workflow execution - -The workflow file is executed, not imported. Therefore: - -- module-level code runs at container startup; -- `__name__ == "__main__"`; -- `deploy()` blocks in the web server; -- the workflow function runs once per request; -- its function name determines the REST route exposed by the compatibility - runtime. - -The deployment platform additionally expects `/main` with a `{query: string}` -body. This platform constraint is stricter than the underlying transport. - -Each stub method returns a Future immediately. `.value()` blocks. Dispatching -and resolving inside one comprehension serializes work without raising an -error; dispatch all calls first, then resolve them. - -The workflow container also starts runtime controller code and has its own -package resolution. Probe it independently from agent images. - -## Build context and collisions - -The runtime copies project files while preserving relative paths, then writes -shared runtime modules, generated stubs, and entrypoints into the image. Later -writes can shadow project files. - -Avoid root project modules named like runtime files, including: - -```text -future.py -ventis_context.py -local_controller.py -local_controller_frontend.py -redis_client.py -grpc_options.py -bedrock.py -deploy.py -session_logging.py -workflow_launcher.py -``` - -Also avoid a yaml basename that shadows a different source module imported by an -adapter. The validator checks deterministic flat-name collisions. - -File sweep and editable-install behavior are runtime capabilities. For nested -imports, follow [packaging.md](packaging.md). - -## Dependencies and protobuf - -Agent and workflow images include a small runtime dependency set. Config -`requirements` adds source-specific distributions. A malformed requirements -value can be normalized away while image generation continues; missing imports -then surface only when the agent loads. - -The build compiles gRPC Python stubs on the host and copies them into images. -The image resolver does not necessarily know the generated-code version. A -source dependency that constrains protobuf below the host generator version can -produce a green image build that dies on: - -```text -import local_controller -``` - -Always run that probe before probing the entrypoint. Treat a generated-code / -runtime-version mismatch as a CanyonOS Core runtime issue, not a reason to alter -source dependencies silently. - -## Credentials capability - -When `env_file` capability is available, the top-level config path is resolved -against the project root and passed at container start. Hidden env files are not -copied into images. Invalid paths are deploy-preflight errors. - -When the capability is unavailable, declaring `env_file` has no effect. If the -source needs credentials, report the capability blocker rather than hardcoding -or vendoring a secret. - -A source that constructs its client at import time works only when credentials -are already in the container environment. Image entrypoint probes therefore use -the same env file as deployment. - -For proxy-specific credential separation, read [llm-proxy.md](llm-proxy.md). - -## Policy and provider behavior - -No policy file means unrestricted service access. If a policy exists, it needs a -non-empty rules list. Rules are evaluated by specificity and first match; -services excluded from the selected rule fail after request acceptance. - -Local provider handling is case-sensitive: use lowercase `local`. EC2 behavior -and remote networking are covered in [ec2.md](ec2.md). - -## Cleanup boundary - -Stopping foreground deploy normally invokes controller cleanup for recorded -containers and Redis. Hard kills and failures before resource registration may -leave resources behind. - -`ventis clean` removes generated `stubs/`, `grpc_stubs/`, and -`docker_container/`. It does not remove containers or images. Remove exact -leftovers explicitly and preserve source, port scaffolding, and requested -evidence. diff --git a/.claude/skills/porting-to-canyonos-core/references/troubleshooting.md b/.claude/skills/porting-to-canyonos-core/references/troubleshooting.md deleted file mode 100644 index 361acf9..0000000 --- a/.claude/skills/porting-to-canyonos-core/references/troubleshooting.md +++ /dev/null @@ -1,60 +0,0 @@ -# Troubleshooting - -Read this after a failed build, image probe, deploy, or request. For mechanisms, -read [runtime-contract.md](runtime-contract.md). For proxy or remote-host failures, -read [llm-proxy.md](llm-proxy.md) or [ec2.md](ec2.md). - -## Build or deploy stops early - -| Symptom | Likely cause | -|---|---| -| Agent image is missing | Config name matched no yaml name, entrypoint is absent, or build skipped it; inspect build warnings | -| Two services produce one image | Config names collide after lowercase normalization | -| `generated grpc_stubs are missing` | Build did not complete on this host, or generated output was cleaned before deploy | -| `int(... NoneType)` during local deploy | Local provider was not written exactly as lowercase `local` | -| Replica conversion `TypeError` | `replicas` is not an integer | -| Policy `AttributeError` | Policy exists but is empty or has null/non-list rules; remove it when unrestricted | -| Port or container name already in use | A previous deployment did not complete cleanup | - -## Container exits or serves nothing - -| Symptom | Likely cause | -|---|---| -| `import local_controller` raises protobuf version error | Host-generated gRPC code is newer than the image's protobuf runtime | -| First request says `No agent loaded` | Entrypoint import, class lookup, or constructor failed and the controller swallowed the exception; inspect container logs | -| Replica is healthy but serves nothing | Health publication does not prove successful agent loading | -| Missing credentials while loading | Env injection is unavailable/misconfigured or the source reads another variable | -| Source module is missing | Original import does not resolve from `/app`; read [packaging.md](packaging.md) | -| Third-party module is missing | Distribution is absent from source metadata and config requirements | -| Stub import raises `NameError` | yaml argument type is not a bare builtin | -| Source module behaves like an empty stub | A generated stub basename shadowed the source module | -| Runtime-named project module disappears | Shared runtime copy overwrote a root project module with the same name | - -## Request is accepted, then fails - -| Symptom | Likely cause | -|---|---| -| Unexpected keyword argument | yaml argument name differs from adapter parameter name | -| Required argument missing | yaml does not declare a required adapter parameter, or workflow platform sent only `query` | -| Unauthorized service | The first matching policy rule excludes that service | -| `.value()` returns dict-like text | Expected; remote values are strings, so use `json.loads` | -| Object is not JSON serializable | Adapter returned framework objects without its JSON-safe serializer | -| Redis contains a coroutine repr | Adapter method is async and the controller did not await it | -| Fan-out is no faster | Dispatch and `.value()` were fused, serializing the calls | -| Debug block runs at startup | Workflow is executed with `__name__ == "__main__"` | - -## Deployment platform endpoint - -| Symptom | Likely cause | -|---|---| -| 404 while workflow container is healthy | Workflow function is not named `main` | -| 400 before host receives request | Body is not the platform's `{query: string}` shape | -| Extra workflow argument is missing | Platform sends only `query`; extra parameters need defaults | - -## Cleanup - -| Symptom | Likely cause | -|---|---| -| `ventis clean` succeeds but containers remain | The command removes generated directories only | -| `ventis clean` succeeds but images remain | Image deletion is separate and requires exact tags | -| Next deployment collides with old resources | Foreground deploy was killed or crashed before controller cleanup | diff --git a/.claude/skills/porting-to-canyonos-core/validate.py b/.claude/skills/porting-to-canyonos-core/validate.py deleted file mode 100755 index 04baf37..0000000 --- a/.claude/skills/porting-to-canyonos-core/validate.py +++ /dev/null @@ -1,1257 +0,0 @@ -#!/usr/bin/env python3 -"""Preflight the runtime traps that `ventis build` cannot see. - -This deliberately does not duplicate build-time validation such as malformed -YAML, missing entrypoints, or config-to-yaml matching. `ventis build` owns those -checks. This script parses Python without importing it and catches failures that -otherwise stay hidden until a container loads an agent, starts a workflow, or -serves its first request. A replica is not evidence: the controller writes -`healthy` to Redis before `_load_agent` runs. - - python validate.py [project_dir] [-c config/global_controller.yaml] - [--json] [--strict] - -Exit 1 if any ERROR was reported, 0 otherwise. --strict also fails on warnings. - -Runtime capabilities vary across CanyonOS Core installations. This script probes -the importable `ventis` package directly. A capability-gated check reports -UNAVAILABLE when its behavior cannot be proven. -""" - -import argparse -import ast -import builtins -import json -import os -import re -import sys -from typing import ClassVar - -try: - import yaml -except ImportError: # pragma: no cover - pyyaml is a CanyonOS Core dependency - sys.stderr.write("validate.py needs pyyaml: pip install pyyaml\n") - raise SystemExit(2) from None - - -DEFAULT_CONFIG_PATH = "config/global_controller.yaml" - -# Copied flat into every image over the swept project tree, so a project module -# landing flat under one of these names is overwritten. -# ventis/stub_generator.py generate_docker / generate_workflow_docker. -RUNTIME_FLAT_NAMES = frozenset( - { - "future.py", - "ventis_context.py", - "local_controller.py", - "local_controller_frontend.py", - "redis_client.py", - "grpc_options.py", - "gpu_metrics.py", - "bedrock.py", - "deploy.py", - "session_logging.py", - "workflow_launcher.py", - } -) - -# ventis/stub_generator.py BASE_AGENT_REQUIREMENTS / BASE_WORKFLOW_REQUIREMENTS. -BASE_AGENT_REQUIREMENTS = [ - "grpcio", - "grpcio-tools", - "redis", - "pyyaml", - "psutil", - "ipdb", - "ipython", - "boto3", -] -# Import name -> distribution name, for the handful where they differ and the -# mismatch would otherwise be reported as a missing requirement. -IMPORT_TO_DISTRIBUTION = { - "attr": "attrs", - "bs4": "beautifulsoup4", - "cv2": "opencv-python", - "dateutil": "python-dateutil", - "dotenv": "python-dotenv", - "grpc": "grpcio", - "grpc_tools": "grpcio-tools", - "jwt": "pyjwt", - "PIL": "pillow", - "psycopg": "psycopg", - "psycopg2": "psycopg2-binary", - "pydantic_settings": "pydantic-settings", - "sklearn": "scikit-learn", - "typing_extensions": "typing-extensions", - "yaml": "pyyaml", -} - -SECRET_PATTERNS = [ - (re.compile(r"sk-[A-Za-z0-9_-]{20,}"), "an OpenAI-style secret key"), - (re.compile(r"AKIA[0-9A-Z]{16}"), "an AWS access key id"), - (re.compile(r"gh[pousr]_[A-Za-z0-9]{20,}"), "a GitHub token"), - (re.compile(r"AIza[0-9A-Za-z_-]{30,}"), "a Google API key"), -] -SECRET_NAME = re.compile(r"(API_KEY|SECRET|TOKEN|PASSWORD|CREDENTIAL)", re.IGNORECASE) - -ERROR = "ERROR" -WARN = "WARN" -INFO = "INFO" - - -# ------------------------------------------------------------------ # -# Capabilities # -# ------------------------------------------------------------------ # -# -# Stable labels for behavior detected from the importable runtime. They contain -# no external development metadata. - -CAPABILITY_SOURCE = { - "env_file": "runtime env-file injection", - "editable_install": "editable project installation", - "sweeps_all_files": "full project-file sweep", - "stub_two_destinations": "flat and package stub destinations", -} - - -def probe_capabilities(): - """Ask the importable ventis package what it actually supports.""" - caps = dict.fromkeys(CAPABILITY_SOURCE, False) - caps["ventis"] = False - try: - from ventis import stub_generator - except Exception: # noqa: BLE001 - a broken install must not crash the check - return caps - - caps["ventis"] = True - caps["editable_install"] = hasattr(stub_generator, "_install_step") - caps["sweeps_all_files"] = hasattr(stub_generator, "_sweep_project_files") - caps["stub_two_destinations"] = hasattr(stub_generator, "_stub_destinations") - - import importlib - - for module_name in ("ventis.controller.utils.env_file", "ventis.utils.env_file"): - try: - module = importlib.import_module(module_name) - except Exception: # noqa: BLE001,S112 - the other path is the live one - continue - if hasattr(module, "resolve_env_file"): - caps["env_file"] = True - break - return caps - - -# ------------------------------------------------------------------ # -# YAML with line numbers # -# ------------------------------------------------------------------ # - - -class LineDict(dict): - """A mapping that remembers where it and each of its keys were written.""" - - line = 0 - key_lines: ClassVar[dict] = {} - - -class LineLoader(yaml.SafeLoader): - pass - - -def _construct_mapping(loader, node): - data = LineDict() - yield data - data.update(loader.construct_mapping(node, deep=False)) - data.line = node.start_mark.line + 1 - data.key_lines = { - key.value: key.start_mark.line + 1 - for key, _ in node.value - if isinstance(key, yaml.ScalarNode) - } - - -LineLoader.add_constructor( - yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, _construct_mapping -) - - -def line_of(mapping, key=None): - """The source line of `key` inside `mapping`, or of the mapping itself.""" - if not isinstance(mapping, LineDict): - return 0 - if key is not None: - return mapping.key_lines.get(key, mapping.line) - return mapping.line - - -def load_yaml(path): - """Parse `path`, returning (data, error). Never raises.""" - try: - with open(path, "r", encoding="utf-8") as handle: - return yaml.load(handle, Loader=LineLoader), None - except Exception as exc: # noqa: BLE001 - any parse failure is a finding - return None, str(exc) - - -# ------------------------------------------------------------------ # -# Findings # -# ------------------------------------------------------------------ # - - -class Report: - def __init__(self, project_dir, capabilities): - self.project_dir = project_dir - self.capabilities = capabilities - self.findings = [] - # A peer agent is imported by the name of its generated stub, which the - # build copies flat into every image. Those are not project modules and - # need no requirement. - self.stub_module_names = set() - - def add(self, check, level, path, line, summary, mechanism): - self.findings.append( - { - "check": check, - "level": level, - "path": self.rel(path) if path else "", - "line": line or 0, - "summary": summary, - "mechanism": mechanism, - } - ) - - def error(self, check, path, line, summary, mechanism): - self.add(check, ERROR, path, line, summary, mechanism) - - def warn(self, check, path, line, summary, mechanism): - self.add(check, WARN, path, line, summary, mechanism) - - def unavailable(self, check, summary): - self.add(check, INFO, "", 0, summary, "") - - def rel(self, path): - try: - return os.path.relpath(path, self.project_dir) - except ValueError: - return path - - def counts(self): - errors = sum(1 for f in self.findings if f["level"] == ERROR) - warnings = sum(1 for f in self.findings if f["level"] == WARN) - return errors, warnings - - -# ------------------------------------------------------------------ # -# Python source helpers # -# ------------------------------------------------------------------ # - - -def parse_python(path): - """AST for `path`, or (None, error). The port is never imported.""" - try: - with open(path, "r", encoding="utf-8") as handle: - source = handle.read() - except OSError as exc: - return None, str(exc) - try: - return ast.parse(source, filename=path), None - except SyntaxError as exc: - return None, f"{exc.msg} (line {exc.lineno})" - - -def find_class(tree, name): - for node in tree.body: - if isinstance(node, ast.ClassDef) and node.name == name: - return node - return None - - -def class_methods(class_node): - return { - node.name: node - for node in class_node.body - if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) - } - - -def parameter_names(func_node): - """Every parameter a caller can pass by keyword, minus self.""" - args = func_node.args - positional = [a.arg for a in args.posonlyargs + args.args] - if positional and positional[0] in ("self", "cls"): - positional = positional[1:] - return positional + [a.arg for a in args.kwonlyargs] - - -def required_parameters(func_node): - """Parameters with no default, minus self.""" - args = func_node.args - positional = [a.arg for a in args.posonlyargs + args.args] - if positional and positional[0] in ("self", "cls"): - positional = positional[1:] - if args.defaults: - positional = positional[: len(positional) - len(args.defaults)] - kwonly = [ - arg.arg - for arg, default in zip(args.kwonlyargs, args.kw_defaults) - if default is None - ] - return positional + kwonly - - -def toplevel_import_names(tree): - """Top-level package name of every import in the module, with line numbers.""" - names = {} - for node in ast.walk(tree): - if isinstance(node, ast.Import): - for alias in node.names: - names.setdefault(alias.name.split(".")[0], node.lineno) - elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module: - names.setdefault(node.module.split(".")[0], node.lineno) - return names - - -# ------------------------------------------------------------------ # -# V006-V010 adapter failures hidden by _load_agent # -# ------------------------------------------------------------------ # - -BUILTIN_TYPE_NAMES = frozenset( - name - for name in dir(builtins) - if isinstance(getattr(builtins, name), type) and not name[0].isupper() -) - - -def check_adapter(report, agent_yaml_path, agent_block, entry, project_dir): - """V006 V007 V008 V009 V010.""" - name = agent_block["name"] - functions = agent_block.get("functions") or [] - check_argument_types(report, agent_yaml_path, functions) - - entrypoint = entry.get("entrypoint") - if not entrypoint: - return - entrypoint_path = os.path.join(project_dir, entrypoint) - if not os.path.isfile(entrypoint_path): - return - - tree, error = parse_python(entrypoint_path) - if tree is None: - report.error( - "V006", - entrypoint_path, - 0, - f"the entrypoint does not parse: {error}", - "_load_agent exec_module's it and swallows the exception; the first " - "request answers 'No agent loaded'.", - ) - return - - class_node = find_class(tree, name) - if class_node is None: - classes = [n.name for n in tree.body if isinstance(n, ast.ClassDef)] - found = ", ".join(classes) if classes else "no classes at all" - report.error( - "V006", - entrypoint_path, - 1, - f"no class named `{name}` at module level (found: {found})", - "_load_agent does getattr(module, VENTIS_AGENT_NAME) and swallows " - "the AttributeError. The class name must equal agent.name exactly.", - ) - return - - methods = class_methods(class_node) - check_constructor(report, entrypoint_path, name, methods) - - for func in functions: - if not isinstance(func, dict) or not isinstance(func.get("name"), str): - continue - check_method(report, entrypoint_path, agent_yaml_path, name, func, methods) - - -def check_argument_types(report, agent_yaml_path, functions): - """V010 -- the type string is pasted into an ast.Name, never checked.""" - for func in functions: - if not isinstance(func, dict): - continue - for arg in func.get("arguments") or []: - if not isinstance(arg, dict) or "type" not in arg: - continue - declared = arg.get("type") - if not isinstance(declared, str): - continue # stub generation reports malformed type values - if declared in BUILTIN_TYPE_NAMES: - continue - report.error( - "V010", - agent_yaml_path, - line_of(arg, "type"), - f"`type: {declared}` is not a builtin", - "stub_generator pastes it verbatim into the generated " - "annotation, and the stub module imports only Future and " - "inspect. Anything else raises NameError when the stub is " - "imported -- after a green build. Use str int float bool dict " - "list.", - ) - - -def check_constructor(report, entrypoint_path, name, methods): - """V007 -- _load_agent calls agent_class() with no arguments.""" - init = methods.get("__init__") - if init is None: - return - required = required_parameters(init) - if required: - report.error( - "V007", - entrypoint_path, - init.lineno, - f"`{name}.__init__` requires {', '.join(required)}", - "_load_agent calls agent_class() with no arguments; the TypeError is " - "swallowed and the first request answers 'No agent loaded'. Read " - "configuration from the environment inside __init__ instead.", - ) - - -def check_method(report, entrypoint_path, agent_yaml_path, class_name, func, methods): - """V008 V009.""" - func_name = func["name"] - method = methods.get(func_name) - if method is None: - report.error( - "V008", - entrypoint_path, - 0, - f"`{class_name}` has no method `{func_name}`", - "The yaml declares it, so callers get a stub for it; the controller " - f"then answers \"Agent {class_name} has no method '{func_name}'\".", - ) - return - - # V009 -- nothing on the execution path awaits. - if isinstance(method, ast.AsyncFunctionDef): - report.error( - "V009", - entrypoint_path, - method.lineno, - f"`{class_name}.{func_name}` is `async def`", - "The executor calls method(**args) with no await, so Redis receives " - "''. Keep the signature synchronous and call " - "asyncio.run(...) inside the body.", - ) - - # V008 -- the controller calls method(**args) with the yaml's names. - declared = [ - arg["name"] - for arg in func.get("arguments") or [] - if isinstance(arg, dict) and isinstance(arg.get("name"), str) - ] - actual = parameter_names(method) - required = required_parameters(method) - - missing = [arg for arg in declared if arg not in actual] - if missing: - report.error( - "V008", - entrypoint_path, - method.lineno, - f"`{class_name}.{func_name}` has no parameter " - f"{', '.join(repr(m) for m in missing)}, but the yaml declares it", - "LocalController does method(**args) with the yaml's argument names. " - "A mismatch is TypeError: unexpected keyword argument, at request " - f"time. See {os.path.basename(agent_yaml_path)}.", - ) - - unfilled = [arg for arg in required if arg not in declared] - if unfilled: - report.error( - "V008", - entrypoint_path, - method.lineno, - f"`{class_name}.{func_name}` requires {', '.join(unfilled)}, which " - "the yaml does not declare", - "Only declared arguments are ever sent, and the generated stub gives " - "none of them a default. Declare them in the yaml or default them " - "in the signature.", - ) - - - -# ------------------------------------------------------------------ # -# V016-V018 the workflow # -# ------------------------------------------------------------------ # - - -def check_stub_imports(report, workflow_path, tree, stub_classes): - """V023 -- the workflow must import a stub as `from agents. import `. - - The build copies each stub to exactly one path, and for the workflow image - that path is agents/.py. Two ways of writing this line fail, and - the project walks you into both: the flat form is what examples/ uses, and - the class name is the one `ventis build` prints, which is not the one it - writes. - """ - for node in ast.walk(tree): - if isinstance(node, ast.Import): - for alias in node.names: - base = alias.name.split(".")[0] - if base in stub_classes: - report.error( - "V023", workflow_path, node.lineno, - f"`import {alias.name}` -- the stub is at " - f"agents/{base}.py, not flat", - "The build copies a stub to one path, and for the " - "workflow that path is under agents/. This is a " - "ModuleNotFoundError the moment the workflow runs. " - f"Write `from agents.{base} import {stub_classes[base]}`.", - ) - continue - - if not isinstance(node, ast.ImportFrom) or not node.module: - continue - - module = node.module - if module in stub_classes: - report.error( - "V023", workflow_path, node.lineno, - f"`from {module} import ...` -- the stub is at " - f"agents/{module}.py, not flat", - "The build copies a stub to one path, and for the workflow " - "that path is under agents/. The flat form is what this " - "repository's own examples use and it raises " - "ModuleNotFoundError in the workflow image. Write " - f"`from agents.{module} import {stub_classes[module]}`.", - ) - continue - - if not module.startswith("agents."): - continue - base = module.split(".", 1)[1] - expected = stub_classes.get(base) - if expected is None: - continue - for alias in node.names: - if alias.name == expected: - continue - if alias.name == f"{expected}Stub": - report.error( - "V023", workflow_path, node.lineno, - f"`{alias.name}` is the name the build prints, not the " - f"class it writes", - "generate_agent_stub sets class_name = agent_config['name'] " - "and then recomputes it with a 'Stub' suffix for the log " - "line only. The message names a class that does not exist; " - f"the class is `{expected}`.", - ) - else: - report.error( - "V023", workflow_path, node.lineno, - f"`{alias.name}` is not a class the stub for {base} defines", - f"The stub's class carries the agent's own name: `{expected}`.", - ) - - -def check_workflow(report, workflow_path, stub_classes=None): - """V016 V017 V018 V023.""" - tree, error = parse_python(workflow_path) - if tree is None: - report.error("V016", workflow_path, 0, f"does not parse: {error}", "") - return - - main = None - for node in tree.body: - if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and ( - node.name == "main" - ): - main = node - break - - if main is None: - defined = [ - n.name - for n in tree.body - if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef)) - ] - found = ", ".join(defined) if defined else "no top-level functions" - report.error( - "V016", - workflow_path, - 1, - f"no top-level function named `main` (found: {found})", - "CanyonOS Core serves POST /, but the deployment platform's " - "test endpoint posts to a hardcoded /main. A differently named " - "workflow builds, deploys and stays unreachable -- 404, container " - "healthy.", - ) - else: - check_main_signature(report, workflow_path, main) - - if stub_classes: - check_stub_imports(report, workflow_path, tree, stub_classes) - - # V016 -- deploy() is what starts Flask. - if not any( - isinstance(node, ast.Call) - and isinstance(node.func, ast.Name) - and node.func.id == "deploy" - for node in ast.walk(tree) - ): - report.error( - "V016", - workflow_path, - 1, - "the workflow never calls `deploy(...)`", - "workflow_launcher.py exec's this file and nothing else starts the " - "HTTP server; the container comes up serving nothing.", - ) - - check_main_guard(report, workflow_path, tree) - check_fused_fanout(report, workflow_path, tree) - - -def check_main_signature(report, workflow_path, main): - """V016 -- the platform sends exactly {"query": ...}.""" - if isinstance(main, ast.AsyncFunctionDef): - report.error( - "V016", - workflow_path, - main.lineno, - "`main` is `async def`", - "deploy() calls workflow_fn(**kwargs) on a Flask worker thread with " - "no await; the response body would be a coroutine repr.", - ) - params = parameter_names(main) - if not params: - report.error( - "V016", - workflow_path, - main.lineno, - "`main` takes no arguments", - 'The platform posts {"query": "..."} and deploy() splats the ' - "body in as kwargs -- TypeError on every request.", - ) - return - if params[0] != "query": - report.error( - "V016", - workflow_path, - main.lineno, - f"`main`'s first parameter is `{params[0]}`, not `query`", - "The platform's body schema is strictly validated as " - "{query: string}; any other key is rejected with 400 in the control " - "plane, before the request reaches the host.", - ) - extra = [p for p in required_parameters(main) if p != "query"] - if extra: - report.error( - "V016", - workflow_path, - main.lineno, - f"`main` requires {', '.join(extra)} beyond `query`", - "Only `query` is ever sent, so every other parameter needs a " - "default or the call raises on every request. Pack richer input " - "into `query`.", - ) - - -def check_main_guard(report, workflow_path, tree): - """V017 -- the workflow is exec'd, so __name__ == "__main__".""" - for node in tree.body: - if not isinstance(node, ast.If): - continue - test = node.test - if ( - isinstance(test, ast.Compare) - and isinstance(test.left, ast.Name) - and test.left.id == "__name__" - and any( - isinstance(c, ast.Constant) and c.value == "__main__" - for c in test.comparators - ) - ): - report.error( - "V017", - workflow_path, - node.lineno, - '`if __name__ == "__main__":` block in the workflow', - "workflow_launcher.py runs exec(open().read()), so " - "__name__ IS '__main__' here and this block executes in " - "production, at container start.", - ) - - -def check_fused_fanout(report, workflow_path, tree): - """V018 -- .value() blocks, so dispatching and resolving in one - comprehension runs the fan-out one call at a time.""" - comprehensions = (ast.ListComp, ast.SetComp, ast.GeneratorExp) - for node in ast.walk(tree): - if not isinstance(node, comprehensions + (ast.DictComp,)): - continue - elements = ( - [node.key, node.value] if isinstance(node, ast.DictComp) else [node.elt] - ) - for element in elements: - for inner in ast.walk(element): - if ( - isinstance(inner, ast.Call) - and isinstance(inner.func, ast.Attribute) - and inner.func.attr == "value" - and isinstance(inner.func.value, ast.Call) - ): - report.error( - "V018", - workflow_path, - node.lineno, - "one comprehension both dispatches a call and resolves " - "it with .value()", - ".value() blocks, so each call completes before the " - "next is dispatched. It does not error -- the fan-out " - "is just silently serial, and with it the reason to be " - "on CanyonOS Core. Dispatch every call first, then resolve: " - "futures = [a.work(i) for i in items] then " - "[f.value() for f in futures].", - ) - return - - -# ------------------------------------------------------------------ # -# V019-V020 what the copy order overwrites # -# ------------------------------------------------------------------ # - - -def check_flat_collisions(report, project_dir, yaml_paths, entrypoints): - """V019 V020 -- later copies land on earlier ones at the context root.""" - for entry in sorted(os.listdir(project_dir)): - path = os.path.join(project_dir, entry) - if not os.path.isfile(path) or not entry.endswith(".py"): - continue - - # V019 -- the shared runtime is copied flat, after the project sweep. - if entry in RUNTIME_FLAT_NAMES: - report.error( - "V019", - path, - 1, - f"a project module named `{entry}` sits at the project root", - "The shared CanyonOS Core runtime is copied flat into the image after " - "the project sweep, so this file is overwritten by CanyonOS Core's own " - f"{entry}. Rename it or move it into a package directory.", - ) - - # V020 -- a stub is copied flat under its yaml's basename. - entrypoint_basenames = {os.path.basename(e) for e in entrypoints if e} - for yaml_path in yaml_paths: - stem = os.path.splitext(os.path.basename(yaml_path))[0] - module = f"{stem}.py" - if module in entrypoint_basenames: - continue # the entrypoint is copied last and wins its flat name back - candidate = os.path.join(project_dir, module) - if os.path.isfile(candidate): - report.error( - "V020", - candidate, - 1, - f"`{os.path.basename(yaml_path)}` generates a stub that lands on " - f"`{module}`", - "The yaml's basename names the stub, and the stub is copied flat " - "over the swept tree. Anything importing this module inside the " - "container gets the generated stub instead of the real code. " - "Rename the yaml to match its own entrypoint.", - ) - - -# ------------------------------------------------------------------ # -# V030-V031 capability-gated rules # -# ------------------------------------------------------------------ # - - -def check_env_file(report, config, config_path, project_dir): - """V030 -- gated on detected env-file injection support.""" - declared = config.get("env_file") - supported = report.capabilities.get("env_file") - - if not supported: - if declared: - report.error( - "V030", - config_path, - line_of(config, "env_file"), - f"`env_file: {declared}` is set, but this CanyonOS Core never reads it", - "No resolve_env_file in the importable ventis package, so the " - "key is silently dropped and the container answers a provider " - "credential error on the first request. This port requires the " - "`env_file` runtime capability.", - ) - else: - report.unavailable( - "V030", - "env_file is not supported by the importable `ventis` runtime. " - "Credentials have no declared path into a container on this tree.", - ) - return - - if not declared: - report.warn( - "V030", - config_path, - line_of(config), - "no `env_file:` in the config", - "Only runtime-managed VENTIS_* variables are guaranteed without it. " - "If the source reads credentials from the environment, the first " - "request fails on a provider error.", - ) - return - - # Path existence and readability are deploy-preflight checks. Do not - # duplicate them here. - - -def check_import_root(report, project_dir, entrypoint_paths): - """V031 -- gated on detected editable-install support.""" - supported = report.capabilities.get("editable_install") - has_metadata = any( - os.path.isfile(os.path.join(project_dir, name)) - for name in ("pyproject.toml", "setup.py", "setup.cfg") - ) - - non_flat = [] - for path in entrypoint_paths: - tree, _ = parse_python(path) - if tree is None: - continue - for name, lineno in toplevel_import_names(tree).items(): - if name in report.stub_module_names or f"{name}.py" in RUNTIME_FLAT_NAMES: - continue - if _resolves_flat(project_dir, name): - continue - location = _resolves_nested(project_dir, name) - if location: - non_flat.append((path, lineno, name, location)) - - if not supported: - report.unavailable( - "V031", - "the editable install (`-e .`) is not supported by the importable " - "`ventis` runtime. Only names rooted at /app import inside a container.", - ) - for path, lineno, name, location in non_flat: - report.error( - "V031", - path, - lineno, - f"`import {name}` resolves to {location}, which is not at the " - "project root", - "sys.path[0] is /app and this CanyonOS Core runs no editable install, " - "so only modules swept to the root import. The adapter raises " - "ModuleNotFoundError inside _load_agent and the first request " - "answers 'No agent loaded'.", - ) - return - - if non_flat and not has_metadata: - for path, lineno, name, location in non_flat: - report.error( - "V031", - path, - lineno, - f"`import {name}` resolves to {location}, and the project root " - "has no packaging metadata", - "A pyproject.toml, setup.py or setup.cfg at the port root is " - "what adds `-e .`; metadata nested inside the untouched source " - "tree is ignored. Add minimal root metadata pointing at the " - "existing package directory. Without it the install is skipped " - "silently.", - ) - - -def _resolves_flat(project_dir, name): - """Whether Python can resolve `name` with /app as its import root. - - A directory does not need __init__.py: PEP 420 namespace packages resolve - from sys.path just like regular packages. - """ - return os.path.isfile(os.path.join(project_dir, f"{name}.py")) or os.path.isdir( - os.path.join(project_dir, name) - ) - - -def _resolves_nested(project_dir, name): - """Where below /app `name` lives but cannot resolve as a top-level name.""" - for root, dirs, files in os.walk(project_dir): - dirs[:] = [d for d in dirs if not d.startswith(".") and d != "__pycache__"] - if root == project_dir: - continue - if f"{name}.py" in files: - return os.path.relpath(os.path.join(root, f"{name}.py"), project_dir) - if name in dirs: - return os.path.relpath(os.path.join(root, name), project_dir) - return None - - -# ------------------------------------------------------------------ # -# W003, W006 secrets and imports a green build does not reject # -# ------------------------------------------------------------------ # - - -def check_secrets(report, port_paths): - """W003 -- env_file is the way in; nothing else is.""" - for path in port_paths: - try: - with open(path, "r", encoding="utf-8") as handle: - lines = handle.read().splitlines() - except OSError: - continue - for number, line in enumerate(lines, start=1): - for pattern, description in SECRET_PATTERNS: - if pattern.search(line): - report.warn( - "W003", - path, - number, - f"this line looks like {description}", - "Never put a secret in the source tree or the build " - "context. The build sweeps the project into every " - "image.", - ) - break - - tree, _ = parse_python(path) - if tree is None: - continue - for node in ast.walk(tree): - if not isinstance(node, ast.Assign): - continue - if not ( - isinstance(node.value, ast.Constant) - and isinstance(node.value.value, str) - and node.value.value.strip() - ): - continue - for target in node.targets: - if isinstance(target, ast.Name) and SECRET_NAME.search(target.id): - report.warn( - "W003", - path, - node.lineno, - f"`{target.id}` is assigned a literal string", - "Read it from the environment instead; the build sweeps " - "this file into every image.", - ) - - -def _pyproject_dependencies(project_dir): - """What `-e .` installs alongside `requirements:`, or None if unreadable. - - None and the empty set mean different things here: empty means the project - declares no dependencies, None means we could not find out -- a setup.py, or - a tomllib this interpreter does not have. The caller must not treat the - second as the first, or it warns about imports the install would satisfy. - """ - path = os.path.join(project_dir, "pyproject.toml") - if not os.path.isfile(path): - return None - try: - import tomllib - except ImportError: # < 3.11 - return None - try: - with open(path, "rb") as handle: - data = tomllib.load(handle) - except Exception: # noqa: BLE001 - malformed metadata is uv's error to give - return None - deps = (data.get("project") or {}).get("dependencies") - if not isinstance(deps, list): - return None - return {_normalize_distribution(d) for d in deps if isinstance(d, str)} - - -def check_requirements_coverage( - report, project_dir, entry, entrypoint_path, config_path -): - """W006 -- an import the container cannot satisfy.""" - tree, _ = parse_python(entrypoint_path) - if tree is None: - return - - declared = { - _normalize_distribution(item) - for item in (entry.get("requirements") or []) - if isinstance(item, str) - } - - # Where the editable install exists, `-e .` resolves the project's own - # [project.dependencies] in the same pass as `requirements:`. Warning about - # those is a false positive, and a false warning about a dependency is worse - # than none: it teaches the reader to dismiss this check. - editable = report.capabilities.get("editable_install") - metadata = any( - os.path.isfile(os.path.join(project_dir, name)) - for name in ("pyproject.toml", "setup.py", "setup.cfg") - ) - unreadable_metadata = False - if editable and metadata: - project_deps = _pyproject_dependencies(project_dir) - if project_deps is None: - unreadable_metadata = True - else: - declared |= project_deps - base = {_normalize_distribution(item) for item in BASE_AGENT_REQUIREMENTS} - stdlib = getattr(sys, "stdlib_module_names", frozenset()) - - for name, lineno in sorted(toplevel_import_names(tree).items()): - if name in stdlib or name == "ventis": - continue - # Provided by the image itself: the shared runtime is copied flat, and - # every agents/*.yaml generates a stub that is copied flat too. - if f"{name}.py" in RUNTIME_FLAT_NAMES or name in report.stub_module_names: - continue - if _resolves_flat(project_dir, name) or _resolves_nested(project_dir, name): - continue - distribution = _normalize_distribution(IMPORT_TO_DISTRIBUTION.get(name, name)) - if distribution in base or distribution in declared: - continue - if unreadable_metadata: - mechanism = ( - "The container installs the base list, `requirements:`, and -- " - "since this project declares packaging metadata -- whatever " - "`-e .` resolves from it. That metadata could not be read here, " - f"so if it already requires `{name}` this line is noise; " - "otherwise it is a ModuleNotFoundError inside _load_agent and " - "'No agent loaded' on the first request." - ) - else: - mechanism = ( - "The container installs the base list plus `requirements:` and " - "nothing else, so this is a ModuleNotFoundError inside " - "_load_agent and 'No agent loaded' on the first request. If the " - f"distribution is named something other than `{name}`, declare " - f"that name in {report.rel(config_path)}." - ) - report.warn( - "W006", - entrypoint_path, - lineno, - f"`import {name}` is in neither the runtime's base list nor this " - "entry's `requirements:`", - mechanism, - ) - - -def _normalize_distribution(name): - return re.split(r"[<>=!\[;\s]", name.strip().lower(), maxsplit=1)[0].replace( - "_", "-" - ) - - -# ------------------------------------------------------------------ # -# Driver # -# ------------------------------------------------------------------ # - - -def validate(project_dir, config_path, capabilities): - """Inspect only failures hidden behind a successful image build.""" - report = Report(project_dir, capabilities) - - # The build owns config/YAML syntax and shape validation. We read only enough - # valid structure to locate code for the deeper checks below. - config, error = load_yaml(config_path) - if error is not None or not isinstance(config, dict): - report.unavailable( - "BUILD", - "runtime preflight skipped because the config cannot be read; " - "ventis build owns and reports this error.", - ) - return report - - import glob - - yaml_paths = sorted(glob.glob(os.path.join(project_dir, "agents", "*.yaml"))) - report.stub_module_names = { - os.path.splitext(os.path.basename(path))[0] for path in yaml_paths - } - - agents_by_name = {} - stub_classes = {} - for path in yaml_paths: - data, yaml_error = load_yaml(path) - agent = data.get("agent") if isinstance(data, dict) else None - name = agent.get("name") if isinstance(agent, dict) else None - if yaml_error is not None or not isinstance(name, str): - continue # ventis build reports malformed agent declarations - agents_by_name[name] = (path, agent) - stub_classes[os.path.splitext(os.path.basename(path))[0]] = name - - entries = config.get("agents") - if not isinstance(entries, list): - report.unavailable( - "BUILD", - "runtime preflight skipped because `agents:` is not a list; " - "ventis build owns and reports this error.", - ) - return report - - entrypoints = [] - for entry in entries: - if not isinstance(entry, dict) or entry.get("type", "agent") == "workflow": - continue - name = entry.get("name") - entrypoint = entry.get("entrypoint") - if isinstance(entrypoint, str): - entrypoints.append(entrypoint) - if name in agents_by_name: - yaml_path, agent_block = agents_by_name[name] - check_adapter(report, yaml_path, agent_block, entry, project_dir) - entrypoint_path = os.path.join(project_dir, entrypoint or "") - if isinstance(entrypoint, str) and os.path.isfile(entrypoint_path): - check_requirements_coverage( - report, project_dir, entry, entrypoint_path, config_path - ) - - for entry in entries: - if not isinstance(entry, dict) or entry.get("type", "agent") != "workflow": - continue - workflow_file = entry.get("workflow_file") - if not isinstance(workflow_file, str): - continue - workflow_path = os.path.join(project_dir, workflow_file) - if os.path.isfile(workflow_path): - check_workflow(report, workflow_path, stub_classes) - - # These survive a green build and otherwise surface only in a container or - # on its first request. - check_flat_collisions(report, project_dir, yaml_paths, entrypoints) - check_env_file(report, config, config_path, project_dir) - - entrypoint_paths = [ - os.path.join(project_dir, e) - for e in entrypoints - if os.path.isfile(os.path.join(project_dir, e)) - ] - check_import_root(report, project_dir, entrypoint_paths) - - port_paths = list(entrypoint_paths) - for entry in entries: - if isinstance(entry, dict) and isinstance(entry.get("workflow_file"), str): - candidate = os.path.join(project_dir, entry["workflow_file"]) - if os.path.isfile(candidate): - port_paths.append(candidate) - - # Secret detection remains because a green image build would permanently - # bake the credential into every image. - check_secrets(report, port_paths) - return report - - - -# ------------------------------------------------------------------ # -# Output # -# ------------------------------------------------------------------ # - -LEVEL_ORDER = {ERROR: 0, WARN: 1, INFO: 2} - - -def _wrap(text, width, indent): - words = text.split() - lines = [] - current = "" - for word in words: - candidate = f"{current} {word}".strip() - if len(candidate) + len(indent) > width and current: - lines.append(indent + current) - current = word - else: - current = candidate - if current: - lines.append(indent + current) - return lines - - -def print_report(report, project_dir): - caps = report.capabilities - if not caps.get("ventis"): - print("ventis is not importable here -- capability-gated rules are") - print("reported UNAVAILABLE rather than checked.\n") - else: - print("CanyonOS Core capabilities detected:") - for key, source in CAPABILITY_SOURCE.items(): - mark = "yes" if caps.get(key) else "no " - print(f" {mark} {key:<22} {source}") - print() - - findings = sorted( - report.findings, - key=lambda f: (LEVEL_ORDER[f["level"]], f["check"], f["path"], f["line"]), - ) - for finding in findings: - where = finding["path"] - if where and finding["line"]: - where = f"{where}:{finding['line']}" - header = f"{finding['check']} {finding['level']:<5}" - print(f"{header} {where}" if where else header) - for line in _wrap(finding["summary"], 78, " "): - print(line) - if finding["mechanism"]: - for line in _wrap(finding["mechanism"], 78, " "): - print(line) - print() - - errors, warnings = report.counts() - if not findings: - print(f"{project_dir}: clean.") - return - print(f"{errors} error(s), {warnings} warning(s).") - - -def main(argv=None): - parser = argparse.ArgumentParser( - description="Check a CanyonOS Core port against the rules in SKILL.md." - ) - parser.add_argument( - "project_dir", nargs="?", default=".", help="the port's project root" - ) - parser.add_argument( - "-c", - "--config", - default=DEFAULT_CONFIG_PATH, - help=f"config path relative to project_dir (default: {DEFAULT_CONFIG_PATH})", - ) - parser.add_argument("--json", action="store_true", help="emit findings as JSON") - parser.add_argument( - "--strict", action="store_true", help="fail on warnings as well as errors" - ) - args = parser.parse_args(argv) - - project_dir = os.path.abspath(args.project_dir) - config_path = ( - args.config - if os.path.isabs(args.config) - else os.path.join(project_dir, args.config) - ) - - capabilities = probe_capabilities() - report = validate(project_dir, config_path, capabilities) - errors, warnings = report.counts() - - if args.json: - print( - json.dumps( - { - "project_dir": project_dir, - "capabilities": capabilities, - "errors": errors, - "warnings": warnings, - "findings": report.findings, - }, - indent=2, - ) - ) - else: - print_report(report, report.rel(project_dir) or project_dir) - - if errors or (args.strict and warnings): - return 1 - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/.claude/skills/porting-to-canyonos/SKILL.md b/.claude/skills/porting-to-canyonos/SKILL.md new file mode 100644 index 0000000..5c23ad6 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/SKILL.md @@ -0,0 +1,68 @@ +--- +name: porting-to-canyonos +description: Port existing Python agents—including LangChain, LangGraph, CrewAI, AutoGen, and custom implementations—to CanyonOS Core. Use for migrations, `.car` packaging, adapter and workflow generation, validation, deployment, or diagnosing build and runtime failures. Creates and validates a self-contained `.car`, then requires explicit approval before `canyonos deploy`. +--- + +# Port an agent project to CanyonOS + +Requires Python, Docker, and the `canyonos` CLI. `prepare.py` uses the Python +standard library; `validate.py` also requires `pyyaml`. + +## Progress + +Copy this checklist into the response and update it while working: + +```text +Port progress: +- [ ] 1. Prepare `.car` +- [ ] 2. Survey the copy and choose service boundaries +- [ ] 3. Write adapters, workflow, declarations, and reviewed configuration +- [ ] 4. Gap validation exits 0; report readiness and stop +``` + +## 1. Prepare `.car` + +Read [references/preparation.md](references/preparation.md) in full. Choose the +import root from actual imports and use `prepare.py`; do not assemble or refresh +`.car` manually. After preparation, edit only `.car`. + +## 2. Survey and design + +Read [references/source-survey.md](references/source-survey.md) in full. Produce +its survey record and choose the smallest useful service map before writing +runtime code or configuration. + +## 3. Implement the port + +Read [references/adapter.md](references/adapter.md) before changing `.car/app` +and [references/manifest.md](references/manifest.md) before changing +`.car/config`. Keep this binding exact: + +```text +config entry name == yaml agent.name == entrypoint class name +``` + +Use the View/Change configuration flow in `manifest.md`; prefer +`canyonos config` in an interactive terminal. Preserve the source-integrity +boundary defined in `preparation.md`. + +Read these only when triggered: + +- [references/llm-proxy.md](references/llm-proxy.md) when the target uses + `llm_proxy`. +- [references/ec2.md](references/ec2.md) when any entry uses `provider: EC2`. + +## 4. Gap validation and stop + +Read +[references/validation-and-deploy.md](references/validation-and-deploy.md). +Validate only authored contracts that CanyonOS tooling does not strongly +guarantee. Fix every reported error, hand off warnings and blockers, then stop. +Do not run `canyonos deploy` without explicit user approval. + +## Diagnose an approved deployment + +After an explicitly approved deployment fails, start with +[references/troubleshooting.md](references/troubleshooting.md). Read +[references/runtime-contract.md](references/runtime-contract.md) only when a +runtime mechanism or validator finding needs explanation. diff --git a/.claude/skills/porting-to-canyonos/prepare.py b/.claude/skills/porting-to-canyonos/prepare.py new file mode 100755 index 0000000..963b0c5 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/prepare.py @@ -0,0 +1,363 @@ +#!/usr/bin/env python3 +"""Create a CanyonOS Core artifact tree from a chosen Python import root. + +The script owns the mechanical part of step 1: it creates ``.car/config`` and +copies the selected import root to ``.car/app`` with development artifacts and +credential files excluded. Choosing the correct import root still requires +reading the application's imports. +""" + +import argparse +import hashlib +import json +import os +import shutil +import sys +import uuid +from pathlib import Path, PurePosixPath + +EXCLUDED_DIRECTORIES = frozenset( + { + ".car", + ".git", + ".hg", + ".mypy_cache", + ".pytest_cache", + ".ruff_cache", + ".tox", + ".venv", + ".svn", + "__pycache__", + "build", + "dist", + "env", + "htmlcov", + "node_modules", + "venv", + } +) +EXCLUDED_FILE_SUFFIXES = (".pyc", ".pyo") +ENV_TEMPLATES = frozenset({".env.example", ".env.sample", ".env.template"}) +STATE_FILENAME = ".porting-state.json" + + +def _is_relative_to(path: Path, parent: Path) -> bool: + try: + path.relative_to(parent) + except ValueError: + return False + return True + + +def _reject_symlinks(import_root: Path) -> None: + """Keep the copied artifact independent of paths outside the copy.""" + for path in import_root.rglob("*"): + if path.is_symlink(): + relative = path.relative_to(import_root) + raise ValueError( + f"import root contains a symbolic link: {relative}; replace it " + "with the intended file or directory before preparing the port" + ) + + +def _ignore_factory(artifact_root: Path): + def ignore(directory: str, names: list[str]) -> set[str]: + directory_path = Path(directory) + ignored = set() + for name in names: + path = directory_path / name + if path.resolve() == artifact_root: + ignored.add(name) + elif path.is_dir() and name in EXCLUDED_DIRECTORIES: + ignored.add(name) + elif name.startswith(".env") and name not in ENV_TEMPLATES: + ignored.add(name) + elif name.endswith(EXCLUDED_FILE_SUFFIXES): + ignored.add(name) + return ignored + + return ignore + + +def _copy_source(import_root: Path, destination: Path, artifact_root: Path) -> None: + shutil.copytree( + import_root, + destination, + ignore=_ignore_factory(artifact_root), + copy_function=shutil.copy2, + symlinks=True, + ) + + +def _hash_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def _file_hashes(root: Path) -> dict[str, str]: + return { + path.relative_to(root).as_posix(): _hash_file(path) + for path in sorted(root.rglob("*")) + if path.is_file() and not path.is_symlink() + } + + +def _load_state(config_dir: Path) -> dict[str, str]: + state_path = config_dir / STATE_FILENAME + if not state_path.is_file(): + raise ValueError( + f"{state_path} is missing; this artifact predates refresh tracking. " + "Use --force only if discarding all edits in .car/app is intentional" + ) + try: + state = json.loads(state_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as error: + raise ValueError(f"cannot read refresh state {state_path}: {error}") from error + files = state.get("source_files") if isinstance(state, dict) else None + if ( + not isinstance(state, dict) + or state.get("version") != 1 + or not isinstance(files, dict) + or not all(_valid_state_entry(path, digest) for path, digest in files.items()) + ): + raise ValueError(f"invalid refresh state: {state_path}") + return files + + +def _valid_state_entry(path: object, digest: object) -> bool: + if not isinstance(path, str) or not isinstance(digest, str): + return False + relative = PurePosixPath(path) + return ( + path == relative.as_posix() + and not relative.is_absolute() + and path not in ("", ".") + and ".." not in relative.parts + and len(digest) == 64 + and all(character in "0123456789abcdef" for character in digest) + ) + + +def _write_state(path: Path, source_hashes: dict[str, str]) -> None: + path.write_text( + json.dumps( + {"version": 1, "source_files": source_hashes}, + indent=2, + sort_keys=True, + ) + + "\n", + encoding="utf-8", + ) + + +def _refresh_app( + app_dir: Path, + source_copy: Path, + staged_app: Path, + previous_source: dict[str, str], +) -> None: + current = _file_hashes(app_dir) + incoming = _file_hashes(source_copy) + conflicts = [] + + for relative in sorted(previous_source.keys() | incoming.keys() | current.keys()): + old = previous_source.get(relative) + new = incoming.get(relative) + edited = current.get(relative) + source_changed = new != old + app_changed = edited != old + if old is None and new is not None and edited not in (None, new): + conflicts.append(relative) + elif old is not None and source_changed and app_changed and edited != new: + conflicts.append(relative) + + # File/directory replacements need the same three-way protection. A new + # source file at `pkg` must not erase a port-only `pkg/adapter.py`, and a + # new source directory must not silently replace a port-authored file at + # `pkg`. + for incoming_path in incoming: + prefix = incoming_path + "/" + for current_path, edited in current.items(): + if not current_path.startswith(prefix): + continue + if edited != previous_source.get(current_path): + conflicts.append(current_path) + for current_path, edited in current.items(): + prefix = current_path + "/" + if any(incoming_path.startswith(prefix) for incoming_path in incoming): + if edited != previous_source.get(current_path): + conflicts.append(current_path) + + if conflicts: + conflicts = sorted(set(conflicts)) + shown = "\n ".join(conflicts[:20]) + suffix = "" if len(conflicts) <= 20 else f"\n ... and {len(conflicts) - 20} more" + raise ValueError( + "refresh found files changed in both the source and .car/app:\n " + f"{shown}{suffix}\nResolve them in .car/app, then update the source " + "or use --force only to discard all port edits" + ) + + shutil.copytree(app_dir, staged_app, copy_function=shutil.copy2, symlinks=True) + + # Apply safe source deletions first so file-to-directory changes have room. + for relative, old in previous_source.items(): + if relative in incoming or current.get(relative) != old: + continue + target = staged_app / relative + if target.is_file(): + target.unlink() + + for directory in sorted( + (path for path in staged_app.rglob("*") if path.is_dir()), + key=lambda path: len(path.parts), + reverse=True, + ): + try: + directory.rmdir() + except OSError: + pass + + for relative, new in incoming.items(): + old = previous_source.get(relative) + edited = current.get(relative) + if new == old or edited == new: + continue + if old is not None and edited != old: + continue + target = staged_app / relative + if target.is_dir(): + shutil.rmtree(target) + target.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(source_copy / relative, target) + + +def prepare( + import_root: Path, + artifact_root: Path, + force: bool = False, + refresh: bool = False, +) -> None: + import_root = import_root.expanduser().resolve() + artifact_root = artifact_root.expanduser().resolve() + app_dir = artifact_root / "app" + config_dir = artifact_root / "config" + + if not import_root.is_dir(): + raise ValueError(f"import root is not a directory: {import_root}") + if import_root == artifact_root: + raise ValueError("artifact root cannot also be the import root") + if _is_relative_to(import_root, artifact_root): + raise ValueError("import root cannot be inside the artifact root") + if app_dir.exists() and not app_dir.is_dir(): + raise ValueError(f"app path exists but is not a directory: {app_dir}") + if force and refresh: + raise ValueError("--force and --refresh are mutually exclusive") + if refresh and not app_dir.is_dir(): + raise ValueError(f"cannot refresh because {app_dir} does not exist") + if app_dir.exists() and not force and not refresh: + raise FileExistsError( + f"{app_dir} already exists; use --refresh to preserve port edits, or " + "--force to discard and replace the entire source copy" + ) + if config_dir.exists() and not config_dir.is_dir(): + raise ValueError(f"config path exists but is not a directory: {config_dir}") + + _reject_symlinks(import_root) + if refresh: + _reject_symlinks(app_dir) + + artifact_root.mkdir(parents=True, exist_ok=True) + config_dir.mkdir(exist_ok=True) + transaction_id = uuid.uuid4().hex + source_copy = artifact_root / f".source-{transaction_id}.tmp" + temporary_app = artifact_root / f".app-{transaction_id}.tmp" + previous_app = artifact_root / f".app-{uuid.uuid4().hex}.previous" + temporary_state = config_dir / f".{STATE_FILENAME}-{transaction_id}.tmp" + state_path = config_dir / STATE_FILENAME + + installed_new_app = False + try: + _copy_source(import_root, source_copy, artifact_root) + source_hashes = _file_hashes(source_copy) + if refresh: + previous_source = _load_state(config_dir) + _refresh_app(app_dir, source_copy, temporary_app, previous_source) + shutil.rmtree(source_copy) + else: + source_copy.rename(temporary_app) + _write_state(temporary_state, source_hashes) + if app_dir.exists(): + app_dir.rename(previous_app) + temporary_app.rename(app_dir) + installed_new_app = True + os.replace(temporary_state, state_path) + except Exception: + if source_copy.exists(): + shutil.rmtree(source_copy) + if temporary_app.exists(): + shutil.rmtree(temporary_app) + if temporary_state.exists(): + temporary_state.unlink() + if installed_new_app and app_dir.exists(): + shutil.rmtree(app_dir) + if previous_app.exists(): + previous_app.rename(app_dir) + raise + + if previous_app.exists(): + shutil.rmtree(previous_app) + + +def parse_args(argv=None): + parser = argparse.ArgumentParser( + description="Create .car/config and copy an import root into .car/app." + ) + parser.add_argument( + "import_root", + help="directory whose contents should become the contents of .car/app", + ) + parser.add_argument( + "artifact_root", + nargs="?", + default=".car", + help="artifact directory to create (default: .car)", + ) + parser.add_argument( + "--force", + action="store_true", + help="discard edits and replace an existing app/ copy; leave config/ unchanged", + ) + parser.add_argument( + "--refresh", + action="store_true", + help="update unchanged source files while preserving port-authored edits", + ) + return parser.parse_args(argv) + + +def main(argv=None) -> int: + args = parse_args(argv) + try: + prepare( + Path(args.import_root), + Path(args.artifact_root), + force=args.force, + refresh=args.refresh, + ) + except (OSError, ValueError) as error: + sys.stderr.write(f"prepare.py: {error}\n") + return 1 + + artifact_root = Path(args.artifact_root) + print(f"Created {artifact_root / 'config'}") + action = "Refreshed" if args.refresh else "Copied" + print(f"{action} {Path(args.import_root)} to {artifact_root / 'app'}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/.claude/skills/porting-to-canyonos/references/adapter.md b/.claude/skills/porting-to-canyonos/references/adapter.md new file mode 100644 index 0000000..da4417e --- /dev/null +++ b/.claude/skills/porting-to-canyonos/references/adapter.md @@ -0,0 +1,107 @@ +# Implement runtime code in `.car/app` + +**When:** after selecting service boundaries, before writing an adapter or +workflow. + +**Output:** one loadable adapter per service and one workflow exposing +`main(query: str)`. + +Use this order: + +1. Choose a safe entrypoint module for each service. +2. Write a no-argument synchronous adapter around source-owned behavior. +3. Bridge async or session state only when the source requires it. +4. Write the workflow and preserve parallel dispatch. + +Complete `manifest.md`, then validate only the authored contracts that CanyonOS +does not already guarantee. + +These rules are required even when static checks pass; violations often appear +only when a container loads. + +## Contents + +- Adapter and workflow shape +- Choosing the entrypoint +- Bridging async +- Multi-turn and session state + +## Adapter and workflow shape + +Write a no-argument, synchronous adapter class at the selected entrypoint. +Import source-owned behavior instead of duplicating it. The workflow exposes +`main(query: str)`, imports every service from its exact entrypoint module, and +calls `deploy(main, port=...)` at module scope. Do not add a main guard: the +workflow executes as `__main__` in production. + +For parallel remote calls, dispatch all work before resolving any result: + +```python +futures = [agent.work(item=item) for item in items] +results = [json.loads(future.value()) for future in futures] +``` + +Combining dispatch and `.value()` in one comprehension serializes the work. + +## Choosing the entrypoint + +The entrypoint is the one module in the copy the build destroys: each agent's +stub is written over its own `entrypoint` path in every image except that +agent's own, so everywhere else that path holds a generated class and nothing +else. Every gate below applies to whichever module you pick; each one +disqualifies it as it stands, so fix the module or point `entrypoint` at +another. + +1. **Does anything else the deployment imports read this module for its real + contents?** The workflow, or any module the workflow imports at module scope. + If yes, the stub replaces it there and the workflow dies at container + startup. Put the adapter in a sibling module that imports this one and point + `entrypoint` at the sibling. This is why "edit the copied module in place" is + a preference and not a rule: it is right only for a module nothing else + imports. +2. **Does the module's package `__init__.py` re-export a name from it?** + (`from .graph import graph`) Python runs `__init__.py` before any submodule, + so every peer image that touches that package re-runs a re-export the stub + cannot satisfy and raises ImportError at startup. Point `entrypoint` at a + module the `__init__` does not re-export from; add one if it re-exports from + all of them. V033. +3. **Is every segment of the path a Python identifier?** `travel-planner.py` and + `steps/06_agent.py` load fine -- the controller loads by file path -- but the + workflow's `from steps.06_agent import X` is a SyntaxError, which no import + guard catches. Rename the file inside the copy, or add a normally-named + sibling that loads it by path and re-exposes the class. V034. +4. **Does the module use relative imports?** (`from . import data_service`) The + controller loads the entrypoint with `spec_from_file_location`, which leaves + `__package__` empty, so every relative import *in the entrypoint itself* + fails at agent load. Make its top-level imports absolute; the modules it + imports keep theirs. V035. +5. **Does module-level code perform a real run?** A script ending in + `result = crew.kickoff(...)` / `print(result)` fires that run whenever the + module loads, before a request exists. Delete the + invocation and keep the construction. The source-integrity boundary in + `preparation.md` protects prompts, tools, schemas, model calls, and node + bodies—not a script's own main body. + +## Bridging async + +Use one `asyncio.run(...)` per declared method, at the method boundary, around +the whole call. One per awaited coroutine builds a fresh event loop and a fresh +connection pool per graph superstep. + +If the instance holds anything bound to a loop -- an `asyncio.Lock`, a client +constructed inside a coroutine -- `asyncio.run` cannot be used at all. The +runtime calls declared methods repeatedly on one instance, and the second call +raises `Lock is bound to a different event loop`. Run one persistent loop on a +background thread and submit with `run_coroutine_threadsafe`. Seed any +`ContextVar` the source's async code reads on the calling thread immediately +before submitting: `call_soon_threadsafe` copies the context at schedule time, +not inside the loop. + +## Multi-turn and session state + +The platform sends one `{query: string}` per request and keeps nothing between +them. A source with per-conversation state -- a `thread_id`, a checkpointer, a +memory keyed by session -- carries that id *inside* `query`: accept either a +bare string or a JSON object in that one field and pass the id through to the +source unchanged. Do not add a second workflow parameter for it; the platform +never sends one. diff --git a/.claude/skills/porting-to-canyonos/references/ec2.md b/.claude/skills/porting-to-canyonos/references/ec2.md new file mode 100644 index 0000000..6dbdfe2 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/references/ec2.md @@ -0,0 +1,51 @@ +# Configure EC2 deployment + +**When:** at least one config entry uses `provider: EC2`. + +**Output:** developer-supplied EC2 settings, reachable service addresses, and a +safe remote cleanup plan. + +## Configuration + +Use `provider: EC2` and declare `instance_type` on every EC2 service entry. The +top-level `ec2` block supplies the runtime's required infrastructure and SSH +settings. Read the target checkout's deploy preflight and EC2 runtime before +writing the block; do not copy values from an example environment. + +These identifiers come from the developer during the configuration review; +they are the one part of the manifest with no safe default. If the round produces no +answer, leave the entry `provider: local` and report that EC2 was requested but +not configured. Never fill the block from an example, a previous port, or +another entry in the same manifest. + +Typical required categories are: + +- AMI and instance type +- region and subnet +- security groups +- SSH user and credentials accepted by the runtime + +`canyonos deploy` owns basic EC2 config validation. A preflight pass is not proof +that provisioning, SSH, image transfer, or remote container startup works. + +## Networking + +A remote container's `host.docker.internal` names its own EC2 Docker host. It +does not name the local controller machine. Databases, model proxies, and other +services must use addresses reachable from every selected host. + +The environment file may be copied temporarily to a remote host by runtimes that +expose the `env_file` capability. Confirm behavior from the capability probe and +target runtime rather than assuming local Docker semantics. + +## Deployment and cleanup + +After the user explicitly approves `canyonos deploy`, verify the remote +container logs; controller health can be green even when agent loading failed. +Do not start a separate build or deployment as part of validation. + +Ctrl+C stops CLI log monitoring, not necessarily the deployment. Ask before +running `canyonos stop` so the controller can terminate recorded EC2 instances. +If provisioning or startup fails before an instance is recorded, inspect the +cloud provider directly and remove exact leaked resources. Never use a broad +cleanup command against unrelated instances. diff --git a/.claude/skills/porting-to-canyonos/references/llm-proxy.md b/.claude/skills/porting-to-canyonos/references/llm-proxy.md new file mode 100644 index 0000000..66d76db --- /dev/null +++ b/.claude/skills/porting-to-canyonos/references/llm-proxy.md @@ -0,0 +1,122 @@ +# Route model calls through `llm_proxy` + +**When:** the target contains `llm_proxy` or deployment explicitly routes model +SDKs through it. + +**Output:** provider-preserving proxy environment settings, verified routing, +and a clear blocker for unsupported call shapes. + +## Contents + +- Preserve provider protocols +- Set every spelling, not the one you expect +- A source with no env hook cannot be proxied +- Start locally +- Supported call shape +- Credential behavior + +## Preserve provider protocols + +The proxy redirects provider endpoints; it does not convert providers. Keep the +source SDK, model ID, request body, and response parsing unchanged. + +## Set every spelling, not the one you expect + +Each SDK generation reads a different base-URL variable, and a wrapper library +reads a different one from the SDK it wraps. Set only the name this reference +used to give and the container reaches the real provider with a placeholder key: +a 401 that reads like a broken port, after validation and the deployment build +have passed. Set all of them for whichever providers the source uses: + +```dotenv +OPENAI_BASE_URL=http://host.docker.internal:8081/openai/v1 +OPENAI_API_BASE=http://host.docker.internal:8081/openai/v1 +ANTHROPIC_BASE_URL=http://host.docker.internal:8081/anthropic +ANTHROPIC_API_URL=http://host.docker.internal:8081/anthropic +ANTHROPIC_API_BASE=http://host.docker.internal:8081/anthropic +AWS_ENDPOINT_URL_BEDROCK_RUNTIME=http://host.docker.internal:8081/bedrock +``` + +Which name actually wins, for when a call still escapes: + +| Caller | Reads | +|---|---| +| `openai` SDK | `OPENAI_BASE_URL` | +| `langchain_openai` | `OPENAI_API_BASE` | +| `llama_index.llms.openai`, `llama_index.embeddings.openai` | `OPENAI_API_BASE` only -- `resolve_openai_credentials()` never looks at `OPENAI_BASE_URL` | +| `anthropic` SDK | `ANTHROPIC_BASE_URL` | +| `langchain_anthropic` | `ANTHROPIC_API_URL` first, `ANTHROPIC_BASE_URL` as fallback | +| LiteLLM | `ANTHROPIC_API_BASE` | + +Confirm the route rather than assuming it: the proxy logs one line per forwarded +call, so an empty proxy log after a successful request means the container went +straight to the provider. + +Some SDKs refuse to initialize without caller credentials. Give agent containers +non-secret placeholders only when required. Keep real OpenAI, Anthropic, or AWS +credentials in the separate proxy process, not in the port's `env_file`. + +## A source with no env hook cannot be proxied + +Some sources build the HTTP call themselves -- `urllib.request` against a module +constant like `API = "https://api.openai.com/v1/responses"` -- and read no +base-URL variable at all. Editing that constant swaps the source provider's +endpoint, which the source-integrity boundary in +[preparation.md](preparation.md) forbids, so the `env_file` is inert and the container can only ever reach the +real provider. Report this as a proxy blocker and stop. Do not hand the +container a real upstream credential instead. + +Detect it before deploying: grep the source for the provider hostname. A literal +`api.openai.com` or `api.anthropic.com` outside a comment means the call bypasses +the SDK's base-URL resolution entirely. + +## Start locally + +The proxy defaults conflict with a typical deployment: host loopback is not +reachable from a container, and port 8080 is normally used by the workflow API. +Use a non-loopback bind and a different port: + +```bash +PROXY_HOST=0.0.0.0 PROXY_PORT=8081 python -m llm_proxy +curl http://127.0.0.1:8081/healthz +``` + +Local CanyonOS Core containers resolve `host.docker.internal` through their +Docker host mapping. On EC2 that name resolves to each EC2 Docker host, not the +machine running `canyonos deploy`. Distributed deployments need a reachable proxy +address or one proxy on each host. + +## Supported call shape + +The implementation buffers complete requests and responses: + +- OpenAI and Anthropic non-streaming HTTP calls are forwarded. +- Bedrock `invoke` is reissued through the proxy's boto3 client. +- Bedrock `invoke-with-response-stream`, `converse`, and `converse-stream` are + unsupported. + +Streaming splits by how the source **consumes** the response, not by whether a +`streaming` flag is set. The proxy buffers, so it forwards anything that reads a +complete response and breaks anything that reads tokens as they arrive: + +- `ChatOpenAI(streaming=True)` reached through `.invoke()` works. LangChain + drains the stream inside the call and returns one message; the proxy sees an + ordinary buffered request. Verified end to end against this proxy. +- `.stream()`, `.astream()`, and a raw `stream=True` read token by token do not. + +Read the call site before deciding. Report and stop only for the second kind; +never silently disable streaming to make the first kind fit. + +## Credential behavior + +- The OpenAI adapter removes caller authorization and inserts the proxy key. +- The Anthropic adapter removes caller key headers and inserts the proxy key. +- Botocore still signs requests sent to a custom endpoint, so a caller may need + placeholder AWS credentials even though the proxy reissues upstream with its + own identity. +- `/healthz` proves provider registration and Flask availability, not upstream + credential validity. + +OpenAI and Anthropic upstream HTTP errors pass through. Proxy exceptions return +JSON 502 with `error: proxy_error`. Bedrock `ClientError` bodies are reconstructed +with the upstream status and are not byte-for-byte passthrough. diff --git a/.claude/skills/porting-to-canyonos/references/manifest.md b/.claude/skills/porting-to-canyonos/references/manifest.md new file mode 100644 index 0000000..0eb00ac --- /dev/null +++ b/.claude/skills/porting-to-canyonos/references/manifest.md @@ -0,0 +1,192 @@ +# Configure `.car/config` + +**When:** before writing an agent declaration or the deployment manifest. + +**Output:** one declaration per service and one reviewed +`global_controller.yaml`, with requirements derived separately for each image. + +Work in this order: + +1. Derive names, entrypoints, workflow path, and requirements from the service + map and copied import graph. +2. Write each agent declaration. +3. Build the complete manifest candidate with documented defaults. +4. Review developer-owned choices through the View/Change flow. +5. Write the reviewed candidate; validate only cross-file contracts not already + guaranteed by the config flow. + +## Contents + +- Ownership of configuration keys +- Configuration review +- Agent declarations +- Per-image requirements +- Complete manifest shape + +## Ownership of configuration keys + +Two kinds of key share one file. A **derived** key has exactly one right answer +and the copy holds it; asking the developer can only make it worse. A +**developer** key is a deployment choice the source does not contain, and +deriving it means guessing and presenting the guess as a reading. + +The configuration review shows the whole manifest and asks about the second +column only, in one round, carrying these defaults. + +| Key | Decided by | Default when unanswered | +|---|---|---| +| `name`, `entrypoint`, `workflow_file`, `type` | derived — selected service map | — | +| `requirements` | derived — the entry's import graph | — | +| `database` | neither; omit it always (see below) | absent | +| `provider` | developer | `local` | +| `ec2:` block, `instance_type` | developer — no default is safe | entry stays `local` | +| `replicas` | developer, *unless* cross-request state forces `1` | `1` | +| `resources.cpu` / `resources.memory` | developer | `1` / `512` MiB | +| `api_port` | developer | `8080` | +| `redis_port`, `redis.host` / `.port` / `.db` | developer | `6379`, `localhost` / `6379` / `0` | +| `poll_interval` | developer | `5` | +| `env_file` | developer — the file's location and whether it exists | `.env` when the survey found credential reads, else absent | +| `policy.yaml` | developer | absent | + +Two entries in that table are not free choices, and saying so is part of showing +the config rather than asking about it: + +- **`replicas` stops being a choice once a service holds cross-request state.** + When the survey finds such state, the service-boundary section in + `source-survey.md` fixes `replicas: 1` as a correctness requirement. Report it as a constraint; do not offer to + raise it. +- **EC2 identifiers are wrong to invent.** ec2.md forbids copying them from an + example environment, and a wrong AMI, subnet, or security group fails at + deploy preflight or, worse, provisions something unreachable. Unanswered + means the entry stays `local`. + +## Configuration review + +Use the interaction implemented by `canyonos config` before writing +`.car/config/global_controller.yaml`: + +1. Build the complete candidate manifest in memory from derived values and the + defaults above. +2. **View** prints the whole candidate, annotating defaults and source-imposed + constraints such as `replicas: 1` for in-memory state. +3. **Change** asks in one batch only for developer-owned values: provider and + EC2 fields, unconstrained replicas, resources, ports, secret-file location, + and access restrictions. Show each current/default value, apply answers, and + show the result. +4. Write the reviewed candidate. Defer gap validation until runtime code and + configuration are both complete. + +Prefer running `canyonos config` when an interactive terminal is available; +otherwise reproduce View/Change in conversation. Do not ask for derived values +such as entrypoints or requirements. + +An unattended `canyonos integrate` run must not block on this interaction. Use +and report the displayed defaults. Never invent EC2 infrastructure identifiers: +without them, keep the entry `local`. + +## Agent declarations + +Declarations go in `.car/config/`, beside the manifest. The build reads every +`*.yaml` there and keeps the ones with a top-level `agent.name`, so the +manifest and `policy.yaml` drop out on their own. + +Use one yaml per deployed service. Argument types are bare builtins only: +`str`, `int`, `float`, `bool`, `dict`, or `list`. Every declared argument is +required by the generated stub. `returns.type` is documentation; use `dict` or +`list` to signal that workflow callers must `json.loads` the returned string. + +## Per-image requirements + +Each image installs the runtime's base list plus that entry's `requirements:` +and nothing else. The source's own `requirements.txt` is never installed -- the +generator writes its own -- and its `pyproject.toml` is installed only where the +editable-install capability is available. Re-declare every runtime distribution +by hand, per entry. + +Build each entry's list from the imports its image *executes*, not from the code +you wrote: + +1. Follow module-scope imports out of the entrypoint (or `workflow_file`) into + the copy, transitively. A workflow that imports `benchmark.py`, which imports + `agent.py`, needs `agent.py`'s distributions even though the workflow makes + no model call. +2. Include the `__init__.py` of every package on those paths -- it runs first. + In a peer image the entrypoint is a stub, but its package `__init__` and its + siblings are real, so that image still installs what they import. +3. Omit distributions reachable only from source files no image imports, such as + a Gradio or Streamlit UI beside the agent. The source-integrity boundary + forbids reclassifying a declared dependency, not declining to ship an + unreachable one; name what you left out in the report. + +`validate.py` walks the same graph and reports what is missing as W006. + +Version them the way the source resolved them, not the way PyPI resolves them +today: + +- **The source has a lockfile** (`poetry.lock`, `uv.lock`, a pinned + `requirements.txt`): copy those exact versions. Repeating the bare names + resolved `langchain` 1.x for one port, which no longer has + `langchain.agents.agent_toolkits` -- the untouched source's own import. +- **The source pins nothing**: cap every fast-moving distribution below its next + major (`langchain<1.0`, `openai<2`). Unpinned means "whatever existed when + this was written", which is not what pip installs today. +- **The source predates a known SDK break**: pin contemporaneous with its last + commit. A 2023 AutoGen script passing `request_timeout=` needs + `pyautogen==0.1.14`, which depends on `openai<1`, not `autogen==0.7.5`, which + floors on `openai>=1.58` where that kwarg is `timeout`. The source-integrity + boundary forbids rewriting that call, so the pin has to absorb the + difference. Compare the source's + commit date against the pin's release date whenever the source hardcodes SDK + kwargs. + +Resolve the list before writing any adapter -- `uv pip compile`, or +`pip install --dry-run -r` into a scratch environment. A source whose own locked +graph is no longer installable (a yanked release series that an unconditional +transitive pin still requires) is a port blocker; one command finds it instead +of one build-fail/pin/rebuild cycle per attempt. Report it and stop rather than +upgrading the source out of the problem. + +## Complete manifest shape + +`.car/config/global_controller.yaml` in full -- every key the runtime reads, +and no others: + +```yaml +agents: + - name: EmailAgent # == yaml agent.name == entrypoint class name + entrypoint: email_assistant.py # relative to .car/app, may not escape it + provider: local # lowercase; `Local` fails an equality test + replicas: 1 # integer + redis_port: 6379 # host port for this node's Redis; default 6379 + resources: # optional; defaults are cpu 1, memory 512 + cpu: 1 + memory: 1024 # MiB + requirements: # see Requirements above + - langgraph + - langchain-openai + + - name: Workflow + type: workflow # the one entry that carries this key + workflow_file: email_workflow.py # relative to .car/app + api_port: 8080 # where /main is served + provider: local + replicas: 1 + redis_port: 6379 + requirements: # its own list; the agent's does not apply here + - langgraph + +poll_interval: 5 # seconds between metrics polls; default 5 + +redis: + host: localhost + port: 6379 + db: 0 + +env_file: .env # relative to the application root, not .car +``` + +Omit `database`. Without it every metrics poll logs `Could not parse SQLAlchemy +URL from given URL string`, once per replica every `poll_interval` seconds -- +expected noise, not a failure, and not a reason to add the key. Adding it drops +a sqlite file at the application root, outside `.car`, which the final source- +integrity check then reports. diff --git a/.claude/skills/porting-to-canyonos/references/preparation.md b/.claude/skills/porting-to-canyonos/references/preparation.md new file mode 100644 index 0000000..c49eb8e --- /dev/null +++ b/.claude/skills/porting-to-canyonos/references/preparation.md @@ -0,0 +1,263 @@ +# Prepare the `.car` artifact + +**When:** before creating or refreshing `.car`, or when imports and runtime +assets need packaging decisions. + +**Output:** a self-contained `.car/config` and `.car/app`, with the original +application tree untouched and existing imports preserved. + +Use this order: + +1. Read **Artifact model and preparation** and choose the import root from actual + imports. +2. Run `prepare.py`; do not recreate guarantees already enforced by the script. +3. Use **Import roots, metadata, and runtime assets** only when direct `/app` + imports are insufficient or the source opens non-Python files. +4. Use **Refresh an existing source copy** only when `.car/app` already exists. + +## Artifact model and preparation + +### Product and runtime names + +CanyonOS Core is the product name and `canyonos` is its user-facing CLI. The +internal compatibility Python package, environment variables, and Docker +resources retain the `ventis`, `VENTIS_*`, and `ventis-*` names. These are +protocol identifiers, not CLI instructions or branding strings. Do not rename +them, and do not tell users to run the obsolete `ventis` CLI. + +### Artifact boundary + +The port lives entirely inside `.car/`, next to the application source: + +```text +.car/config/global_controller.yaml deployment manifest +.car/config/policy.yaml optional access restriction +.car/config/.yaml one callable surface per service +.car/app/ a copy of the application source +.car/app//.py adapter beside the code it wraps +.car/app//_workflow.py HTTP entry point; calls deploy() +.car/app/pyproject.toml conditional nested-import scaffolding +/ untouched and unaware of the port +``` + +`.car` has exactly two authored directories: `config/`, which holds every +Canyon-owned declaration, and `app/`, which becomes `/app` in every container. +Nothing under `.car` points back into the original source, and nothing in the +original source points at `.car`. Deleting `.car` must restore the project to +its pre-port state. + +Preserve the source's directory structure. Put adapters in the copied module +whose behavior they wrap unless the entrypoint rules require a sibling module; +do not invent generic `agents/` or `workflow/` directories. `canyonos` commands +run from the application root and read `.car` below it. + +The file count follows the deployment: one yaml/adapter pair per independently +deployed service. If a copied source class already satisfies the runtime +contract, point its declaration at that class and do not add an adapter. + +### Prepare the copy + +Choose the source's **import root**, not automatically its repository root. +Without editable-install support, `/app` is the only source entry on +`sys.path`. For example, source under `src/` that says `from tools import ...` +needs the contents of `src/` copied directly into `.car/app/`. Decide from the +source's imports. Continue to **Import roots, metadata, and runtime assets** below when the +copy has those concerns. + +Create the artifact with the skill script rather than ad hoc copy commands: + +```bash +python3 /prepare.py .car +``` + +The script creates `.car/config/` and copies the import root's **contents** to +`.car/app/`. It excludes VCS data, `.car`, virtual environments, caches, build +outputs, bytecode, and credential-bearing `.env*` files while retaining +`.env.example`, `.env.sample`, and `.env.template`. It rejects symbolic links: +they can escape the artifact and may be skipped by runtime source sweeps. + +If `.car/app` already exists, follow **Refresh an existing source copy** below. Use +`--force` only when every edit in `.car/app` may be discarded; it leaves +`.car/config/` unchanged. + +After preparation, edit only `.car` and survey the copy using +[source-survey.md](source-survey.md). Do not add validation for preparation +postconditions that `prepare.py` already guarantees. + +### Source-integrity boundary + +The gap validator owns authored runtime contracts that CanyonOS tooling does +not strongly guarantee. The porter owns constraints static analysis cannot +prove: + +- Never edit outside `.car`, or duplicate source-owned prompts, tools, schemas, + model calls, parsing, retries, and node bodies in an adapter. +- Never swap providers, invent runtime configuration, or silently move, drop, + or reclassify a dependency. +- Rewrite framework control flow only where it crosses a chosen service + boundary; preserve it inside a service. +- Never hardcode or bake a real credential into `.car`. + +When a source defect or unsupported runtime capability requires crossing one of +these boundaries, report the blocker and obtain approval for that specific +change. Do not broaden that approval to unrelated source edits. + +## Import roots, metadata, and runtime assets + +### What `/app` can import + +CanyonOS Core copies `.car/app/` into the image with its paths intact and +starts Python at `/app`, so `/app` is that copy. Without an editable install, +Python resolves names rooted there: + +- `/app/tools.py` as `import tools` +- `/app/pkg/__init__.py` as `import pkg` +- `/app/src/agents/...` as `import src.agents`, including PEP 420 namespace + directories without `__init__.py` + +It does not resolve `/app/source/pkg` as `import pkg`; `/app/source` must become +an import root first. + +### Re-root the copy before reaching for metadata + +`.car/app/` is a copy Canyon owns, so the cheapest fix is usually to root it +where the source already imports from. A project laid out as + +```text +repo/src/email_assistant.py imports `tools`, `prompts`, `utils` +repo/src/tools/ +repo/pyproject.toml +``` + +has `src/` as its import root. Copy `src/`'s contents to `.car/app/` and every +one of those imports resolves from `/app` with no metadata, no editable install +and no `sys.path` hack. `entrypoint` and `workflow_file` then name modules +relative to that root, and the workflow imports the agent the same way. + +Reach for the metadata below only when one copy root cannot serve every import +-- for instance when the source imports both `tools` and `src.tools`. + +### Detect support, do not infer it from release history + +Run: + +```bash +python3 /validate.py .car +``` + +Read the `editable_install` capability. If it is unavailable and the original +import cannot resolve from `/app`, report a runtime capability blocker and stop. +Do not add a `sys.path` hack or relocate source files. + +### Root metadata is the trigger + +When editable install is supported, only packaging metadata at the **root of +the copy** triggers `pip install -e .`: + +```text +.car/app/pyproject.toml detected +.car/app/source/pyproject.toml ignored as an install trigger +``` + +If the application keeps its metadata deeper in the tree, that copy stays where +it is. Add minimal scaffolding at `.car/app/` that points package discovery at +the existing package: + +```toml +[build-system] +requires = ["setuptools>=64"] +build-backend = "setuptools.build_meta" + +[project] +name = "canyonos-port" +version = "0.0.0" +dependencies = [] + +[tool.setuptools.packages.find] +where = ["source/src"] +include = ["pkg*"] +namespaces = true +``` + +Set `where` and `include` from the actual tree and original import spelling. Do +not reference a README or license from this wrapper metadata; file sweeps differ +by runtime capability and a missing referenced file makes the image build fail. + +### Dependencies in nested metadata + +A nested `pyproject.toml` is not installed merely because its Python files are +copied. Keep the source declaration unchanged and repeat its runtime +distributions in each relevant config entry's `requirements` list. This is +compatibility scaffolding, not permission to drop, move, or reclassify declared +dependencies. + +If the application's own metadata already sits at the root of the copy, do not +create a wrapper. Its project dependencies participate in the same resolver as +config requirements. +Report declared-but-unused toolchain dependencies and their image cost; let the +owner decide whether source metadata should change. + +### Runtime data and configuration files + +`prepare.py` copies non-Python files into `.car/app`, but that does not prove the +runtime's image sweep carries them into a container. Inventory every file opened +by the selected import graph: prompt templates, JSON schemas, PDFs, local +corpora, certificates, and framework configuration such as CrewAI +`agents.yaml` and `tasks.yaml`. + +Run `validate.py` and read its `sweeps_all_files` capability: + +- When available, retain each asset at the same path relative to the chosen + import root. Check any path derived from the original repository root or + process working directory; the container starts from `/app`. +- When unavailable, a required non-Python asset is a runtime blocker. Report it + and stop after validation. Do not conceal the gap by base64-encoding the file + into Python, changing a hardcoded path, or duplicating framework config into + adapter code; those changes restate source-owned data and behavior. + +Do not treat successful construction as evidence that configuration loaded. +Frameworks such as CrewAI may warn about a missing yaml and create an empty +configuration, then fail only when the first agent or task is accessed. Inspect +those decorators and file references statically during the survey. + +### Validation boundary + +The build phase of `canyonos deploy` owns packaging syntax and installation +errors. `validate.py` checks only whether adapter imports appear to require a +nested root that the runtime will not expose. + +## Refresh an existing source copy + +Run the same preparation command with `--refresh` and the same import root: + +```bash +python3 /prepare.py .car --refresh +``` + +The initial copy records source-file hashes in +`.car/config/.porting-state.json`. Refresh compares three states: + +```text +previous source hash → current source + ↘ current .car/app +``` + +- Only the source changed: update `.car/app`. +- Only `.car/app` changed: preserve the port edit. +- The source added a path unused by the port: add it. +- The source deleted an unmodified path: delete it from `.car/app`. +- Both sides changed the same path differently: make no changes and report all + conflicts. + +Resolve a conflict in `.car/app`, then either make the source match that result +or intentionally start over. The script does not guess a merge because an +adapter and its source often change for different reasons while sharing one +module. + +`--force` is not refresh. It discards the entire `.car/app` tree and replaces it +with a clean source copy while retaining `.car/config`. Use it only when every +adapter and workflow edit in `.car/app` is intentionally disposable. + +After refresh, survey changed imports, dependencies, runtime assets, and service +boundaries again. Run gap validation for affected authored contracts, but do not +revalidate merge mechanics already guaranteed by the successful atomic refresh. diff --git a/.claude/skills/porting-to-canyonos/references/runtime-contract.md b/.claude/skills/porting-to-canyonos/references/runtime-contract.md new file mode 100644 index 0000000..49d26d1 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/references/runtime-contract.md @@ -0,0 +1,241 @@ +# Explain the CanyonOS Core runtime contract + +**When:** a validator finding needs explanation, runtime behavior is unclear, +or an approved deployment fails in a way the troubleshooting table attributes +to the runtime. + +**Purpose:** explain discovery, generated stubs, loading, execution, image +assembly, and cleanup. This is diagnostic background; implementation rules live +in `adapter.md`, `manifest.md`, and `preparation.md`. + +The product is CanyonOS Core and its user-facing CLI is `canyonos`. +Compatibility identifiers remain `ventis` for the internal Python package, +`VENTIS_*` for runtime variables, and `ventis-*` for Docker resources. +Runtime-dependent behavior is expressed as capabilities; run `validate.py` +against the target environment instead of inferring support from release +history. + +## Contents + +- Artifact root and discovery +- Agent yaml and generated stubs +- Agent loading and execution +- Workflow execution +- Build context and collisions +- Dependencies and protobuf +- Credentials capability +- Policy and provider behavior +- Cleanup boundary + +## Artifact root and discovery + +The build phase of `canyonos deploy` runs from the application root and reads +`.car` below it. That artifact root holds `config/` beside `app/`, the copy of +the application source that becomes `/app` inside every image. Paths in the +config are relative to `app/` and may not escape it. + +| Input | Discovery | +|---|---| +| agent declarations | any `.car/config/*.yaml` with a top-level `agent.name` | +| `.car/config/global_controller.yaml` | default config, overridable with `-c` | +| entrypoint | `entrypoint` on an agent entry, relative to `app/` | +| workflow | `workflow_file` on a `type: workflow` entry, relative to `app/` | +| policy | `policy.yaml` beside the selected config file | +| generated files | `stubs/`, `grpc_stubs/`, `docker_container/`, all under `.car` beside `app/` | + +Because build products sit next to the copy rather than inside it, an +application directory named `stubs/` or `build/` survives into the image. + +The config name, yaml `agent.name`, and entrypoint class name form one binding: + +```text +config entry name == yaml agent.name == entrypoint class name +``` + +A missing match may skip an image while the command continues, so inspect build +output and generated image tags. + +## Agent yaml and generated stubs + +The consumed yaml shape is: + +```yaml +agent: + name: ExampleAgent + functions: + - name: work + arguments: + - name: query + type: str + returns: + type: dict +``` + +Argument annotations are generated from bare names without adding imports. Use +builtins. Generated methods have no defaults, so every declared argument is +required at the stub call site. `returns` does not control runtime conversion; +it documents whether workflow code should parse the returned string. + +A stub has exactly one destination: the agent's own `entrypoint` path. In every +image except that agent's own, the stub is written over the real module there, +so an import of the agent from its source location resolves to the stub and +travels over gRPC. The agent's own image keeps its real module and receives +only its peers' stubs. The validator checks workflow imports against those +entrypoints before the workflow image starts. + +The `entrypoint` path is the whole of it. The declaration file's own basename +names neither the stub nor the agent, and has no runtime meaning beyond being +discovered in `config/`. + +What the stub replaces is one whole module, and everything else in the copy +still runs around it. In a peer image: + +- the entrypoint's package `__init__.py` is real and runs before the stub is + reached, so a re-export from the stubbed module (`from .graph import graph`) + raises ImportError at container startup -- V033; +- the entrypoint's sibling modules are real, so that image installs *their* + dependencies even though its own code never names them -- W006; +- the stub defines the declared class and nothing else: no module-level + constants, no helper functions, no other class the source module exported. + +The third has no static check. Read what the workflow imports out of that +module. + +## Agent loading and execution + +The local controller effectively performs: + +```python +module = load(entrypoint) +agent_class = getattr(module, configured_name) +agent = agent_class() +result = getattr(agent, method_name)(**args) +``` + +Consequences: + +- The class is module-level and named exactly as configured. +- Construction takes no arguments. +- The module name is `VENTIS_AGENT_FILE` with `.py` stripped -- directory + separators and all -- so an entrypoint at `pkg/agent.py` loads as the module + `pkg/agent`, which has no parent package. Relative imports in the entrypoint + raise `attempted relative import with no known parent package`; modules it + imports absolutely are unaffected. V035. +- Declared methods accept yaml argument names as keyword arguments. +- Methods are synchronous; this path does not await a coroutine. +- Dicts and lists are JSON-encoded before entering Redis; other results become + strings. +- A remote Future's `.value()` returns text, not the original Python object. + +Agent import and construction exceptions are caught by the controller. A failed +agent may still advertise healthy because health is written independently of +successful agent loading. After an explicitly approved deployment, inspect +container logs rather than treating health as proof that the entrypoint loaded. + +## Workflow execution + +The workflow file is executed, not imported. Therefore: + +- module-level code runs at container startup; +- `__name__ == "__main__"`; +- `deploy()` blocks in the web server; +- the workflow function runs once per request; +- its function name determines the REST route exposed by the compatibility + runtime. + +The deployment platform additionally expects `/main` with a `{query: string}` +body. This platform constraint is stricter than the underlying transport. + +Each stub method returns a Future immediately. `.value()` blocks. Dispatching +and resolving inside one comprehension serializes work without raising an +error; dispatch all calls first, then resolve them. + +The workflow container also starts runtime controller code and has its own +package resolution. A failure there can differ from failures in agent images. + +## Build context and collisions + +The runtime sweeps `app/` while preserving relative paths, then writes shared +runtime modules, generated stubs, and entrypoints into the image. Later writes +can shadow swept files. + +Avoid modules at the root of the copy named like runtime files, including: + +```text +future.py +ventis_context.py +local_controller.py +local_controller_frontend.py +redis_client.py +grpc_options.py +bedrock.py +deploy.py +session_logging.py +workflow_launcher.py +``` + +Two agents also may not share one entrypoint: each stub is written over its own +entrypoint, so the second lands on the first and every caller reaches whichever +was built last. The validator checks both collisions. + +File sweep and editable-install behavior are runtime capabilities. For nested +imports, follow [preparation.md](preparation.md#import-roots-metadata-and-runtime-assets). + +## Dependencies and protobuf + +Agent and workflow images include a small runtime dependency set. Config +`requirements` adds source-specific distributions. A malformed requirements +value can be normalized away while image generation continues; missing imports +then surface only when the agent loads. + +The build compiles gRPC Python stubs on the host and copies them into images. +The image resolver does not necessarily know the generated-code version. A +source dependency that constrains protobuf below the host generator version can +produce a green image build that dies on: + +```text +import local_controller +``` + +Treat a generated-code/runtime-version mismatch during an explicitly approved +`canyonos deploy` as a CanyonOS Core runtime issue, not a reason to alter source +dependencies silently. + +## Credentials capability + +When `env_file` capability is available, the top-level config path is resolved +against the application root -- the directory the command runs from, not `.car` +-- and passed at container start. A `.env` beside the source stays out of the +artifacts. Hidden env files are not +copied into images. Invalid paths are deploy-preflight errors. + +When the capability is unavailable, declaring `env_file` has no effect. If the +source needs credentials, report the capability blocker rather than hardcoding +or vendoring a secret. + +A source that constructs its client at import time works only when credentials +are already in the container environment. After an explicitly approved deploy, +check loading failures against the same env file configured for deployment. + +For proxy-specific credential separation, read [llm-proxy.md](llm-proxy.md). + +## Policy and provider behavior + +No policy file means unrestricted service access. If a policy exists, it needs a +non-empty rules list. Rules are evaluated by specificity and first match; +services excluded from the selected rule fail after request acceptance. + +Local provider handling is case-sensitive: use lowercase `local`. EC2 behavior +and remote networking are covered in [ec2.md](ec2.md). + +## Cleanup boundary + +`canyonos deploy` follows controller logs after starting the deployment. Ctrl+C +stops that log stream; use `canyonos stop` to request controller teardown when +the user asks to stop the deployment. Hard kills and failures before resource +registration may leave resources behind. + +`canyonos clean` removes generated `stubs/`, `grpc_stubs/`, and +`docker_container/`; it does not remove containers or images. Remove exact +leftovers explicitly and preserve `.car/app`, `.car/config`, and requested +evidence. diff --git a/.claude/skills/porting-to-canyonos/references/source-survey.md b/.claude/skills/porting-to-canyonos/references/source-survey.md new file mode 100644 index 0000000..d63485e --- /dev/null +++ b/.claude/skills/porting-to-canyonos/references/source-survey.md @@ -0,0 +1,93 @@ +# Survey the copied source + +**When:** after `prepare.py`, before writing adapters or configuration. + +**Inspect:** `.car/app`, not a framework-based guess about the original tree. + +**Output:** a short survey record and the smallest useful service map. Do not +start adapter or config work until each section is resolved or marked as a +blocker. + +## 1. Public behavior + +Record: + +- the production entrypoint and callable input/output; +- the documented route, CLI, or launch path that proves this is the entrypoint; +- prompts, tools, schemas, parsing, retries, model clients, and node bodies that + remain source-owned. + +If multiple implementations look plausible, trace imports from the documented +launch path instead of choosing by filename. + +## 2. Control flow and state + +Record: + +- framework-owned graphs, crews, chats, routing, fan-out, commands, and + interrupts; +- independent work that may justify separate resources or replicas; +- injected stores, context, memory, sessions, checkpointers, and callback + managers; +- sync/async boundaries and objects tied to an event loop. + +This evidence feeds the service-boundary decision below and `adapter.md`. + +## 3. Imports and runtime inputs + +Record: + +- the transitive import graph from the selected entrypoint; +- source lockfiles and pinned runtime distributions; +- whether every import resolves with `.car/app` mounted as `/app`; +- model providers, credential variable names, streaming calls, and optional + `llm_proxy` use; +- non-Python files opened at runtime: prompts, framework YAML, PDFs, templates, + schemas, certificates, and corpora. + +Use the import-root and runtime-assets sections of `preparation.md` when imports +do not resolve, packaging metadata matters, or runtime code reads non-Python +files. + +## 4. Capability questions + +Record only capabilities that affect this source, such as editable installation, +full-file sweeping, or environment-file injection. Inspect the installed runtime +or its capability probe; do not run validation merely to reconfirm properties +already guaranteed by `prepare.py`. + +An existing syntax error on the selected import graph is a source defect; +obtain approval before changing even the copied version. The final gap validator +checks authored runtime code and cross-file bindings after the port is complete. +If a required runtime capability is unavailable, report a blocker instead of +assuming support. + +## 5. Choose service boundaries + +**Output:** the smallest useful service map, including which framework edges +cross services and which services require `replicas: 1`. + +Start with one service. Split only when doing so creates independently parallel +work or a genuinely distinct resource or replica profile. + +- Keep a ReAct loop together; each turn needs shared message history. +- Hoist supervisor task lists and `Send`-style fan-out into the workflow. +- Do not create a one-replica service with no distinct resource profile merely + to mirror every source graph node. + +Rewrite framework-owned edges as ordinary Python **only where they cross a +service boundary**. If every graph node stays in one service, preserve +`graph.compile().invoke(...)` and wrap it. Rewriting internal edges restates +working source behavior without creating a deployment benefit. Where an edge +must move into the workflow, import the connected source node functions +unchanged. + +Construct runtime-injected service objects from source configuration. Never +invent models, embedding dimensions, stores, or defaults silently; report any +choice the source does not specify. + +A service object that holds state across requests—for example a vector store, +memory, or checkpointer created in `__init__`—requires `replicas: 1` for +correctness. The controller can route each call to a different replica, and +those replicas cannot see one another's in-memory state. Record this as a +constraint in the configuration review and handoff, not as a sizing preference. diff --git a/.claude/skills/porting-to-canyonos/references/troubleshooting.md b/.claude/skills/porting-to-canyonos/references/troubleshooting.md new file mode 100644 index 0000000..ef2faa4 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/references/troubleshooting.md @@ -0,0 +1,79 @@ +# Diagnose an approved deployment failure + +**When:** an explicitly approved `canyonos deploy` fails during build, startup, +or a request. Do not use this table to extend the pre-deployment porting flow. + +**How:** find the symptom, verify the likely cause, then follow the linked +canonical reference. For runtime mechanisms read +[runtime-contract.md](runtime-contract.md); for proxy or remote-host failures +read [llm-proxy.md](llm-proxy.md) or [ec2.md](ec2.md). + +## Build or deploy stops early + +| Symptom | Likely cause | +|---|---| +| `Application source copy not found` | The command ran somewhere other than the application root, or `.car/app/` was never created | +| `Config file not found: .car/config/...` while `.car` exists | The command ran inside `.car`; it belongs one level up | +| Agent image is missing | Config name matched no declaration in `config/`, entrypoint is absent, or build skipped it; inspect build warnings | +| Wrong declaration is used | Two files in `config/` declare the same `agent.name`; the later filename silently wins | +| Two services produce one image | Config names collide after lowercase normalization | +| `generated grpc_stubs are missing` | Build did not complete on this host, or generated output was cleaned before deploy | +| `int(... NoneType)` during local deploy | Local provider was not written exactly as lowercase `local` | +| Replica conversion `TypeError` | `replicas` is not an integer | +| Policy `AttributeError` | Policy exists but is empty or has null/non-list rules; remove it when unrestricted | +| Port or container name already in use | A previous deployment did not complete cleanup | + +## Container exits or serves nothing + +| Symptom | Likely cause | +|---|---| +| `import local_controller` raises protobuf version error | Host-generated gRPC code is newer than the image's protobuf runtime | +| First request says `No agent loaded` | Entrypoint import, class lookup, or constructor failed and the controller swallowed the exception; inspect container logs | +| Replica is healthy but serves nothing | Health publication does not prove successful agent loading | +| Missing credentials while loading | Env injection is unavailable/misconfigured or the source reads another variable | +| Source module is missing | `.car/app` is not rooted at the source's import root, so the original import does not resolve from `/app`; read [preparation.md](preparation.md#import-roots-metadata-and-runtime-assets) | +| Third-party module is missing | Distribution is absent from source metadata and config requirements | +| Stub import raises `NameError` | yaml argument type is not a bare builtin | +| An agent runs in the workflow process instead of a container | The workflow reached the class by a path other than the agent's `entrypoint`, so it got the real module rather than the stub | +| Calls to one agent reach another | Two config entries share an `entrypoint`, so one stub was written over the other | +| Runtime-named module disappears | Shared runtime copy overwrote a module at the root of the source copy with the same name | +| An application file is missing from the image | Only `.py` files are swept out of the copy unless the `sweeps_all_files` capability is available | +| Peer container raises `ImportError` for a name in another agent's module | That agent's package `__init__.py` re-exports from its entrypoint, which is a stub in this image; V033 | +| `attempted relative import with no known parent package` | The entrypoint's own imports are relative, and it is loaded by path with no parent package; V035 | +| `ModuleNotFoundError` for a distribution this image's own code never imports | The entrypoint's package `__init__` or a sibling imports it; add it to this entry's `requirements:` | +| `SyntaxError` on the workflow's agent import | An `entrypoint` path segment is not a Python identifier; V034 | +| A model call fires at container start, before any request | Module-level code in the entrypoint performs a real run | +| `ModuleNotFoundError` for a submodule that used to exist | An unpinned requirement resolved to a newer major; pin it to what the source resolved | +| `unexpected keyword argument` inside an SDK call | The pinned distribution is newer than the source; pin contemporaneous with the source's commit date | + +## Request is accepted, then fails + +| Symptom | Likely cause | +|---|---| +| Unexpected keyword argument | yaml argument name differs from adapter parameter name | +| Required argument missing | yaml does not declare a required adapter parameter, or workflow platform sent only `query` | +| Unauthorized service | The first matching policy rule excludes that service | +| `.value()` returns dict-like text | Expected; remote values are strings, so use `json.loads` | +| Object is not JSON serializable | Adapter returned framework objects without its JSON-safe serializer | +| Redis contains a coroutine repr | Adapter method is async and the controller did not await it | +| Fan-out is no faster | Dispatch and `.value()` were fused, serializing the calls | +| Debug block runs at startup | Workflow is executed with `__name__ == "__main__"` | +| `Lock is bound to a different event loop` on the second request | `asyncio.run` per call, while the instance holds loop-bound state; use one persistent background loop | +| Model call reaches the real provider with the proxy configured | The SDK reads a base-URL variable the env file does not set; read [llm-proxy.md](llm-proxy.md) | + +## Deployment platform endpoint + +| Symptom | Likely cause | +|---|---| +| 404 while workflow container is healthy | Workflow function is not named `main` | +| 400 before host receives request | Body is not the platform's `{query: string}` shape | +| Extra workflow argument is missing | Platform sends only `query`; extra parameters need defaults | + +## Cleanup + +| Symptom | Likely cause | +|---|---| +| Ctrl+C leaves the deployment running | `canyonos deploy` follows logs; Ctrl+C stops monitoring, not the deployment. Ask before running `canyonos stop` | +| `canyonos clean` succeeds but containers remain | The command removes generated directories only | +| `canyonos clean` succeeds but images remain | Image deletion is separate and requires exact tags | +| Next deployment collides with old resources | The deployment was killed or crashed before controller cleanup | diff --git a/.claude/skills/porting-to-canyonos/references/validation-and-deploy.md b/.claude/skills/porting-to-canyonos/references/validation-and-deploy.md new file mode 100644 index 0000000..9f5f515 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/references/validation-and-deploy.md @@ -0,0 +1,70 @@ +# Validate, hand off, and optionally deploy + +**When:** after runtime code and configuration are complete. + +**Output:** a clean validation report and a stopped porting workflow. Deployment +is a separate action that requires explicit approval. + +## Validation scope + +Validate only contracts that authored port files can violate and CanyonOS does +not fail closed on—for example declaration/adapter bindings, generated-stub +imports, workflow call shape, per-image dependency coverage, and +capability-dependent behavior. + +Do not add duplicate checks for postconditions strongly guaranteed by code: + +- a successful `prepare.py` run already creates the artifact directories, + rejects source symlinks, applies exclusions, and installs the copy atomically; +- `canyonos config` owns the structure of configuration it generates; +- deploy preflight owns checks that already fail before resources are changed. + +Treat required inputs such as a readable manifest and `.car/app` as validator +preconditions, not independent port rules. If a guarantee changes in CanyonOS, +change the owning code or capability probe rather than maintaining a parallel +rule in prose and validation. + +## Run the gap validator + +From the application root, run: + +```bash +python3 /validate.py .car +``` + +Fix every `ERROR` and rerun until the command exits 0. Do not hide warnings or +capability limitations: list each in the handoff and state whether it blocks +this source. Confirm with `git status` that no developer-owned file outside +`.car` changed. + +Report: + +- that the `.car` port validated; +- files created; +- validator warnings; +- unresolved runtime blockers; +- intentionally omitted unreachable dependencies or source surfaces. + +Then stop and ask exactly one direct approval question: + +> Validation passed. Run `canyonos deploy` now? This will build images and start +> the deployment. + +Do not treat silence, an unattended run, or the original request to “port” as +approval. + +## Deploy only after approval + +If the user explicitly approves, run from the application root: + +```bash +canyonos deploy +``` + +Do not run a standalone build first: `canyonos deploy` performs both build and +deployment. Do not add probing, deployment debugging, or cleanup to the porting +flow. + +If an approved deploy fails during build, startup, or a request, read +`troubleshooting.md`. Read `runtime-contract.md` when a validator finding or +runtime mechanism needs explanation. diff --git a/.claude/skills/porting-to-canyonos/validate.py b/.claude/skills/porting-to-canyonos/validate.py new file mode 100755 index 0000000..91631f7 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validate.py @@ -0,0 +1,313 @@ +#!/usr/bin/env python3 +"""Preflight a CanyonOS port before an approved deployment. + +This checks the public `.car` artifact contract first, then parses Python without +importing it to catch failures that would otherwise stay hidden until a +container loads an agent, starts a workflow, or serves its first request. It +fails closed when the required inputs cannot be checked. A replica is not +evidence: the controller writes `healthy` to Redis before `_load_agent` runs. + + python3 validate.py [artifact_root] [-c config/global_controller.yaml] + [--json] [--strict] + +`artifact_root` is the `.car` directory: `config/` beside `app/`, the copy of +the application source that becomes /app inside every container. + +Exit 1 if any ERROR was reported, 0 otherwise. --strict also fails on warnings. + +Runtime capabilities vary across CanyonOS Core installations. This script probes +the importable `ventis` package directly. A capability-gated check reports +UNAVAILABLE when its behavior cannot be proven. +""" + +import argparse +import json +import os +import sys + +SKILL_DIR = os.path.dirname(os.path.abspath(__file__)) +if SKILL_DIR not in sys.path: + sys.path.insert(0, SKILL_DIR) + +from validation.adapter import check_adapter +from validation.core import ERROR, INFO, WARN, Report, load_yaml +from validation.dependencies import ( + check_requirements_coverage, + check_secrets, +) +from validation.entrypoint import ( + check_entrypoint_module, + check_flat_collisions, +) +from validation.manifest import ( + check_declaration_bindings, + check_manifest_structure, + check_policy, + discover_agent_declarations, +) +from validation.packaging import check_env_file, check_import_root +from validation.python_source import module_path +from validation.runtime import ( + BASE_AGENT_REQUIREMENTS, + BASE_WORKFLOW_REQUIREMENTS, + CAPABILITY_SOURCE, + probe_capabilities, +) +from validation.workflow import check_workflow + +DEFAULT_CONFIG_PATH = "config/global_controller.yaml" +# ventis/cli.py SOURCE_DIR_NAME -- the duplicated application source. +SOURCE_DIR_NAME = "app" + + +# Path existence and readability are deploy-preflight checks. Do not +# duplicate them here. + + +# ------------------------------------------------------------------ # +# Driver # +# ------------------------------------------------------------------ # + + +def validate(artifact_dir, config_path, capabilities): + """Check the public artifact contract and deeper runtime failure modes.""" + report = Report(artifact_dir, capabilities) + + config, error = load_yaml(config_path) + if error is not None or not isinstance(config, dict): + report.error( + "V001", + config_path, + 0, + f"the global manifest cannot be read: {error or 'expected a YAML mapping'}", + "Validation finishes before an approved `canyonos deploy`, so an " + "unreadable manifest cannot be deferred to deploy.", + ) + return report + + source_dir = os.path.join(artifact_dir, SOURCE_DIR_NAME) + if not os.path.isdir(source_dir): + report.error( + "V032", + artifact_dir, + 0, + f"no `{SOURCE_DIR_NAME}/` beside `config/`", + "The artifact root holds the application source it deploys: " + f"`{SOURCE_DIR_NAME}/` is the copy that becomes /app, and every " + "entrypoint is relative to it. Without it the port has nothing to " + "build and nothing to keep it decoupled from the developer's tree.", + ) + return report + + # `prepare.py` already rejects symlinks and creates the artifact layout + # atomically. Do not duplicate postcondition checks that the preparation + # code strongly guarantees; validation focuses on authored cross-file and + # runtime contracts. + + config_dir = os.path.dirname(config_path) + entries = check_manifest_structure(report, config, config_path, source_dir) + if entries is None: + return report + + agents_by_name = discover_agent_declarations(report, config_dir, config_path) + check_declaration_bindings(report, entries, agents_by_name, config_path) + check_policy(report, config_dir) + + entrypoints = [] + for entry in entries: + if not isinstance(entry, dict) or entry.get("type", "agent") == "workflow": + continue + name = entry.get("name") + entrypoint = entry.get("entrypoint") + if isinstance(entrypoint, str): + entrypoints.append((name, entrypoint)) + if name in agents_by_name: + yaml_path, agent_block = agents_by_name[name] + check_adapter(report, yaml_path, agent_block, entry, source_dir) + entrypoint_path = os.path.join(source_dir, entrypoint or "") + if isinstance(entrypoint, str) and os.path.isfile(entrypoint_path): + check_entrypoint_module(report, source_dir, name, entrypoint) + check_requirements_coverage( + report, + source_dir, + entry, + entrypoint_path, + config_path, + BASE_AGENT_REQUIREMENTS, + ) + + # Where each agent's stub is written, and therefore the only import that + # reaches it over gRPC. + stub_modules = { + name: module_path(entrypoint) + for name, entrypoint in entrypoints + if name in agents_by_name + } + stubbed_entrypoint_paths = [ + os.path.join(source_dir, entrypoint) + for name, entrypoint in entrypoints + if name in agents_by_name + ] + + for entry in entries: + if not isinstance(entry, dict) or entry.get("type", "agent") != "workflow": + continue + workflow_file = entry.get("workflow_file") + if not isinstance(workflow_file, str): + continue + workflow_path = os.path.join(source_dir, workflow_file) + if os.path.isfile(workflow_path): + check_workflow(report, workflow_path, stub_modules) + # The workflow image installs its own list. A module it imports for + # a helper drags that module's dependencies in even though the + # workflow makes no model call of its own. + check_requirements_coverage( + report, + source_dir, + entry, + workflow_path, + config_path, + BASE_WORKFLOW_REQUIREMENTS, + shadowed_paths=stubbed_entrypoint_paths, + ) + + # These survive a green build and otherwise surface only in a container or + # on its first request. + check_flat_collisions(report, source_dir, entrypoints) + check_env_file(report, config, config_path, artifact_dir) + + entrypoint_paths = [ + os.path.join(source_dir, e) + for _, e in entrypoints + if os.path.isfile(os.path.join(source_dir, e)) + ] + check_import_root(report, source_dir, entrypoint_paths) + + port_paths = list(entrypoint_paths) + for entry in entries: + if isinstance(entry, dict) and isinstance(entry.get("workflow_file"), str): + candidate = os.path.join(source_dir, entry["workflow_file"]) + if os.path.isfile(candidate): + port_paths.append(candidate) + + # Secret detection remains because a green image build would permanently + # bake the credential into every image. + check_secrets(report, port_paths) + return report + + +# ------------------------------------------------------------------ # +# Output # +# ------------------------------------------------------------------ # + +LEVEL_ORDER = {ERROR: 0, WARN: 1, INFO: 2} + + +def _wrap(text, width, indent): + words = text.split() + lines = [] + current = "" + for word in words: + candidate = f"{current} {word}".strip() + if len(candidate) + len(indent) > width and current: + lines.append(indent + current) + current = word + else: + current = candidate + if current: + lines.append(indent + current) + return lines + + +def print_report(report, artifact_root): + caps = report.capabilities + if not caps.get("ventis"): + print("ventis is not importable here -- capability-gated rules are") + print("reported UNAVAILABLE rather than checked.\n") + else: + print("CanyonOS Core capabilities detected:") + for key, source in CAPABILITY_SOURCE.items(): + mark = "yes" if caps.get(key) else "no " + print(f" {mark} {key:<22} {source}") + print() + + findings = sorted( + report.findings, + key=lambda f: (LEVEL_ORDER[f["level"]], f["check"], f["path"], f["line"]), + ) + for finding in findings: + where = finding["path"] + if where and finding["line"]: + where = f"{where}:{finding['line']}" + header = f"{finding['check']} {finding['level']:<5}" + print(f"{header} {where}" if where else header) + for line in _wrap(finding["summary"], 78, " "): + print(line) + if finding["mechanism"]: + for line in _wrap(finding["mechanism"], 78, " "): + print(line) + print() + + errors, warnings = report.counts() + if not findings: + print(f"{artifact_root}: clean.") + return + print(f"{errors} error(s), {warnings} warning(s).") + + +def main(argv=None): + parser = argparse.ArgumentParser( + description="Check authored CanyonOS port contracts not guaranteed by tooling." + ) + parser.add_argument( + "artifact_root", + nargs="?", + default=".", + help="the .car directory holding config/ and app/ (default: the cwd)", + ) + parser.add_argument( + "-c", + "--config", + default=DEFAULT_CONFIG_PATH, + help=f"config path relative to artifact_root (default: {DEFAULT_CONFIG_PATH})", + ) + parser.add_argument("--json", action="store_true", help="emit findings as JSON") + parser.add_argument( + "--strict", action="store_true", help="fail on warnings as well as errors" + ) + args = parser.parse_args(argv) + + artifact_root = os.path.abspath(args.artifact_root) + config_path = ( + args.config + if os.path.isabs(args.config) + else os.path.join(artifact_root, args.config) + ) + + capabilities = probe_capabilities() + report = validate(artifact_root, config_path, capabilities) + errors, warnings = report.counts() + + if args.json: + print( + json.dumps( + { + "artifact_root": artifact_root, + "capabilities": capabilities, + "errors": errors, + "warnings": warnings, + "findings": report.findings, + }, + indent=2, + ) + ) + else: + print_report(report, report.rel(artifact_root) or artifact_root) + + if errors or (args.strict and warnings): + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/.claude/skills/porting-to-canyonos/validation/__init__.py b/.claude/skills/porting-to-canyonos/validation/__init__.py new file mode 100644 index 0000000..aa7a1a3 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/__init__.py @@ -0,0 +1 @@ +"""Composable validation checks for the CanyonOS porting skill.""" diff --git a/.claude/skills/porting-to-canyonos/validation/adapter.py b/.claude/skills/porting-to-canyonos/validation/adapter.py new file mode 100644 index 0000000..445ff10 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/adapter.py @@ -0,0 +1,175 @@ +"""V006-V010 -- adapter faults the controller swallows inside _load_agent.""" + +import ast +import builtins +import os + +from validation.core import line_of +from validation.python_source import ( + class_methods, + find_class, + parameter_names, + parse_python, + required_parameters, +) + +BUILTIN_TYPE_NAMES = frozenset( + name + for name in dir(builtins) + if isinstance(getattr(builtins, name), type) and not name[0].isupper() +) + + +def check_adapter(report, agent_yaml_path, agent_block, entry, project_dir): + """V006 V007 V008 V009 V010.""" + name = agent_block["name"] + functions = agent_block.get("functions") or [] + check_argument_types(report, agent_yaml_path, functions) + + entrypoint = entry.get("entrypoint") + if not entrypoint: + return + entrypoint_path = os.path.join(project_dir, entrypoint) + if not os.path.isfile(entrypoint_path): + return + + tree, error = parse_python(entrypoint_path) + if tree is None: + report.error( + "V006", + entrypoint_path, + 0, + f"the entrypoint does not parse: {error}", + "_load_agent exec_module's it and swallows the exception; the first " + "request answers 'No agent loaded'.", + ) + return + + class_node = find_class(tree, name) + if class_node is None: + classes = [n.name for n in tree.body if isinstance(n, ast.ClassDef)] + found = ", ".join(classes) if classes else "no classes at all" + report.error( + "V006", + entrypoint_path, + 1, + f"no class named `{name}` at module level (found: {found})", + "_load_agent does getattr(module, VENTIS_AGENT_NAME) and swallows " + "the AttributeError. The class name must equal agent.name exactly.", + ) + return + + methods = class_methods(class_node) + check_constructor(report, entrypoint_path, name, methods) + + for func in functions: + if not isinstance(func, dict) or not isinstance(func.get("name"), str): + continue + check_method(report, entrypoint_path, agent_yaml_path, name, func, methods) + + +def check_argument_types(report, agent_yaml_path, functions): + """V010 -- the type string is pasted into an ast.Name, never checked.""" + for func in functions: + if not isinstance(func, dict): + continue + for arg in func.get("arguments") or []: + if not isinstance(arg, dict) or "type" not in arg: + continue + declared = arg.get("type") + if not isinstance(declared, str): + continue # stub generation reports malformed type values + if declared in BUILTIN_TYPE_NAMES: + continue + report.error( + "V010", + agent_yaml_path, + line_of(arg, "type"), + f"`type: {declared}` is not a builtin", + "stub_generator pastes it verbatim into the generated " + "annotation, and the stub module imports only Future and " + "inspect. Anything else raises NameError when the stub is " + "imported -- after a green build. Use str int float bool dict " + "list.", + ) + + +def check_constructor(report, entrypoint_path, name, methods): + """V007 -- _load_agent calls agent_class() with no arguments.""" + init = methods.get("__init__") + if init is None: + return + required = required_parameters(init) + if required: + report.error( + "V007", + entrypoint_path, + init.lineno, + f"`{name}.__init__` requires {', '.join(required)}", + "_load_agent calls agent_class() with no arguments; the TypeError is " + "swallowed and the first request answers 'No agent loaded'. Read " + "configuration from the environment inside __init__ instead.", + ) + + +def check_method(report, entrypoint_path, agent_yaml_path, class_name, func, methods): + """V008 V009.""" + func_name = func["name"] + method = methods.get(func_name) + if method is None: + report.error( + "V008", + entrypoint_path, + 0, + f"`{class_name}` has no method `{func_name}`", + "The yaml declares it, so callers get a stub for it; the controller " + f"then answers \"Agent {class_name} has no method '{func_name}'\".", + ) + return + + # V009 -- nothing on the execution path awaits. + if isinstance(method, ast.AsyncFunctionDef): + report.error( + "V009", + entrypoint_path, + method.lineno, + f"`{class_name}.{func_name}` is `async def`", + "The executor calls method(**args) with no await, so Redis receives " + "''. Keep the signature synchronous and call " + "asyncio.run(...) inside the body.", + ) + + # V008 -- the controller calls method(**args) with the yaml's names. + declared = [ + arg["name"] + for arg in func.get("arguments") or [] + if isinstance(arg, dict) and isinstance(arg.get("name"), str) + ] + actual = parameter_names(method) + required = required_parameters(method) + + missing = [arg for arg in declared if arg not in actual] + if missing: + report.error( + "V008", + entrypoint_path, + method.lineno, + f"`{class_name}.{func_name}` has no parameter " + f"{', '.join(repr(m) for m in missing)}, but the yaml declares it", + "LocalController does method(**args) with the yaml's argument names. " + "A mismatch is TypeError: unexpected keyword argument, at request " + f"time. See {os.path.basename(agent_yaml_path)}.", + ) + + unfilled = [arg for arg in required if arg not in declared] + if unfilled: + report.error( + "V008", + entrypoint_path, + method.lineno, + f"`{class_name}.{func_name}` requires {', '.join(unfilled)}, which " + "the yaml does not declare", + "Only declared arguments are ever sent, and the generated stub gives " + "none of them a default. Declare them in the yaml or default them " + "in the signature.", + ) diff --git a/.claude/skills/porting-to-canyonos/validation/core.py b/.claude/skills/porting-to-canyonos/validation/core.py new file mode 100644 index 0000000..490b949 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/core.py @@ -0,0 +1,98 @@ +"""Shared result and YAML primitives for validation checks.""" + +import os +from typing import ClassVar + +try: + import yaml +except ImportError: # pragma: no cover - pyyaml is a CanyonOS dependency + raise RuntimeError("validate.py needs pyyaml: pip install pyyaml") from None + + +ERROR = "ERROR" +WARN = "WARN" +INFO = "INFO" + + +class LineDict(dict): + """A mapping that remembers where it and each of its keys were written.""" + + line = 0 + key_lines: ClassVar[dict] = {} + + +class LineLoader(yaml.SafeLoader): + pass + + +def _construct_mapping(loader, node): + data = LineDict() + yield data + data.update(loader.construct_mapping(node, deep=False)) + data.line = node.start_mark.line + 1 + data.key_lines = { + key.value: key.start_mark.line + 1 + for key, _ in node.value + if isinstance(key, yaml.ScalarNode) + } + + +LineLoader.add_constructor( + yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, _construct_mapping +) + + +def line_of(mapping, key=None): + if not isinstance(mapping, LineDict): + return 0 + if key is not None: + return mapping.key_lines.get(key, mapping.line) + return mapping.line + + +def load_yaml(path): + """Parse path and return ``(data, error)`` without raising.""" + try: + with open(path, "r", encoding="utf-8") as handle: + return yaml.load(handle, Loader=LineLoader), None + except Exception as exc: # noqa: BLE001 - every parse failure is a finding + return None, str(exc) + + +class Report: + def __init__(self, project_dir, capabilities): + self.project_dir = project_dir + self.capabilities = capabilities + self.findings = [] + + def add(self, check, level, path, line, summary, mechanism): + self.findings.append( + { + "check": check, + "level": level, + "path": self.rel(path) if path else "", + "line": line or 0, + "summary": summary, + "mechanism": mechanism, + } + ) + + def error(self, check, path, line, summary, mechanism): + self.add(check, ERROR, path, line, summary, mechanism) + + def warn(self, check, path, line, summary, mechanism): + self.add(check, WARN, path, line, summary, mechanism) + + def unavailable(self, check, summary): + self.add(check, INFO, "", 0, summary, "") + + def rel(self, path): + try: + return os.path.relpath(path, self.project_dir) + except ValueError: + return path + + def counts(self): + errors = sum(1 for finding in self.findings if finding["level"] == ERROR) + warnings = sum(1 for finding in self.findings if finding["level"] == WARN) + return errors, warnings diff --git a/.claude/skills/porting-to-canyonos/validation/dependencies.py b/.claude/skills/porting-to-canyonos/validation/dependencies.py new file mode 100644 index 0000000..4060319 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/dependencies.py @@ -0,0 +1,207 @@ +"""W003, W006 -- credentials and imports a successful build does not reject.""" + +import ast +import os +import re + +from validation.python_source import ( + parse_python, + reachable_imports, + resolves_flat, + resolves_nested, +) +from validation.runtime import ( + IMPORT_TO_DISTRIBUTION, + NAMESPACE_DISTRIBUTIONS, + RUNTIME_FLAT_NAMES, + STDLIB_MODULE_NAMES, +) + +SECRET_PATTERNS = [ + (re.compile(r"sk-[A-Za-z0-9_-]{20,}"), "an OpenAI-style secret key"), + (re.compile(r"AKIA[0-9A-Z]{16}"), "an AWS access key id"), + (re.compile(r"gh[pousr]_[A-Za-z0-9]{20,}"), "a GitHub token"), + (re.compile(r"AIza[0-9A-Za-z_-]{30,}"), "a Google API key"), +] + + +SECRET_NAME = re.compile(r"(API_KEY|SECRET|TOKEN|PASSWORD|CREDENTIAL)", re.IGNORECASE) + + +def check_secrets(report, port_paths): + """W003 -- env_file is the way in; nothing else is.""" + for path in port_paths: + try: + with open(path, "r", encoding="utf-8") as handle: + lines = handle.read().splitlines() + except OSError: + continue + for number, line in enumerate(lines, start=1): + for pattern, description in SECRET_PATTERNS: + if pattern.search(line): + report.error( + "W003", + path, + number, + f"this line looks like {description}", + "Never put a secret in the source tree or the build " + "context. The build sweeps the project into every " + "image.", + ) + break + + tree, _ = parse_python(path) + if tree is None: + continue + for node in ast.walk(tree): + if not isinstance(node, ast.Assign): + continue + if not ( + isinstance(node.value, ast.Constant) + and isinstance(node.value.value, str) + and node.value.value.strip() + ): + continue + for target in node.targets: + if isinstance(target, ast.Name) and SECRET_NAME.search(target.id): + report.warn( + "W003", + path, + node.lineno, + f"`{target.id}` is assigned a literal string", + "Read it from the environment instead; the build sweeps " + "this file into every image.", + ) + + +def _pyproject_dependencies(project_dir): + """What `-e .` installs alongside `requirements:`, or None if unreadable. + + None and the empty set mean different things here: empty means the project + declares no dependencies, None means we could not find out -- a setup.py, or + a tomllib this interpreter does not have. The caller must not treat the + second as the first, or it warns about imports the install would satisfy. + """ + path = os.path.join(project_dir, "pyproject.toml") + if not os.path.isfile(path): + return None + try: + import tomllib + except ImportError: # < 3.11 + return None + try: + with open(path, "rb") as handle: + data = tomllib.load(handle) + except Exception: # noqa: BLE001 - malformed metadata is uv's error to give + return None + deps = (data.get("project") or {}).get("dependencies") + if not isinstance(deps, list): + return None + return {_normalize_distribution(d) for d in deps if isinstance(d, str)} + + +def check_requirements_coverage( + report, + project_dir, + entry, + root_path, + config_path, + base_requirements, + shadowed_paths=(), +): + """W006 -- an import the container cannot satisfy. + + Walks the whole import graph the image executes from `root_path`, not just + that one file: a distribution reached through a local module or a package + __init__ is exactly as missing, and exactly as invisible until the container + starts. + """ + declared = { + _normalize_distribution(item) + for item in (entry.get("requirements") or []) + if isinstance(item, str) + } + + # Where the editable install exists, `-e .` resolves the project's own + # [project.dependencies] in the same pass as `requirements:`. Warning about + # those is a false positive, and a false warning about a dependency is worse + # than none: it teaches the reader to dismiss this check. + editable = report.capabilities.get("editable_install") + metadata = any( + os.path.isfile(os.path.join(project_dir, name)) + for name in ("pyproject.toml", "setup.py", "setup.cfg") + ) + unreadable_metadata = False + if editable and metadata: + project_deps = _pyproject_dependencies(project_dir) + if project_deps is None: + unreadable_metadata = True + else: + declared |= project_deps + base = {_normalize_distribution(item) for item in base_requirements} + satisfied = base | declared + + external = reachable_imports(project_dir, root_path, shadowed_paths) + for dotted, (where, lineno) in sorted(external.items()): + name = dotted.split(".")[0] + if name in STDLIB_MODULE_NAMES or name == "ventis": + continue + # Provided by the image itself: the shared runtime is copied flat over + # the swept tree. A stub is not listed here -- it replaces a module the + # source copy already carries, so the tree checks below cover it. + if f"{name}.py" in RUNTIME_FLAT_NAMES: + continue + if resolves_flat(project_dir, name) or resolves_nested(project_dir, name): + continue + prefix = NAMESPACE_DISTRIBUTIONS.get(name) + if prefix and any( + item == prefix or item.startswith(prefix + "-") for item in declared + ): + continue + if _candidate_distributions(name) & satisfied: + continue + if unreadable_metadata: + mechanism = ( + "The container installs the base list, `requirements:`, and -- " + "since this project declares packaging metadata -- whatever " + "`-e .` resolves from it. That metadata could not be read here, " + f"so if it already requires `{name}` this line is noise; " + "otherwise it is a ModuleNotFoundError inside _load_agent and " + "'No agent loaded' on the first request." + ) + else: + mechanism = ( + "The container installs the base list plus `requirements:` and " + "nothing else, so this is a ModuleNotFoundError inside " + "_load_agent and 'No agent loaded' on the first request. If the " + f"distribution is named something other than `{name}`, declare " + f"that name in {report.rel(config_path)}." + ) + if os.path.realpath(where) != os.path.realpath(root_path): + mechanism += ( + f" This image never names `{name}` in {report.rel(root_path)}; " + f"it runs {report.rel(where)} on the way there, and that module " + "needs it." + ) + report.error( + "W006", + where, + lineno, + f"`import {dotted}` is in neither the runtime's base list nor " + f"{entry.get('name') or 'this entry'}'s `requirements:`", + mechanism, + ) + + +def _candidate_distributions(name): + """Every distribution name that would satisfy `import `.""" + return { + _normalize_distribution(item) + for item in IMPORT_TO_DISTRIBUTION.get(name, (name,)) + } + + +def _normalize_distribution(name): + return re.split(r"[<>=!\[;\s]", name.strip().lower(), maxsplit=1)[0].replace( + "_", "-" + ) diff --git a/.claude/skills/porting-to-canyonos/validation/entrypoint.py b/.claude/skills/porting-to-canyonos/validation/entrypoint.py new file mode 100644 index 0000000..902c3f8 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/entrypoint.py @@ -0,0 +1,141 @@ +"""V019, V020, V033-V035 -- traps set by which module the entrypoint names.""" + +import ast +import os + +from validation.python_source import module_path, parse_python +from validation.runtime import RUNTIME_FLAT_NAMES + + +def check_flat_collisions(report, source_dir, entrypoints): + """V019 V020 -- later copies land on earlier ones at the context root.""" + for entry in sorted(os.listdir(source_dir)): + path = os.path.join(source_dir, entry) + if not os.path.isfile(path) or not entry.endswith(".py"): + continue + + # V019 -- the shared runtime is copied flat, after the project sweep. + if entry in RUNTIME_FLAT_NAMES: + report.error( + "V019", + path, + 1, + f"a project module named `{entry}` sits at the root of the source copy", + "The shared CanyonOS Core runtime is copied flat into the image after " + "the project sweep, so this file is overwritten by CanyonOS Core's own " + f"{entry}. Rename it or move it into a package directory.", + ) + + # V020 -- one module cannot be the entrypoint of two agents. + owners = {} + for name, entrypoint in entrypoints: + owners.setdefault(entrypoint, []).append(name) + for entrypoint, names in sorted(owners.items()): + if len(names) < 2: + continue + report.error( + "V020", + os.path.join(source_dir, entrypoint), + 1, + f"{' and '.join(sorted(names))} both declare `{entrypoint}` as their " + "entrypoint", + "Each agent's stub is written over its own entrypoint, so the two " + "land on one path and the last one built wins. Every caller then " + "reaches whichever agent that was. Give each agent its own module.", + ) + + +def check_entrypoint_module(report, source_dir, name, entrypoint): + """V033 V034 V035. + + Two runtime facts collide here. The build writes this agent's stub over + `entrypoint` in every image except this agent's own, and the controller + loads the real file by path rather than by import. Each breaks a module + layout that is correct everywhere else in Python. + """ + path = os.path.join(source_dir, entrypoint) + if not os.path.isfile(path): + return + + segments = os.path.splitext(entrypoint)[0].replace("\\", "/").split("/") + invalid = [part for part in segments if not part.isidentifier()] + if invalid: + report.error( + "V034", + path, + 0, + f"`{invalid[0]}` in the entrypoint path is not a Python identifier", + "The controller loads the entrypoint by file path, so this file runs " + "-- but the workflow has to import the class from " + f"`{module_path(entrypoint)}` (V023), and that is a SyntaxError, not " + "an ImportError. Rename the file inside the copy, or point " + "`entrypoint` at a normally-named sibling that loads this file by " + "path and re-exposes the class.", + ) + + tree, _ = parse_python(path) + if tree is not None: + for node in ast.walk(tree): + if isinstance(node, ast.ImportFrom) and node.level: + spelling = "." * node.level + (node.module or "") + report.error( + "V035", + path, + node.lineno, + f"the entrypoint's own `from {spelling} import ...` is relative", + "_load_agent loads this file with spec_from_file_location(" + "VENTIS_AGENT_FILE.replace('.py', ''), path). That name keeps " + "the entrypoint's directory separator, so it has no parent " + "package and __package__ is empty: every relative import in " + "this file raises 'attempted relative import with no known " + "parent package' at agent load, behind 'No agent loaded'. " + "Make this file's own top-level imports absolute; modules it " + "imports may keep theirs.", + ) + break + + directory = os.path.dirname(entrypoint) + if not directory: + return + init_path = os.path.join(source_dir, directory, "__init__.py") + if not os.path.isfile(init_path): + return + module = os.path.splitext(os.path.basename(entrypoint))[0] + package = directory.replace("\\", "/").replace("/", ".") + init_tree, _ = parse_python(init_path) + if init_tree is None: + return + for node in ast.walk(init_tree): + if not isinstance(node, ast.ImportFrom): + continue + target = node.module or "" + hit = ( + target == module + if node.level + else target + in ( + module, + f"{package}.{module}", + ) + ) + if not hit and node.level and not node.module: + hit = any(alias.name == module for alias in node.names) + if not hit: + continue + report.error( + "V033", + init_path, + node.lineno, + f"`{package}/__init__.py` re-exports from `{module}`, the entrypoint " + f"for {name}", + "Python runs a package's __init__.py before any of its submodules, " + "and in every image except this agent's own the module at the " + "entrypoint is the generated stub, which defines the agent class and " + f"nothing else. Any peer image that imports anything from `{package}` " + "-- the workflow importing the agent class included -- re-runs this " + "re-export against the stub and dies at container startup with " + f"ImportError. Point `entrypoint` at a module `{package}/__init__.py` " + "does not re-export from; add one that imports the real module if " + "every existing module is re-exported.", + ) + break diff --git a/.claude/skills/porting-to-canyonos/validation/manifest.py b/.claude/skills/porting-to-canyonos/validation/manifest.py new file mode 100644 index 0000000..dab75a2 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/manifest.py @@ -0,0 +1,276 @@ +"""Fail-closed checks for the public CanyonOS artifact contract.""" + +import glob +import os + +from validation.core import line_of, load_yaml + + +def _safe_relative_python_path(value): + if not isinstance(value, str) or not value.strip(): + return False + normalized = value.replace("\\", "/") + return ( + not normalized.startswith("/") + and ".." not in normalized.split("/") + and normalized.endswith(".py") + ) + + +def check_manifest_structure(report, config, config_path, source_dir): + """Return entries only when deeper checks can traverse them safely.""" + entries = config.get("agents") + if not isinstance(entries, list): + report.error( + "V001", + config_path, + line_of(config, "agents"), + "`agents:` must be a list", + "The CanyonOS manifest cannot be traversed or built without an agents list.", + ) + return None + + valid = True + agent_count = sum( + 1 + for entry in entries + if isinstance(entry, dict) and entry.get("type", "agent") == "agent" + ) + workflow_count = sum( + 1 + for entry in entries + if isinstance(entry, dict) and entry.get("type", "agent") == "workflow" + ) + if agent_count < 1: + report.error( + "V002", + config_path, + line_of(config, "agents"), + "the manifest must contain at least one agent service", + "A CanyonOS port needs a callable service behind its workflow.", + ) + valid = False + if workflow_count != 1: + report.error( + "V002", + config_path, + line_of(config, "agents"), + f"the manifest must contain exactly one workflow service; found {workflow_count}", + "The deployment exposes one `/main` workflow and builds one workflow image.", + ) + valid = False + for index, entry in enumerate(entries): + if not isinstance(entry, dict): + report.error( + "V002", + config_path, + 0, + f"agents[{index}] must be a mapping", + "CanyonOS reads each agents item as a service declaration.", + ) + valid = False + continue + + name = entry.get("name") + if not isinstance(name, str) or not name.strip(): + report.error( + "V002", + config_path, + line_of(entry, "name"), + f"agents[{index}] has no non-empty string `name`", + "Names bind manifest entries, declarations, generated stubs, and images.", + ) + valid = False + else: + earlier = [ + item.get("name") + for item in entries[:index] + if isinstance(item, dict) and isinstance(item.get("name"), str) + ] + collision = next( + (other for other in earlier if other.lower() == name.lower()), None + ) + if collision is not None: + report.error( + "V002", + config_path, + line_of(entry, "name"), + f"`{name}` collides with `{collision}` after lowercase normalization", + "CanyonOS uses lowercase image and target names, so one " + "service overwrites the other.", + ) + valid = False + + service_type = entry.get("type", "agent") + if service_type not in ("agent", "workflow"): + report.error( + "V002", + config_path, + line_of(entry, "type"), + f"`type: {service_type}` is neither `agent` nor `workflow`", + "Only those two service shapes have a CanyonOS build contract.", + ) + valid = False + + provider = entry.get("provider", "local") + if provider not in ("local", "EC2"): + report.error( + "V002", + config_path, + line_of(entry, "provider"), + f"unsupported provider spelling `{provider}`", + "Use lowercase `local` or uppercase `EC2`; runtime provider " + "handling is case-sensitive.", + ) + valid = False + + replicas = entry.get("replicas", 1) + if isinstance(replicas, bool) or not isinstance(replicas, int) or replicas < 1: + report.error( + "V002", + config_path, + line_of(entry, "replicas"), + "`replicas` must be an integer greater than zero", + "CanyonOS creates one placement per replica and cannot deploy " + "an empty or fractional set.", + ) + valid = False + + requirements = entry.get("requirements", []) + if not isinstance(requirements, list) or not all( + isinstance(item, str) and item.strip() for item in requirements + ): + report.error( + "V002", + config_path, + line_of(entry, "requirements"), + "`requirements` must be a list of non-empty strings", + "CanyonOS writes this list into the image requirements file.", + ) + valid = False + + path_key = "workflow_file" if service_type == "workflow" else "entrypoint" + relative = entry.get(path_key) + if not _safe_relative_python_path(relative): + report.error( + "V002", + config_path, + line_of(entry, path_key), + f"`{path_key}` must be a relative .py path contained by `.car/app`", + "Absolute and parent-relative paths escape the self-contained CanyonOS artifact.", + ) + valid = False + elif not os.path.isfile(os.path.join(source_dir, relative)): + report.error( + "V002", + config_path, + line_of(entry, path_key), + f"`{path_key}: {relative}` does not exist in `.car/app`", + "The deploy build cannot create this service without its Python entry file.", + ) + valid = False + + return entries if valid else None + + +def discover_agent_declarations(report, config_dir, config_path): + """Load declarations without silently discarding malformed or duplicate YAML.""" + declarations = {} + for path in sorted(glob.glob(os.path.join(config_dir, "*.yaml"))): + data, error = load_yaml(path) + if error is not None: + report.error( + "V003", + path, + 0, + f"YAML cannot be parsed: {error}", + "CanyonOS reads every YAML file in the config directory during deploy.", + ) + continue + if not isinstance(data, dict): + if os.path.realpath(path) == os.path.realpath(config_path): + report.error( + "V003", + path, + 0, + "the global manifest must be a mapping", + "A scalar or empty manifest has no CanyonOS configuration contract.", + ) + continue + agent = data.get("agent") + if agent is None: + continue + if ( + not isinstance(agent, dict) + or not isinstance(agent.get("name"), str) + or not agent["name"] + ): + report.error( + "V003", + path, + line_of(data, "agent"), + "`agent` must contain a non-empty string `name`", + "The declaration name is the binding used to generate its stub.", + ) + continue + name = agent["name"] + if name in declarations: + report.error( + "V003", + path, + line_of(agent, "name"), + f"duplicate declaration for `{name}`", + "Filename ordering would otherwise choose one declaration silently.", + ) + continue + declarations[name] = (path, agent) + return declarations + + +def check_declaration_bindings(report, entries, declarations, config_path): + """Require a one-to-one binding for every agent service.""" + configured = { + entry["name"] for entry in entries if entry.get("type", "agent") != "workflow" + } + for name in sorted(configured - declarations.keys()): + report.error( + "V004", + config_path, + 0, + f"agent `{name}` has no matching declaration in `.car/config`", + f"Without `agent.name: {name}`, CanyonOS cannot generate the service stub.", + ) + for name in sorted(declarations.keys() - configured): + path, _ = declarations[name] + report.warn( + "V004", + path, + 0, + f"declaration `{name}` has no agent service in the manifest", + "It is stale or unused and will not produce a deployable service.", + ) + + +def check_policy(report, config_dir): + path = os.path.join(config_dir, "policy.yaml") + if not os.path.exists(path): + return + data, error = load_yaml(path) + if error is not None or not isinstance(data, dict): + report.error( + "V005", + path, + 0, + "policy.yaml must be a YAML mapping", + error or "CanyonOS reads `rules` from this mapping during deploy.", + ) + return + rules = data.get("rules") + if not isinstance(rules, list) or not rules: + report.error( + "V005", + path, + line_of(data, "rules"), + "policy.yaml must contain a non-empty `rules` list", + "Remove the file for unrestricted access; an empty policy is not a valid restriction.", + ) diff --git a/.claude/skills/porting-to-canyonos/validation/packaging.py b/.claude/skills/porting-to-canyonos/validation/packaging.py new file mode 100644 index 0000000..d0682fc --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/packaging.py @@ -0,0 +1,108 @@ +"""V030-V031 -- capability-gated rules about credentials and import roots.""" + +import os + +from validation.core import line_of +from validation.python_source import ( + parse_python, + resolves_flat, + resolves_nested, + toplevel_import_names, +) +from validation.runtime import RUNTIME_FLAT_NAMES + + +def check_env_file(report, config, config_path, artifact_dir): + """V030 -- gated on detected env-file injection support.""" + declared = config.get("env_file") + supported = report.capabilities.get("env_file") + + if not supported: + if declared: + report.error( + "V030", + config_path, + line_of(config, "env_file"), + f"`env_file: {declared}` is set, but this CanyonOS Core never reads it", + "No resolve_env_file in the importable ventis package, so the " + "key is silently dropped and the container answers a provider " + "credential error on the first request. This port requires the " + "`env_file` runtime capability.", + ) + else: + report.unavailable( + "V030", + "env_file is not supported by the importable `ventis` runtime. " + "Credentials have no declared path into a container on this tree.", + ) + return + + if not declared: + report.warn( + "V030", + config_path, + line_of(config), + "no `env_file:` in the config", + "Only runtime-managed VENTIS_* variables are guaranteed without it. " + "If the source reads credentials from the environment, the first " + "request fails on a provider error.", + ) + return + + +def check_import_root(report, source_dir, entrypoint_paths): + """V031 -- gated on detected editable-install support.""" + supported = report.capabilities.get("editable_install") + has_metadata = any( + os.path.isfile(os.path.join(source_dir, name)) + for name in ("pyproject.toml", "setup.py", "setup.cfg") + ) + + non_flat = [] + for path in entrypoint_paths: + tree, _ = parse_python(path) + if tree is None: + continue + for name, lineno in toplevel_import_names(tree).items(): + if f"{name}.py" in RUNTIME_FLAT_NAMES: + continue + if resolves_flat(source_dir, name): + continue + location = resolves_nested(source_dir, name) + if location: + non_flat.append((path, lineno, name, location)) + + if not supported: + report.unavailable( + "V031", + "the editable install (`-e .`) is not supported by the importable " + "`ventis` runtime. Only names rooted at /app import inside a container.", + ) + for path, lineno, name, location in non_flat: + report.error( + "V031", + path, + lineno, + f"`import {name}` resolves to {location}, which is not at the " + "root of the source copy", + "sys.path[0] is /app and this CanyonOS Core runs no editable install, " + "so only modules swept to the root import. The adapter raises " + "ModuleNotFoundError inside _load_agent and the first request " + "answers 'No agent loaded'.", + ) + return + + if non_flat and not has_metadata: + for path, lineno, name, location in non_flat: + report.error( + "V031", + path, + lineno, + f"`import {name}` resolves to {location}, and the source copy's " + "root has no packaging metadata", + "A pyproject.toml, setup.py or setup.cfg at the root of the " + "source copy is what adds `-e .`; metadata nested deeper in the " + "tree is ignored. Add minimal root metadata pointing at the " + "existing package directory. Without it the install is skipped " + "silently.", + ) diff --git a/.claude/skills/porting-to-canyonos/validation/python_source.py b/.claude/skills/porting-to-canyonos/validation/python_source.py new file mode 100644 index 0000000..50df1e6 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/python_source.py @@ -0,0 +1,179 @@ +"""Static Python-source discovery used by adapter and packaging checks.""" + +import ast +import os + + +def parse_python(path): + """Return ``(AST, None)`` or ``(None, error)`` without importing the file.""" + try: + with open(path, "r", encoding="utf-8") as handle: + source = handle.read() + except OSError as exc: + return None, str(exc) + try: + return ast.parse(source, filename=path), None + except SyntaxError as exc: + return None, f"{exc.msg} (line {exc.lineno})" + + +def find_class(tree, name): + for node in tree.body: + if isinstance(node, ast.ClassDef) and node.name == name: + return node + return None + + +def class_methods(class_node): + return { + node.name: node + for node in class_node.body + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) + } + + +def parameter_names(func_node): + """Return every keyword-callable parameter, excluding ``self``/``cls``.""" + args = func_node.args + positional = [arg.arg for arg in args.posonlyargs + args.args] + if positional and positional[0] in ("self", "cls"): + positional = positional[1:] + return positional + [arg.arg for arg in args.kwonlyargs] + + +def required_parameters(func_node): + """Return parameters without defaults, excluding ``self``/``cls``.""" + args = func_node.args + positional = [arg.arg for arg in args.posonlyargs + args.args] + if positional and positional[0] in ("self", "cls"): + positional = positional[1:] + if args.defaults: + positional = positional[: len(positional) - len(args.defaults)] + kwonly = [ + arg.arg + for arg, default in zip(args.kwonlyargs, args.kw_defaults) + if default is None + ] + return positional + kwonly + + +def toplevel_import_names(tree): + """Return top-level import names and their first line numbers.""" + names = {} + for node in ast.walk(tree): + if isinstance(node, ast.Import): + for alias in node.names: + names.setdefault(alias.name.split(".")[0], node.lineno) + elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module: + names.setdefault(node.module.split(".")[0], node.lineno) + return names + + +def dotted_import_names(tree): + """Return absolute dotted imports and their first line numbers.""" + names = {} + for node in ast.walk(tree): + if isinstance(node, ast.Import): + for alias in node.names: + names.setdefault(alias.name, node.lineno) + elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module: + names.setdefault(node.module, node.lineno) + return names + + +def _local_module_files(project_dir, dotted): + """Return local files executed by importing ``dotted``, outermost first.""" + parts = dotted.split(".") + found = [] + prefix = project_dir + for depth, part in enumerate(parts): + if os.path.isdir(os.path.join(prefix, part)): + init_path = os.path.join(prefix, part, "__init__.py") + if os.path.isfile(init_path): + found.append(init_path) + prefix = os.path.join(prefix, part) + continue + leaf = os.path.join(prefix, part + ".py") + if depth == len(parts) - 1 and os.path.isfile(leaf): + found.append(leaf) + return found + return found + + +def _relative_import_files(project_dir, path, tree): + """Return local files executed by a module's relative imports.""" + root = os.path.realpath(project_dir) + found = [] + for node in ast.walk(tree): + if not isinstance(node, ast.ImportFrom) or not node.level: + continue + base = os.path.dirname(path) + for _ in range(node.level - 1): + base = os.path.dirname(base) + resolved = os.path.realpath(base) + if resolved != root and not resolved.startswith(root + os.sep): + continue + target = os.path.join(base, *(node.module.split(".") if node.module else [])) + candidates = [target + ".py", os.path.join(target, "__init__.py")] + candidates += [os.path.join(target, alias.name + ".py") for alias in node.names] + candidates += [ + os.path.join(target, alias.name, "__init__.py") for alias in node.names + ] + found += [candidate for candidate in candidates if os.path.isfile(candidate)] + return found + + +def reachable_imports(project_dir, root_path, shadowed_paths=()): + """Return third-party imports reachable from ``root_path`` transitively.""" + external = {} + seen = set() + shadowed = {os.path.realpath(path) for path in shadowed_paths} + queue = [os.path.realpath(root_path)] + while queue: + path = queue.pop() + if path in shadowed or path in seen or not os.path.isfile(path): + continue + seen.add(path) + tree, _ = parse_python(path) + if tree is None: + continue + for dotted, lineno in dotted_import_names(tree).items(): + local = _local_module_files(project_dir, dotted) + if local: + queue += [os.path.realpath(item) for item in local] + else: + external.setdefault(dotted, (path, lineno)) + queue += [ + os.path.realpath(item) + for item in _relative_import_files(project_dir, path, tree) + ] + return external + + +def module_path(entrypoint): + """Dotted module name an entrypoint has inside the container.""" + return os.path.splitext(entrypoint)[0].replace("\\", "/").replace("/", ".") + + +def resolves_flat(project_dir, name): + """Whether Python can resolve `name` with /app as its import root. + + A directory does not need __init__.py: PEP 420 namespace packages resolve + from sys.path just like regular packages. + """ + return os.path.isfile(os.path.join(project_dir, f"{name}.py")) or os.path.isdir( + os.path.join(project_dir, name) + ) + + +def resolves_nested(project_dir, name): + """Where below /app `name` lives but cannot resolve as a top-level name.""" + for root, dirs, files in os.walk(project_dir): + dirs[:] = [d for d in dirs if not d.startswith(".") and d != "__pycache__"] + if root == project_dir: + continue + if f"{name}.py" in files: + return os.path.relpath(os.path.join(root, f"{name}.py"), project_dir) + if name in dirs: + return os.path.relpath(os.path.join(root, name), project_dir) + return None diff --git a/.claude/skills/porting-to-canyonos/validation/runtime.py b/.claude/skills/porting-to-canyonos/validation/runtime.py new file mode 100644 index 0000000..4ec438a --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/runtime.py @@ -0,0 +1,120 @@ +"""Runtime capabilities and dependency facts used by validation checks.""" + +import importlib +import os +import sys + + +RUNTIME_FLAT_NAMES = frozenset( + { + "future.py", + "ventis_context.py", + "local_controller.py", + "local_controller_frontend.py", + "redis_client.py", + "grpc_options.py", + "gpu_metrics.py", + "bedrock.py", + "deploy.py", + "session_logging.py", + "workflow_launcher.py", + } +) + +IMPORT_TO_DISTRIBUTION = { + "attr": ("attrs",), + "autogen": ("pyautogen", "ag2", "autogen", "autogen-agentchat"), + "bs4": ("beautifulsoup4",), + "cv2": ("opencv-python",), + "dateutil": ("python-dateutil",), + "dotenv": ("python-dotenv",), + "grpc": ("grpcio",), + "grpc_tools": ("grpcio-tools",), + "jwt": ("pyjwt",), + "PIL": ("pillow",), + "psycopg": ("psycopg",), + "psycopg2": ("psycopg2-binary",), + "pydantic_settings": ("pydantic-settings",), + "sklearn": ("scikit-learn",), + "typing_extensions": ("typing-extensions",), + "yaml": ("pyyaml",), +} + +NAMESPACE_DISTRIBUTIONS = {"llama_index": "llama-index"} + +CAPABILITY_SOURCE = { + "env_file": "runtime env-file injection", + "editable_install": "editable project installation", + "sweeps_all_files": "full project-file sweep", +} + + +def _base_requirements(): + agent = [ + "grpcio", + "grpcio-tools", + "redis", + "pyyaml", + "psutil", + "ipdb", + "ipython", + "boto3", + ] + workflow = [*agent, "flask", "sqlalchemy", "psycopg[binary]"] + try: + from ventis import stub_generator + except Exception: # noqa: BLE001 - a broken install must not crash validation + return agent, workflow + return ( + list(getattr(stub_generator, "BASE_AGENT_REQUIREMENTS", agent)), + list(getattr(stub_generator, "BASE_WORKFLOW_REQUIREMENTS", workflow)), + ) + + +def _stdlib_names(): + names = getattr(sys, "stdlib_module_names", None) + if names: + return frozenset(names) + found = set(sys.builtin_module_names) + library = os.path.dirname(os.__file__) + try: + entries = os.listdir(library) + except OSError: + return frozenset(found) + for entry in entries: + if entry.endswith(".py"): + found.add(entry[:-3]) + elif "." not in entry and "-" not in entry: + found.add(entry) + return frozenset(found) + + +BASE_AGENT_REQUIREMENTS, BASE_WORKFLOW_REQUIREMENTS = _base_requirements() +STDLIB_MODULE_NAMES = _stdlib_names() + + +def probe_capabilities(): + """Probe the installed compatibility runtime behind the CanyonOS CLI.""" + capabilities = dict.fromkeys(CAPABILITY_SOURCE, False) + capabilities["ventis"] = False + try: + from ventis import stub_generator + except Exception: # noqa: BLE001 - unavailable runtime is reported, not fatal + return capabilities + + capabilities["ventis"] = True + capabilities["editable_install"] = hasattr(stub_generator, "_install_step") + capabilities["sweeps_all_files"] = hasattr(stub_generator, "_sweep_project_files") + + for module_name in ( + "ventis.controller.utils.env_file", + "ventis.utils.env_file", + ): + try: + module = importlib.import_module(module_name) + except Exception: # noqa: BLE001 - try the other supported location + continue + if hasattr(module, "resolve_env_file"): + capabilities["env_file"] = True + break + return capabilities diff --git a/.claude/skills/porting-to-canyonos/validation/workflow.py b/.claude/skills/porting-to-canyonos/validation/workflow.py new file mode 100644 index 0000000..613c4ce --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/workflow.py @@ -0,0 +1,213 @@ +"""V016-V018, V023 -- the workflow module and how it reaches an agent.""" + +import ast + +from validation.python_source import parameter_names, parse_python, required_parameters + + +def check_stub_imports(report, workflow_path, tree, stub_modules): + """V023 -- the workflow must import each agent from its own entrypoint module. + + The build writes a stub over exactly one path: the agent's `entrypoint` + inside the source copy. An import that reaches the class any other way -- + flat, through a package re-export, or from a second copy of the module -- + resolves to the real class instead, and the workflow runs the agent + in-process with none of the deployment behind it. The class name is another + trap: the deploy build prints one with a `Stub` suffix that it never writes. + """ + for node in ast.walk(tree): + if not isinstance(node, ast.ImportFrom) or not node.module: + continue + for alias in node.names: + name = alias.name + base = name.removesuffix("Stub") + expected = stub_modules.get(base) + if expected is None: + continue + if name.endswith("Stub"): + report.error( + "V023", + workflow_path, + node.lineno, + f"`{name}` is the name the build prints, not the class it writes", + "generate_stub sets class_name = agent_config['name'] and " + "then recomputes it with a 'Stub' suffix for the log line " + "only. The message names a class that does not exist; the " + f"class is `{base}`.", + ) + elif node.module != expected: + report.error( + "V023", + workflow_path, + node.lineno, + f"`from {node.module} import {name}` -- the stub for {name} " + f"is written to {expected.replace('.', '/')}.py", + "The build replaces the module at the agent's entrypoint " + "and nothing else, so this import reaches the real class " + "and runs the agent in this process instead of over gRPC. " + f"Import it from `{expected}`, where the source already " + "keeps it.", + ) + + +def check_workflow(report, workflow_path, stub_modules=None): + """V016 V017 V018 V023.""" + tree, error = parse_python(workflow_path) + if tree is None: + report.error("V016", workflow_path, 0, f"does not parse: {error}", "") + return + + main = None + for node in tree.body: + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and ( + node.name == "main" + ): + main = node + break + + if main is None: + defined = [ + n.name + for n in tree.body + if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef)) + ] + found = ", ".join(defined) if defined else "no top-level functions" + report.error( + "V016", + workflow_path, + 1, + f"no top-level function named `main` (found: {found})", + "CanyonOS Core serves POST /, but the deployment platform's " + "test endpoint posts to a hardcoded /main. A differently named " + "workflow builds, deploys and stays unreachable -- 404, container " + "healthy.", + ) + else: + check_main_signature(report, workflow_path, main) + + if stub_modules: + check_stub_imports(report, workflow_path, tree, stub_modules) + + # V016 -- deploy() is what starts Flask. + if not any( + isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "deploy" + for node in ast.walk(tree) + ): + report.error( + "V016", + workflow_path, + 1, + "the workflow never calls `deploy(...)`", + "workflow_launcher.py exec's this file and nothing else starts the " + "HTTP server; the container comes up serving nothing.", + ) + + check_main_guard(report, workflow_path, tree) + check_fused_fanout(report, workflow_path, tree) + + +def check_main_signature(report, workflow_path, main): + """V016 -- the platform sends exactly {"query": ...}.""" + if isinstance(main, ast.AsyncFunctionDef): + report.error( + "V016", + workflow_path, + main.lineno, + "`main` is `async def`", + "deploy() calls workflow_fn(**kwargs) on a Flask worker thread with " + "no await; the response body would be a coroutine repr.", + ) + params = parameter_names(main) + if not params: + report.error( + "V016", + workflow_path, + main.lineno, + "`main` takes no arguments", + 'The platform posts {"query": "..."} and deploy() splats the ' + "body in as kwargs -- TypeError on every request.", + ) + return + if params[0] != "query": + report.error( + "V016", + workflow_path, + main.lineno, + f"`main`'s first parameter is `{params[0]}`, not `query`", + "The platform's body schema is strictly validated as " + "{query: string}; any other key is rejected with 400 in the control " + "plane, before the request reaches the host.", + ) + extra = [p for p in required_parameters(main) if p != "query"] + if extra: + report.error( + "V016", + workflow_path, + main.lineno, + f"`main` requires {', '.join(extra)} beyond `query`", + "Only `query` is ever sent, so every other parameter needs a " + "default or the call raises on every request. Pack richer input " + "into `query`.", + ) + + +def check_main_guard(report, workflow_path, tree): + """V017 -- the workflow is exec'd, so __name__ == "__main__".""" + for node in tree.body: + if not isinstance(node, ast.If): + continue + test = node.test + if ( + isinstance(test, ast.Compare) + and isinstance(test.left, ast.Name) + and test.left.id == "__name__" + and any( + isinstance(c, ast.Constant) and c.value == "__main__" + for c in test.comparators + ) + ): + report.error( + "V017", + workflow_path, + node.lineno, + '`if __name__ == "__main__":` block in the workflow', + "workflow_launcher.py runs exec(open().read()), so " + "__name__ IS '__main__' here and this block executes in " + "production, at container start.", + ) + + +def check_fused_fanout(report, workflow_path, tree): + """V018 -- .value() blocks, so dispatching and resolving in one + comprehension runs the fan-out one call at a time.""" + comprehensions = (ast.ListComp, ast.SetComp, ast.GeneratorExp) + for node in ast.walk(tree): + if not isinstance(node, comprehensions + (ast.DictComp,)): + continue + elements = ( + [node.key, node.value] if isinstance(node, ast.DictComp) else [node.elt] + ) + for element in elements: + for inner in ast.walk(element): + if ( + isinstance(inner, ast.Call) + and isinstance(inner.func, ast.Attribute) + and inner.func.attr == "value" + and isinstance(inner.func.value, ast.Call) + ): + report.error( + "V018", + workflow_path, + node.lineno, + "one comprehension both dispatches a call and resolves " + "it with .value()", + ".value() blocks, so each call completes before the " + "next is dispatched. It does not error -- the fan-out " + "is just silently serial, and with it the reason to be " + "on CanyonOS Core. Dispatch every call first, then resolve: " + "futures = [a.work(i) for i in items] then " + "[f.value() for f in futures].", + ) + return