From 2928d8b004fd49733a8a06ed8e3a3aeb3ef462d1 Mon Sep 17 00:00:00 2001 From: Nick Huo Date: Thu, 3 Sep 2026 12:42:00 -0700 Subject: [PATCH 1/7] Guide porters through canyonos config choices --- .../skills/porting-to-canyonos-core/SKILL.md | 36 +++++++++++++++++- .../references/ec2.md | 6 +++ .../references/manifest.md | 38 +++++++++++++++++++ 3 files changed, 79 insertions(+), 1 deletion(-) diff --git a/.claude/skills/porting-to-canyonos-core/SKILL.md b/.claude/skills/porting-to-canyonos-core/SKILL.md index 3b8f261..11b1250 100644 --- a/.claude/skills/porting-to-canyonos-core/SKILL.md +++ b/.claude/skills/porting-to-canyonos-core/SKILL.md @@ -22,7 +22,8 @@ maps to one line here. Port progress: - [ ] 1. Copy the source into .car/app, rooted at its import root - [ ] 2. Survey the copy and choose service boundaries -- [ ] 3. Read adapter.md + manifest.md; write declarations, adapters, workflow, config +- [ ] 3. Read adapter.md + manifest.md; write declarations, adapters, workflow; + use the `canyonos config` flow to review deployment choices, then write config - [ ] 4. validate.py reports 0 errors - [ ] 5. ventis build succeeds - [ ] 6. Every image passes its probes, including the peer import @@ -251,6 +252,39 @@ access must be restricted; if present, give it a non-empty `rules` list. manifest.md carries the complete manifest and how to build each `requirements` list. +Part of the manifest is derived from the copy; the rest is the developer's +deployment choice, and no reading of the source produces it. Derive what the +source decides. Do not derive what it does not. + +Use the interaction exposed by `cli/canyonos/config.py` as the configuration UX. +Before writing `.car/config/global_controller.yaml`, present the same two +choices -- **View** and **Change** -- rather than silently choosing deployment +settings: + +1. Build the complete candidate manifest in memory from derived values plus the + defaults in manifest.md's ownership table. +2. **View** prints the whole candidate, not a summary. Clearly annotate defaults + and values constrained by the source, such as `replicas: 1` for stateful + in-memory services. +3. **Change** asks, in one batch, only for developer-owned values: provider and + its EC2 block, replicas that are not constrained, resources, ports, secret + file location, and whether access needs restricting. Show the current/default + value for every choice. Apply the answers and show the resulting manifest. +4. Write the reviewed candidate, then run the validator. + +If the coding environment can launch an interactive command, prefer running +`canyonos config` for this review. Otherwise reproduce its View/Change flow in +the conversation; do not skip the review merely because the CLI has no TTY. +Do not ask the developer for derived values such as `entrypoint` or +`requirements`: walking the copied source gives a more reliable answer. + +Never block an unattended port on this interaction. `canyonos integrate` +launches this skill from a prompt, and a corpus run has no one at the terminal. +If no answer is available, write the displayed defaults, name them in the final +report, and keep going. The one exception is a value with no safe default: +`provider: EC2` needs infrastructure identifiers that are wrong to invent, so +an unanswered EC2 choice leaves the entry `local` and says so. + ## Hard rules Capitalized **MUST** and **NEVER** are reserved for port-breaking or diff --git a/.claude/skills/porting-to-canyonos-core/references/ec2.md b/.claude/skills/porting-to-canyonos-core/references/ec2.md index e06daa0..3a80363 100644 --- a/.claude/skills/porting-to-canyonos-core/references/ec2.md +++ b/.claude/skills/porting-to-canyonos-core/references/ec2.md @@ -9,6 +9,12 @@ top-level `ec2` block supplies the runtime's required infrastructure and SSH settings. Read the target checkout's deploy preflight and EC2 runtime before writing the block; do not copy values from an example environment. +These identifiers come from the developer, in step 3's config round -- they are +the one part of the manifest with no safe default. If the round produces no +answer, leave the entry `provider: local` and report that EC2 was requested but +not configured. Never fill the block from an example, a previous port, or +another entry in the same manifest. + Typical required categories are: - AMI and instance type diff --git a/.claude/skills/porting-to-canyonos-core/references/manifest.md b/.claude/skills/porting-to-canyonos-core/references/manifest.md index e1ffb7d..a098469 100644 --- a/.claude/skills/porting-to-canyonos-core/references/manifest.md +++ b/.claude/skills/porting-to-canyonos-core/references/manifest.md @@ -6,10 +6,48 @@ and then decide whether a container can import its own dependencies. ## Contents +- Who decides each key - Agent yaml - Requirements - The manifest, in full +## Who decides each key + +Two kinds of key share one file. A **derived** key has exactly one right answer +and the copy holds it; asking the developer can only make it worse. A +**developer** key is a deployment choice the source does not contain, and +deriving it means guessing and presenting the guess as a reading. + +SKILL.md step 3 shows the whole manifest and then asks about the second column +only, in one round, carrying these defaults. + +| Key | Decided by | Default when unanswered | +|---|---|---| +| `name`, `entrypoint`, `workflow_file`, `type` | derived — service boundaries, step 2 | — | +| `requirements` | derived — the entry's import graph | — | +| `database` | neither; omit it always (see below) | absent | +| `provider` | developer | `local` | +| `ec2:` block, `instance_type` | developer — no default is safe | entry stays `local` | +| `replicas` | developer, *unless* cross-request state forces `1` | `1` | +| `resources.cpu` / `resources.memory` | developer | `1` / `512` MiB | +| `api_port` | developer | `8080` | +| `redis_port`, `redis.host` / `.port` / `.db` | developer | `6379`, `localhost` / `6379` / `0` | +| `poll_interval` | developer | `5` | +| `env_file` | developer — the file's location and whether it exists | `.env` when the survey found credential reads, else absent | +| `policy.yaml` | developer | absent | + +Two entries in that table are not free choices, and saying so is part of showing +the config rather than asking about it: + +- **`replicas` stops being a choice once a service holds cross-request state.** + Where the step-2 survey found such state, SKILL.md already fixes `replicas: 1` + as a correctness requirement, so `1` is derived: report it as a constraint and + do not offer to raise it. +- **EC2 identifiers are wrong to invent.** ec2.md forbids copying them from an + example environment, and a wrong AMI, subnet, or security group fails at + deploy preflight or, worse, provisions something unreachable. Unanswered + means the entry stays `local`. + ## Agent yaml Declarations go in `.car/config/`, beside the manifest. The build reads every From dd9f6b789c0a537beb57a65dcadc2206c284ceaa Mon Sep 17 00:00:00 2001 From: Nick Huo Date: Thu, 3 Sep 2026 14:49:27 -0700 Subject: [PATCH 2/7] Ignore stub-shadowed dependencies in workflow validation --- .../porting-to-canyonos-core/validate.py | 28 +++++++++++++++---- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/.claude/skills/porting-to-canyonos-core/validate.py b/.claude/skills/porting-to-canyonos-core/validate.py index edc21c6..a8f38c9 100755 --- a/.claude/skills/porting-to-canyonos-core/validate.py +++ b/.claude/skills/porting-to-canyonos-core/validate.py @@ -432,8 +432,8 @@ def _relative_import_files(project_dir, path, tree): return found -def reachable_imports(project_dir, root_path): - """Every third-party import the image executes from `root_path`, transitively. +def reachable_imports(project_dir, root_path, shadowed_paths=()): + """Every third-party import the built image executes, transitively. An image runs far more than the file the config names. `tools/parser.py` is one hop from an entrypoint and its pdfplumber import is invisible to an AST @@ -441,15 +441,21 @@ def reachable_imports(project_dir, root_path): graphql-core. Both surface only as a ModuleNotFoundError deep inside an import chain at agent load, long after a green build. + `shadowed_paths` are real source modules replaced in this image before it + runs. In particular, a workflow image receives generated stubs over every + agent entrypoint, so following the overwritten implementations would report + dependencies that image never imports. + Returns {dotted import name: (file that imports it, line)} for names that do not resolve inside the copy. """ external = {} seen = set() + shadowed = {os.path.realpath(path) for path in shadowed_paths} queue = [os.path.realpath(root_path)] while queue: path = queue.pop() - if path in seen or not os.path.isfile(path): + if path in shadowed or path in seen or not os.path.isfile(path): continue seen.add(path) tree, _ = parse_python(path) @@ -1191,7 +1197,13 @@ def _pyproject_dependencies(project_dir): def check_requirements_coverage( - report, project_dir, entry, root_path, config_path, base_requirements + report, + project_dir, + entry, + root_path, + config_path, + base_requirements, + shadowed_paths=(), ): """W006 -- an import the container cannot satisfy. @@ -1225,7 +1237,7 @@ def check_requirements_coverage( base = {_normalize_distribution(item) for item in base_requirements} satisfied = base | declared - external = reachable_imports(project_dir, root_path) + external = reachable_imports(project_dir, root_path, shadowed_paths) for dotted, (where, lineno) in sorted(external.items()): name = dotted.split(".")[0] if name in STDLIB_MODULE_NAMES or name == "ventis": @@ -1392,6 +1404,11 @@ def validate(artifact_dir, config_path, capabilities): for name, entrypoint in entrypoints if name in agents_by_name } + stubbed_entrypoint_paths = [ + os.path.join(source_dir, entrypoint) + for name, entrypoint in entrypoints + if name in agents_by_name + ] for entry in entries: if not isinstance(entry, dict) or entry.get("type", "agent") != "workflow": @@ -1412,6 +1429,7 @@ def validate(artifact_dir, config_path, capabilities): workflow_path, config_path, BASE_WORKFLOW_REQUIREMENTS, + shadowed_paths=stubbed_entrypoint_paths, ) # These survive a green build and otherwise surface only in a container or From 42a6efe5aa48cc233dc10ff480b39753b1ff5d5e Mon Sep 17 00:00:00 2001 From: Nick Huo Date: Thu, 3 Sep 2026 16:03:50 -0700 Subject: [PATCH 3/7] Standardize CanyonOS artifact preparation --- .../SKILL.md | 46 +++--- .claude/skills/porting-to-canyonos/prepare.py | 156 ++++++++++++++++++ .../references/adapter.md | 0 .../references/ec2.md | 0 .../references/example-port.md | 6 +- .../references/llm-proxy.md | 0 .../references/manifest.md | 0 .../references/packaging.md | 0 .../references/runtime-contract.md | 0 .../references/troubleshooting.md | 0 .../validate.py | 0 11 files changed, 186 insertions(+), 22 deletions(-) rename .claude/skills/{porting-to-canyonos-core => porting-to-canyonos}/SKILL.md (91%) create mode 100755 .claude/skills/porting-to-canyonos/prepare.py rename .claude/skills/{porting-to-canyonos-core => porting-to-canyonos}/references/adapter.md (100%) rename .claude/skills/{porting-to-canyonos-core => porting-to-canyonos}/references/ec2.md (100%) rename .claude/skills/{porting-to-canyonos-core => porting-to-canyonos}/references/example-port.md (96%) rename .claude/skills/{porting-to-canyonos-core => porting-to-canyonos}/references/llm-proxy.md (100%) rename .claude/skills/{porting-to-canyonos-core => porting-to-canyonos}/references/manifest.md (100%) rename .claude/skills/{porting-to-canyonos-core => porting-to-canyonos}/references/packaging.md (100%) rename .claude/skills/{porting-to-canyonos-core => porting-to-canyonos}/references/runtime-contract.md (100%) rename .claude/skills/{porting-to-canyonos-core => porting-to-canyonos}/references/troubleshooting.md (100%) rename .claude/skills/{porting-to-canyonos-core => porting-to-canyonos}/validate.py (100%) diff --git a/.claude/skills/porting-to-canyonos-core/SKILL.md b/.claude/skills/porting-to-canyonos/SKILL.md similarity index 91% rename from .claude/skills/porting-to-canyonos-core/SKILL.md rename to .claude/skills/porting-to-canyonos/SKILL.md index 11b1250..9f8ddc1 100644 --- a/.claude/skills/porting-to-canyonos-core/SKILL.md +++ b/.claude/skills/porting-to-canyonos/SKILL.md @@ -5,8 +5,8 @@ description: Ports existing LangChain, LangGraph, CrewAI, AutoGen, and hand-roll # Port an agent project to CanyonOS Core -Requires Python, Docker, and the `ventis` CLI. `validate.py` in this skill needs -Python 3 and `pyyaml`. +Requires Python, Docker, and the `ventis` CLI. `prepare.py` uses only the Python +standard library; `validate.py` needs Python 3 and `pyyaml`. CanyonOS Core is the product name. Its compatibility executable and Python package remain `ventis`; environment variables and Docker resources retain the @@ -20,7 +20,7 @@ maps to one line here. ``` Port progress: -- [ ] 1. Copy the source into .car/app, rooted at its import root +- [ ] 1. Choose the import root; run prepare.py to create .car/config and .car/app - [ ] 2. Survey the copy and choose service boundaries - [ ] 3. Read adapter.md + manifest.md; write declarations, adapters, workflow; use the `canyonos config` flow to review deployment choices, then write config @@ -109,29 +109,35 @@ model clients, and node bodies—is imported from where the copy keeps it. The port re-expresses only the CanyonOS Core boundary and framework-owned orchestration. -## 1. Duplicate the source, then survey it +## 1. Prepare the artifact tree, then survey it -Copy the application source into `.car/app/`, preserving its structure. Leave -out only what no container should carry: `.git/`, `.car/` itself, virtualenvs, -caches, build outputs, and `.env` files holding real credentials. - -**Root the copy at the source's import root, which is not always its repository -root.** `/app` is the copy, and without the editable-install capability it is -the only entry on `sys.path`, so a source under `src/` that imports -`from tools import ...` needs the *contents* of `src/` at `.car/app/`. Read the -source's own imports, not its directory names, to decide. Getting it wrong -builds green and answers `No agent loaded` on the first request; +**Choose the source's import root, which is not always its repository root.** +`/app` is the copy, and without the editable-install capability it is the only +entry on `sys.path`, so a source under `src/` that imports `from tools import +...` needs the *contents* of `src/` at `.car/app/`. Read the source's own +imports, not its directory names, to decide. Getting it wrong builds green and +answers `No agent loaded` on the first request; [references/packaging.md](references/packaging.md) works the case through. +Once the import root is known, use the skill's preparation script rather than +assembling `.car` with ad hoc copy commands: + ```bash -mkdir -p .car/config -rsync -a --exclude '.git' --exclude '.car' --exclude '.venv' --exclude 'venv' \ - --exclude '__pycache__' --exclude '.env' / .car/app/ +python /prepare.py .car ``` -Every edit from here on is inside `.car`. The application source outside it is -read-only for the rest of the port -- `git status` at the end shows `.car/` and -nothing else. +The script creates `.car/config/` and copies the import root's **contents** into +`.car/app/`, preserving its structure. It excludes version-control data, +`.car`, virtualenvs, caches, build outputs, bytecode, and credential-bearing +`.env*` files while retaining `.env.example`, `.env.sample`, and +`.env.template`. It refuses to merge into an existing `.car/app`; use `--force` +only when intentionally replacing the entire source copy. `--force` leaves an +existing `.car/config/` unchanged. + +Choosing the import root remains a porter decision; the script standardizes +only directory creation and copying. After it runs, every edit is inside +`.car`. The application source outside it is read-only for the rest of the port +-- `git status` at the end shows `.car/` and nothing else. Then survey the copy. Identify: diff --git a/.claude/skills/porting-to-canyonos/prepare.py b/.claude/skills/porting-to-canyonos/prepare.py new file mode 100755 index 0000000..9f282b7 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/prepare.py @@ -0,0 +1,156 @@ +#!/usr/bin/env python3 +"""Create a CanyonOS Core artifact tree from a chosen Python import root. + +The script owns the mechanical part of step 1: it creates ``.car/config`` and +copies the selected import root to ``.car/app`` with development artifacts and +credential files excluded. Choosing the correct import root still requires +reading the application's imports. +""" + +import argparse +import shutil +import sys +import uuid +from pathlib import Path + +EXCLUDED_DIRECTORIES = frozenset( + { + ".car", + ".git", + ".hg", + ".mypy_cache", + ".pytest_cache", + ".ruff_cache", + ".tox", + ".venv", + ".svn", + "__pycache__", + "build", + "dist", + "env", + "htmlcov", + "node_modules", + "venv", + } +) +EXCLUDED_FILE_SUFFIXES = (".pyc", ".pyo") +ENV_TEMPLATES = frozenset({".env.example", ".env.sample", ".env.template"}) + + +def _is_relative_to(path: Path, parent: Path) -> bool: + try: + path.relative_to(parent) + except ValueError: + return False + return True + + +def _ignore_factory(artifact_root: Path): + def ignore(directory: str, names: list[str]) -> set[str]: + directory_path = Path(directory) + ignored = set() + for name in names: + path = directory_path / name + if path.resolve() == artifact_root: + ignored.add(name) + elif path.is_dir() and name in EXCLUDED_DIRECTORIES: + ignored.add(name) + elif name.startswith(".env") and name not in ENV_TEMPLATES: + ignored.add(name) + elif name.endswith(EXCLUDED_FILE_SUFFIXES): + ignored.add(name) + return ignored + + return ignore + + +def prepare(import_root: Path, artifact_root: Path, force: bool = False) -> None: + import_root = import_root.expanduser().resolve() + artifact_root = artifact_root.expanduser().resolve() + app_dir = artifact_root / "app" + config_dir = artifact_root / "config" + + if not import_root.is_dir(): + raise ValueError(f"import root is not a directory: {import_root}") + if import_root == artifact_root: + raise ValueError("artifact root cannot also be the import root") + if _is_relative_to(import_root, artifact_root): + raise ValueError("import root cannot be inside the artifact root") + if app_dir.exists() and not app_dir.is_dir(): + raise ValueError(f"app path exists but is not a directory: {app_dir}") + if app_dir.exists() and not force: + raise FileExistsError( + f"{app_dir} already exists; use --force to replace only the source copy" + ) + if config_dir.exists() and not config_dir.is_dir(): + raise ValueError(f"config path exists but is not a directory: {config_dir}") + + artifact_root.mkdir(parents=True, exist_ok=True) + temporary_app = artifact_root / f".app-{uuid.uuid4().hex}.tmp" + previous_app = artifact_root / f".app-{uuid.uuid4().hex}.previous" + + installed_new_app = False + try: + shutil.copytree( + import_root, + temporary_app, + ignore=_ignore_factory(artifact_root), + copy_function=shutil.copy2, + symlinks=True, + ) + if app_dir.exists(): + app_dir.rename(previous_app) + temporary_app.rename(app_dir) + installed_new_app = True + config_dir.mkdir(exist_ok=True) + except Exception: + if temporary_app.exists(): + shutil.rmtree(temporary_app) + if installed_new_app and app_dir.exists(): + shutil.rmtree(app_dir) + if previous_app.exists(): + previous_app.rename(app_dir) + raise + + if previous_app.exists(): + shutil.rmtree(previous_app) + + +def parse_args(argv=None): + parser = argparse.ArgumentParser( + description="Create .car/config and copy an import root into .car/app." + ) + parser.add_argument( + "import_root", + help="directory whose contents should become the contents of .car/app", + ) + parser.add_argument( + "artifact_root", + nargs="?", + default=".car", + help="artifact directory to create (default: .car)", + ) + parser.add_argument( + "--force", + action="store_true", + help="replace an existing app/ copy; leave config/ unchanged", + ) + return parser.parse_args(argv) + + +def main(argv=None) -> int: + args = parse_args(argv) + try: + prepare(Path(args.import_root), Path(args.artifact_root), args.force) + except (OSError, ValueError) as error: + sys.stderr.write(f"prepare.py: {error}\n") + return 1 + + artifact_root = Path(args.artifact_root) + print(f"Created {artifact_root / 'config'}") + print(f"Copied {Path(args.import_root)} to {artifact_root / 'app'}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/.claude/skills/porting-to-canyonos-core/references/adapter.md b/.claude/skills/porting-to-canyonos/references/adapter.md similarity index 100% rename from .claude/skills/porting-to-canyonos-core/references/adapter.md rename to .claude/skills/porting-to-canyonos/references/adapter.md diff --git a/.claude/skills/porting-to-canyonos-core/references/ec2.md b/.claude/skills/porting-to-canyonos/references/ec2.md similarity index 100% rename from .claude/skills/porting-to-canyonos-core/references/ec2.md rename to .claude/skills/porting-to-canyonos/references/ec2.md diff --git a/.claude/skills/porting-to-canyonos-core/references/example-port.md b/.claude/skills/porting-to-canyonos/references/example-port.md similarity index 96% rename from .claude/skills/porting-to-canyonos-core/references/example-port.md rename to .claude/skills/porting-to-canyonos/references/example-port.md index 8308f18..56bb4f0 100644 --- a/.claude/skills/porting-to-canyonos-core/references/example-port.md +++ b/.claude/skills/porting-to-canyonos/references/example-port.md @@ -33,10 +33,12 @@ until the model calls `Done`. is `src/`, not the repository root: ```bash -rsync -a --exclude '.git' --exclude '.car' --exclude '__pycache__' \ - --exclude '.env' src/ .car/app/ +python /prepare.py src .car ``` +This creates `.car/config/` and copies the contents of `src/` into `.car/app/` +with the standard source and credential exclusions. + Copying the repository root instead puts those modules at `/app/src/tools` while `/app` is the only entry on `sys.path`. The build stays green, the replica reports healthy, and the first request answers `No agent loaded` with diff --git a/.claude/skills/porting-to-canyonos-core/references/llm-proxy.md b/.claude/skills/porting-to-canyonos/references/llm-proxy.md similarity index 100% rename from .claude/skills/porting-to-canyonos-core/references/llm-proxy.md rename to .claude/skills/porting-to-canyonos/references/llm-proxy.md diff --git a/.claude/skills/porting-to-canyonos-core/references/manifest.md b/.claude/skills/porting-to-canyonos/references/manifest.md similarity index 100% rename from .claude/skills/porting-to-canyonos-core/references/manifest.md rename to .claude/skills/porting-to-canyonos/references/manifest.md diff --git a/.claude/skills/porting-to-canyonos-core/references/packaging.md b/.claude/skills/porting-to-canyonos/references/packaging.md similarity index 100% rename from .claude/skills/porting-to-canyonos-core/references/packaging.md rename to .claude/skills/porting-to-canyonos/references/packaging.md diff --git a/.claude/skills/porting-to-canyonos-core/references/runtime-contract.md b/.claude/skills/porting-to-canyonos/references/runtime-contract.md similarity index 100% rename from .claude/skills/porting-to-canyonos-core/references/runtime-contract.md rename to .claude/skills/porting-to-canyonos/references/runtime-contract.md diff --git a/.claude/skills/porting-to-canyonos-core/references/troubleshooting.md b/.claude/skills/porting-to-canyonos/references/troubleshooting.md similarity index 100% rename from .claude/skills/porting-to-canyonos-core/references/troubleshooting.md rename to .claude/skills/porting-to-canyonos/references/troubleshooting.md diff --git a/.claude/skills/porting-to-canyonos-core/validate.py b/.claude/skills/porting-to-canyonos/validate.py similarity index 100% rename from .claude/skills/porting-to-canyonos-core/validate.py rename to .claude/skills/porting-to-canyonos/validate.py From 7b7713c6746ccbf47b76a488bd91a4c881a8f9d2 Mon Sep 17 00:00:00 2001 From: Nick Huo Date: Thu, 3 Sep 2026 18:00:45 -0700 Subject: [PATCH 4/7] Stop CanyonOS ports at validation --- .claude/skills/porting-to-canyonos/SKILL.md | 170 +++++++----------- .../porting-to-canyonos/references/adapter.md | 8 +- .../porting-to-canyonos/references/ec2.md | 19 +- .../references/example-port.md | 30 ++-- .../references/llm-proxy.md | 6 +- .../references/manifest.md | 7 +- .../references/packaging.md | 33 +++- .../references/runtime-contract.md | 44 ++--- .../references/troubleshooting.md | 13 +- 9 files changed, 169 insertions(+), 161 deletions(-) diff --git a/.claude/skills/porting-to-canyonos/SKILL.md b/.claude/skills/porting-to-canyonos/SKILL.md index 9f8ddc1..2f8f02a 100644 --- a/.claude/skills/porting-to-canyonos/SKILL.md +++ b/.claude/skills/porting-to-canyonos/SKILL.md @@ -1,17 +1,18 @@ --- -name: porting-to-canyonos-core -description: Ports existing LangChain, LangGraph, CrewAI, AutoGen, and hand-rolled Python agent projects onto CanyonOS Core, whose CLI is `ventis` and whose artifacts live in a `.car` directory. Writes the `.car/config` manifest and declarations, duplicates the source into `.car/app`, writes adapters and the workflow, then validates, builds, deploys and probes. Use when converting, migrating, adapting, packaging, building or deploying an existing agent or multi-agent project onto CanyonOS Core or ventis, when running `ventis build` or `ventis deploy`, or when a `.car` port fails to build, load an agent, or answer a request. +name: porting-to-canyonos +description: Ports existing LangChain, LangGraph, CrewAI, AutoGen, and hand-rolled Python agent projects onto CanyonOS Core, whose CLI is `canyonos` and whose artifacts live in a `.car` directory. Writes `.car/config`, copies source into `.car/app`, writes adapters and the workflow, and validates the port. Stops after validation and asks before running `canyonos deploy`, which performs both build and deployment. Use when converting, migrating, adapting, packaging, validating, or deploying an existing agent or multi-agent project onto CanyonOS Core, or when a `.car` port fails validation, build, load, or deployment. --- # Port an agent project to CanyonOS Core -Requires Python, Docker, and the `ventis` CLI. `prepare.py` uses only the Python -standard library; `validate.py` needs Python 3 and `pyyaml`. +Requires Python, Docker, and the `canyonos` CLI. `prepare.py` uses only the +Python standard library; `validate.py` needs Python 3 and `pyyaml`. -CanyonOS Core is the product name. Its compatibility executable and Python -package remain `ventis`; environment variables and Docker resources retain the -`VENTIS_*` and `ventis-*` prefixes. These are protocol identifiers, not branding -strings. Do not rename them. +CanyonOS Core is the product name and `canyonos` is its user-facing CLI. The +internal compatibility Python package, environment variables, and Docker +resources retain the `ventis`, `VENTIS_*`, and `ventis-*` names. These are +protocol identifiers, not CLI instructions or branding strings. Do not rename +them, and do not tell users to run the obsolete `ventis` CLI. ## Port checklist @@ -24,15 +25,12 @@ Port progress: - [ ] 2. Survey the copy and choose service boundaries - [ ] 3. Read adapter.md + manifest.md; write declarations, adapters, workflow; use the `canyonos config` flow to review deployment choices, then write config -- [ ] 4. validate.py reports 0 errors -- [ ] 5. ventis build succeeds -- [ ] 6. Every image passes its probes, including the peer import -- [ ] 7. A real request returns through /status -- [ ] 8. Clean up; git status shows nothing outside .car +- [ ] 4. validate.py reports 0 errors; report readiness and stop ``` -Do not skip step 4. Every failure mode it reports survives a green build and a -healthy replica, and then costs a deploy cycle to rediscover. +Do not skip step 4. The porting workflow ends when validation reports 0 errors: +report the files created, warnings and unresolved runtime blockers, then stop. +Never build or deploy as an implicit continuation of the port. ## References @@ -53,8 +51,8 @@ optional. deployment, all three knowable at step 1. - [references/packaging.md](references/packaging.md) -- read when a source - import does not resolve from `/app`, the source is nested, or packaging - metadata is involved. + import does not resolve from `/app`, the source is nested, packaging metadata + is involved, or the source reads non-Python files at runtime. - [references/llm-proxy.md](references/llm-proxy.md) -- read when the target includes `llm_proxy`. - [references/ec2.md](references/ec2.md) -- read when any config entry uses @@ -62,8 +60,9 @@ deployment, all three knowable at step 1. **After something failed.** Triggered by a symptom. -- [references/troubleshooting.md](references/troubleshooting.md) -- read after a - failed build, image probe, deploy, or request; symptom-to-cause tables. +- [references/troubleshooting.md](references/troubleshooting.md) -- read after an + explicitly approved deploy fails during build, startup, or a request; + symptom-to-cause tables. - [references/runtime-contract.md](references/runtime-contract.md) -- read when a validator finding needs explanation or the runtime mechanism is unclear. @@ -92,7 +91,7 @@ never inside it: declaration Canyon owns, and `app/`, the copy that becomes `/app` in every container. The container keeps the directory structure the application already had. Write adapters into that copy, in the module the code they wrap already -lives in -- not into new `agents/` and `workflow/` directories. `ventis` +lives in -- not into new `agents/` and `workflow/` directories. `canyonos` commands run from the application root and read `.car` below it. Nothing under `.car` points back out at the application source, and nothing in @@ -141,7 +140,10 @@ only directory creation and copying. After it runs, every edit is inside Then survey the copy. Identify: -1. The source entry point and callable input/output. +1. The source entry point and callable input/output. When the repository has + several plausible implementations, trace the imports from the production + route, CLI, or documented launch path; do not choose the most convenient + graph by filename. 2. Framework-owned control flow (`StateGraph`, `Crew`, `GroupChat`, routing, `Send`, `Command`, interrupts). 3. Runtime-injected services nodes read: stores, context, memory, sessions, or @@ -151,8 +153,20 @@ Then survey the copy. Identify: 6. Model provider, credential names, streaming use, and optional `llm_proxy`. 7. Whether independent work fans out and benefits from separate replicas. 8. Whether source imports resolve from `.car/app`, the root that becomes - `/app`. This is the check the validator turns into V031, and it is the one - most likely to survive a green build and a healthy replica. + `/app`. This is the check the validator turns into V031. +9. Every non-Python file reached at runtime: prompts, CrewAI + `agents.yaml`/`tasks.yaml`, PDFs, templates, schemas, and local corpora. If + `sweeps_all_files` is unavailable, these are runtime blockers even though + `prepare.py` copied them. Do not invent base64 embedding or rewrite hardcoded + paths; report and stop. +10. Whether every Python file on the selected import graph parses. A syntax + error already present in the source is a source defect, not permission to + repair behavior silently. Report it and obtain approval before fixing only + the `.car/app` copy. +11. Suspicious module-level behavior before any import or execution: obfuscated + payloads, network downloads, shell/process calls, credential harvesting, or + destructive filesystem operations. Stop and ask the user when found; do not + import, build, or deploy untrusted code merely because it is a port target. Run the validator now, and again after every change until it reports 0 errors. Execute it; do not read it. Its header detects capabilities directly from the @@ -162,8 +176,9 @@ importable runtime rather than from release history: python /validate.py .car ``` -If config or agent yaml is malformed, the validator defers to `ventis build`. -Capability-gated findings say which runtime behavior is available. +If config or agent yaml is malformed, the validator defers that failure to the +build phase inside a later `canyonos deploy`. Capability-gated findings say +which runtime behavior is available. ## 2. Choose service boundaries @@ -303,18 +318,18 @@ source-integrity rules. The owner column states where each is decided. | M3 | yaml argument names MUST match Python parameter names | V008 | | M4 | yaml argument types MUST be bare builtins | V010 | | M5 | Declared adapter methods MUST be synchronous | V009 | -| M6 | Config names MUST match yaml agent names | build | -| M7 | Config names MUST not collide after lowercase normalization | build output | -| M8 | Local provider MUST be lowercase `local` | deploy preflight | -| M9 | `replicas` MUST be an integer | deploy preflight | -| M10 | `requirements` MUST be a list of strings | build | +| M6 | Config names MUST match yaml agent names | `canyonos deploy` build phase | +| M7 | Config names MUST not collide after lowercase normalization | `canyonos deploy` build output | +| M8 | Local provider MUST be lowercase `local` | `canyonos deploy` preflight | +| M9 | `replicas` MUST be an integer | `canyonos deploy` preflight | +| M10 | `requirements` MUST be a list of strings | `canyonos deploy` build phase | | M11 | Workflow MUST expose `main(query)`; extra parameters MUST default | V016 | | M12 | Workflow MUST NEVER contain a main guard | V017 | | M13 | Fan-out MUST dispatch all calls before resolving any | V018 | | M14 | A module at the root of the copy MUST not take a runtime flat name | V019 | | M15 | Workflow MUST import each agent from its own `entrypoint` module | V023 | -| M16 | Policy MUST be absent or contain a non-empty `rules` list | deploy preflight | -| M17 | EC2 entries MUST satisfy the EC2 deployment contract | deploy preflight | +| M16 | Policy MUST be absent or contain a non-empty `rules` list | `canyonos deploy` preflight | +| M17 | EC2 entries MUST satisfy the EC2 deployment contract | `canyonos deploy` preflight | | M18 | NEVER copy source prompts, tools, schemas, or model calls | review | | M19 | NEVER hardcode or bake a real credential into an image | W003 | | M20 | NEVER write outside `.car`; the application source stays untouched | `git status` | @@ -325,91 +340,42 @@ source-integrity rules. The owner column states where each is decided. | M25 | Two agents MUST NOT share one `entrypoint` | V020 | | M26 | `.car` MUST hold `config/` beside `app/`, the source copy | V032 | | M27 | `.car/app` MUST be rooted at the source's import root | V031 | -| M28 | `requirements` MUST cover every distribution that entry's import graph reaches, transitively | W006, probe 2 | -| M29 | The entrypoint MUST NOT be a module another image imports for its real contents | probe 3 | +| M28 | `requirements` MUST cover every distribution that entry's import graph reaches, transitively | W006, dependency review | +| M29 | The entrypoint MUST NOT be a module another image imports for its real contents | adapter.md review | | M30 | The entrypoint's package `__init__.py` MUST NOT re-export from it | V033 | | M31 | Every segment of the `entrypoint` path MUST be a Python identifier | V034 | | M32 | The entrypoint's own imports MUST be absolute | V035 | -## 4. Validate, build, and probe +## 4. Validate and stop -Run static preflight, then let the build own build-time validation. Both run -from the application root: +Run static preflight from the application root: ```bash python /validate.py .car -ventis build ``` -Fix every ERROR and re-run the validator until it reports 0 errors before -running `ventis build`. A build that skips this passes, and the port then fails -at `docker run` or on the first request, where the message names a container -rather than the mistake. +Fix every ERROR and re-run until it reports 0 errors. Warnings and capability +limitations are not permission to hide risk: list each one in the handoff and +say whether it blocks this source. Confirm that `git status` outside `.car` +shows no change to a file the developer owns. -A green build never imports the adapter. Probe in this order: +At that point, report that the `.car` port is validated and stop. Ask the user a +direct yes/no question before taking the next step: -```bash -# 1. Runtime startup path. Every image, agent and workflow alike. - docker run --rm ventis- \ - python -c "import local_controller" - -# 2. Agent load path. Name the module after the entrypoint's own path, exactly -# as _load_agent does -- spec_from_file_location('m', ...) sets a __name__ the -# runtime never uses and hides relative-import failures until deploy. - docker run --rm --env-file ventis- \ - python -c "import importlib.util,sys; \ -p='';n=p[:-3]; \ -s=importlib.util.spec_from_file_location(n,p); \ -m=importlib.util.module_from_spec(s);sys.modules[n]=m;s.loader.exec_module(m); \ -m.();print('ok')" - -# 3. Peer-import path, in the workflow image: here the stub stands where the -# entrypoint was and the package around it is real. - docker run --rm ventis- \ - python -c "from import ;print('ok')" -``` - -Probe 2 needs `--env-file` in the ordinary case, not the exceptional one: a -source that builds its client at module scope (`client = Anthropic()`) fails at -import without it, and the SDK raises on a missing key even when the key is a -placeholder pointed at a proxy. +> Validation passed. Run `canyonos deploy` now? This will build images and start +> the deployment. -Probe 3 is the only one that exercises what the workflow container does at -startup. It is what catches a package `__init__` re-export against a stub, and a -distribution the workflow image needs only because the entrypoint's package -siblings import it. Also probe the workflow image with -`python -c "import local_controller"`; it has its own dependency resolve. +Do not run a standalone build first. `canyonos deploy` owns both build and +deployment, and must run only after explicit user approval. Silence, an +unattended run, or the original request to "port" is not approval. -Then deploy, send a representative request, and poll its status: +If the user approves, run from the application root: ```bash -ventis deploy -curl -X POST http://localhost:8080/main \ - -H 'Content-Type: application/json' -d '{"query":""}' -curl http://localhost:8080/status/ +canyonos deploy ``` -A successful outer request with a source-level failure still proves the port -reached and returned the source behavior. Record the distinction. - -## 5. Clean up - -After collecting evidence, stop foreground deploy with Ctrl+C and wait for -controller cleanup. Remove exact leftovers if startup crashed. Then remove build -products and exact images from this config: - -```bash -ventis clean -docker image rm ventis- \ - ventis- - -test ! -e .car/stubs && test ! -e .car/grpc_stubs && test ! -e .car/docker_container -docker ps -a --format '{{.Names}}' -``` - -`ventis clean` removes only `.car/stubs/`, `.car/grpc_stubs/`, and -`.car/docker_container/`; it does not remove containers or images. Keep -`.car/config`, `.car/app`, and requested logs or reports. - -Finally, confirm the decoupling held: `git status` outside `.car` reports no -change to any file the developer owns. +Report build or deployment failures without silently changing source behavior, +dependencies, provider, or deployment settings. `canyonos deploy` follows the +controller logs; Ctrl+C stops log monitoring, not necessarily the deployment. +Use `canyonos stop` only when the user asks to stop it. diff --git a/.claude/skills/porting-to-canyonos/references/adapter.md b/.claude/skills/porting-to-canyonos/references/adapter.md index 61d6c07..00e8d37 100644 --- a/.claude/skills/porting-to-canyonos/references/adapter.md +++ b/.claude/skills/porting-to-canyonos/references/adapter.md @@ -1,7 +1,7 @@ # Writing what goes into `.car/app` -Read this before writing any adapter. Every rule here builds green, passes -`ventis build`, and fails inside a container -- which is why the trigger is the +Read this before writing any adapter. Every rule here can pass static build +checks and fail only when a container loads -- which is why the trigger is the step, not a symptom. ## Contents @@ -43,8 +43,8 @@ another. fails at agent load. Make its top-level imports absolute; the modules it imports keep theirs. V035. 5. **Does module-level code perform a real run?** A script ending in - `result = crew.kickoff(...)` / `print(result)` fires that run on every - container start and every probe, before a request exists. Delete the + `result = crew.kickoff(...)` / `print(result)` fires that run whenever the + module loads, before a request exists. Delete the invocation and keep the construction: M18 protects prompts, tools, schemas and model calls, not a script's own main body. diff --git a/.claude/skills/porting-to-canyonos/references/ec2.md b/.claude/skills/porting-to-canyonos/references/ec2.md index 3a80363..486049d 100644 --- a/.claude/skills/porting-to-canyonos/references/ec2.md +++ b/.claude/skills/porting-to-canyonos/references/ec2.md @@ -22,7 +22,7 @@ Typical required categories are: - security groups - SSH user and credentials accepted by the runtime -`ventis deploy` owns basic EC2 config validation. A preflight pass is not proof +`canyonos deploy` owns basic EC2 config validation. A preflight pass is not proof that provisioning, SSH, image transfer, or remote container startup works. ## Networking @@ -35,13 +35,14 @@ The environment file may be copied temporarily to a remote host by runtimes that expose the `env_file` capability. Confirm behavior from the capability probe and target runtime rather than assuming local Docker semantics. -## Probes and cleanup +## Deployment and cleanup -Run the same runtime and adapter probes against the exact image before remote -deployment. After deploy, verify the remote container logs; controller health -can be green even when agent loading failed. +After the user explicitly approves `canyonos deploy`, verify the remote +container logs; controller health can be green even when agent loading failed. +Do not start a separate build or deployment as part of validation. -Stop foreground deploy normally so the controller can terminate recorded EC2 -instances. If provisioning or startup fails before an instance is recorded, -inspect the cloud provider directly and remove exact leaked resources. Never use -a broad cleanup command against unrelated instances. +Ctrl+C stops CLI log monitoring, not necessarily the deployment. Ask before +running `canyonos stop` so the controller can terminate recorded EC2 instances. +If provisioning or startup fails before an instance is recorded, inspect the +cloud provider directly and remove exact leaked resources. Never use a broad +cleanup command against unrelated instances. diff --git a/.claude/skills/porting-to-canyonos/references/example-port.md b/.claude/skills/porting-to-canyonos/references/example-port.md index 56bb4f0..326160a 100644 --- a/.claude/skills/porting-to-canyonos/references/example-port.md +++ b/.claude/skills/porting-to-canyonos/references/example-port.md @@ -1,7 +1,8 @@ # One port, end to end -A LangGraph email assistant, ported and deployed. Read this for the shape of the -decisions; the rules themselves are in SKILL.md. +A LangGraph email assistant, ported and validated, then deployed with explicit +approval. Read this for the shape of the decisions; the rules themselves are in +SKILL.md. ## Contents @@ -99,18 +100,27 @@ relative to `.car/app`. The workflow imports the agent from its entrypoint -- replaces with a stub. The platform sends `{query: string}` only, so the four email fields ride inside -`query` as JSON and the workflow unpacks them. `main` returns a dict, and -`GET /status/` hands it back under `result`. +`query` as JSON and the workflow unpacks them. The adapter returns a dict; the +runtime encodes it once for transport, so the workflow decodes the Future once +and returns an ordinary dict without another `json.dumps`: + +```python +def main(query: str) -> dict: + email = json.loads(query) + triage = json.loads(agent.triage(email_input=email).value()) + if triage["goto"] == "END": + return triage + return json.loads(agent.respond(messages=triage["messages"]).value()) +``` + +`GET /status/` hands that result back under `result`. ## The evidence ```text -validate.py .car 0 errors -ventis build ventis-emailagent, ventis-workflow -docker run ... import local_controller both images -docker run --env-file .env ... EmailAgent() loads -docker run ventis-workflow ... from email_assistant import EmailAgent stub imports -ventis deploy 2 replicas ready +validate.py .car 0 errors; porting workflow stopped +user approval yes, run deployment +canyonos deploy build complete; 2 replicas ready POST /main 202 {"request_id": ...} GET /status/ status: error, 401 from OpenAI ``` diff --git a/.claude/skills/porting-to-canyonos/references/llm-proxy.md b/.claude/skills/porting-to-canyonos/references/llm-proxy.md index e43885c..f660672 100644 --- a/.claude/skills/porting-to-canyonos/references/llm-proxy.md +++ b/.claude/skills/porting-to-canyonos/references/llm-proxy.md @@ -22,8 +22,8 @@ source SDK, model ID, request body, and response parsing unchanged. Each SDK generation reads a different base-URL variable, and a wrapper library reads a different one from the SDK it wraps. Set only the name this reference used to give and the container reaches the real provider with a placeholder key: -a 401 that reads like a broken port, after validate, build and both probes have -passed. Set all of them for whichever providers the source uses: +a 401 that reads like a broken port, after validation and the deployment build +have passed. Set all of them for whichever providers the source uses: ```dotenv OPENAI_BASE_URL=http://host.docker.internal:8081/openai/v1 @@ -79,7 +79,7 @@ curl http://127.0.0.1:8081/healthz Local CanyonOS Core containers resolve `host.docker.internal` through their Docker host mapping. On EC2 that name resolves to each EC2 Docker host, not the -machine running `ventis deploy`. Distributed deployments need a reachable proxy +machine running `canyonos deploy`. Distributed deployments need a reachable proxy address or one proxy on each host. ## Supported call shape diff --git a/.claude/skills/porting-to-canyonos/references/manifest.md b/.claude/skills/porting-to-canyonos/references/manifest.md index a098469..8dc7e07 100644 --- a/.claude/skills/porting-to-canyonos/references/manifest.md +++ b/.claude/skills/porting-to-canyonos/references/manifest.md @@ -1,8 +1,9 @@ # Writing what goes into `.car/config` -Read this before writing the manifest or an agent declaration. `ventis build` -owns yaml syntax; nothing here is syntax. These are the values that build green -and then decide whether a container can import its own dependencies. +Read this before writing the manifest or an agent declaration. The build phase +of `canyonos deploy` owns yaml syntax; nothing here is syntax. These are the +values that can build successfully and then decide whether a container imports +its own dependencies. ## Contents diff --git a/.claude/skills/porting-to-canyonos/references/packaging.md b/.claude/skills/porting-to-canyonos/references/packaging.md index cd8265a..2b7b727 100644 --- a/.claude/skills/porting-to-canyonos/references/packaging.md +++ b/.claude/skills/porting-to-canyonos/references/packaging.md @@ -1,7 +1,8 @@ # Packaging and import roots Read this reference when an adapter imports nested source code, the source uses a -`src/` layout, or V031 reports an import-root problem. +`src/` layout, V031 reports an import-root problem, or the source reads +non-Python files at runtime. ## Contents @@ -10,6 +11,7 @@ Read this reference when an adapter imports nested source code, the source uses - Detect support, do not infer it from release history - Root metadata is the trigger - Dependencies in nested metadata +- Runtime data and configuration files - Validation boundary ## What `/app` can import @@ -105,8 +107,31 @@ config requirements. Report declared-but-unused toolchain dependencies and their image cost; let the owner decide whether source metadata should change. +## Runtime data and configuration files + +`prepare.py` copies non-Python files into `.car/app`, but that does not prove the +runtime's image sweep carries them into a container. Inventory every file opened +by the selected import graph: prompt templates, JSON schemas, PDFs, local +corpora, certificates, and framework configuration such as CrewAI +`agents.yaml` and `tasks.yaml`. + +Run `validate.py` and read its `sweeps_all_files` capability: + +- When available, retain each asset at the same path relative to the chosen + import root. Check any path derived from the original repository root or + process working directory; the container starts from `/app`. +- When unavailable, a required non-Python asset is a runtime blocker. Report it + and stop after validation. Do not conceal the gap by base64-encoding the file + into Python, changing a hardcoded path, or duplicating framework config into + adapter code; those changes restate source-owned data and behavior. + +Do not treat successful construction as evidence that configuration loaded. +Frameworks such as CrewAI may warn about a missing yaml and create an empty +configuration, then fail only when the first agent or task is accessed. Inspect +those decorators and file references statically during the survey. + ## Validation boundary -`ventis build` owns packaging syntax and installation errors. `validate.py` -checks only whether adapter imports appear to require a nested root that the -runtime will not expose. +The build phase of `canyonos deploy` owns packaging syntax and installation +errors. `validate.py` checks only whether adapter imports appear to require a +nested root that the runtime will not expose. diff --git a/.claude/skills/porting-to-canyonos/references/runtime-contract.md b/.claude/skills/porting-to-canyonos/references/runtime-contract.md index eb6ce0b..9d96aff 100644 --- a/.claude/skills/porting-to-canyonos/references/runtime-contract.md +++ b/.claude/skills/porting-to-canyonos/references/runtime-contract.md @@ -1,8 +1,8 @@ # CanyonOS Core runtime contract -The product is CanyonOS Core. Compatibility identifiers remain `ventis` for the -CLI and Python package, `VENTIS_*` for runtime variables, and `ventis-*` for -Docker resources. +The product is CanyonOS Core and its user-facing CLI is `canyonos`. Compatibility +identifiers remain `ventis` for the internal Python package, `VENTIS_*` for +runtime variables, and `ventis-*` for Docker resources. Read this reference when implementing an adapter or explaining a validator finding. Runtime-dependent behavior is expressed as capabilities; run @@ -23,10 +23,10 @@ release history. ## Artifact root and discovery -`ventis build` runs from the application root and reads `.car` below it. That -artifact root holds `config/` beside `app/`, the copy of the application source -that becomes `/app` inside every image. Paths in the config are relative to -`app/` and may not escape it. +The build phase of `canyonos deploy` runs from the application root and reads +`.car` below it. That artifact root holds `config/` beside `app/`, the copy of +the application source that becomes `/app` inside every image. Paths in the +config are relative to `app/` and may not escape it. | Input | Discovery | |---|---| @@ -123,8 +123,8 @@ Consequences: Agent import and construction exceptions are caught by the controller. A failed agent may still advertise healthy because health is written independently of -successful agent loading. That is why image probes import both the runtime and -the entrypoint explicitly. +successful agent loading. After an explicitly approved deployment, inspect +container logs rather than treating health as proof that the entrypoint loaded. ## Workflow execution @@ -145,7 +145,7 @@ and resolving inside one comprehension serializes work without raising an error; dispatch all calls first, then resolve them. The workflow container also starts runtime controller code and has its own -package resolution. Probe it independently from agent images. +package resolution. A failure there can differ from failures in agent images. ## Build context and collisions @@ -191,9 +191,9 @@ produce a green image build that dies on: import local_controller ``` -Always run that probe before probing the entrypoint. Treat a generated-code / -runtime-version mismatch as a CanyonOS Core runtime issue, not a reason to alter -source dependencies silently. +Treat a generated-code/runtime-version mismatch during an explicitly approved +`canyonos deploy` as a CanyonOS Core runtime issue, not a reason to alter source +dependencies silently. ## Credentials capability @@ -208,8 +208,8 @@ source needs credentials, report the capability blocker rather than hardcoding or vendoring a secret. A source that constructs its client at import time works only when credentials -are already in the container environment. Image entrypoint probes therefore use -the same env file as deployment. +are already in the container environment. After an explicitly approved deploy, +check loading failures against the same env file configured for deployment. For proxy-specific credential separation, read [llm-proxy.md](llm-proxy.md). @@ -224,10 +224,12 @@ and remote networking are covered in [ec2.md](ec2.md). ## Cleanup boundary -Stopping foreground deploy normally invokes controller cleanup for recorded -containers and Redis. Hard kills and failures before resource registration may -leave resources behind. +`canyonos deploy` follows controller logs after starting the deployment. Ctrl+C +stops that log stream; use `canyonos stop` to request controller teardown when +the user asks to stop the deployment. Hard kills and failures before resource +registration may leave resources behind. -`ventis clean` removes generated `stubs/`, `grpc_stubs/`, and -`docker_container/` under `.car`. It does not remove containers or images. Remove exact leftovers explicitly and preserve `app/`, -`config/`, and requested evidence. +`canyonos clean` removes generated `stubs/`, `grpc_stubs/`, and +`docker_container/`; it does not remove containers or images. Remove exact +leftovers explicitly and preserve `.car/app`, `.car/config`, and requested +evidence. diff --git a/.claude/skills/porting-to-canyonos/references/troubleshooting.md b/.claude/skills/porting-to-canyonos/references/troubleshooting.md index 3744ec2..ef40ecd 100644 --- a/.claude/skills/porting-to-canyonos/references/troubleshooting.md +++ b/.claude/skills/porting-to-canyonos/references/troubleshooting.md @@ -1,7 +1,8 @@ # Troubleshooting -Read this after a failed build, image probe, deploy, or request. For mechanisms, -read [runtime-contract.md](runtime-contract.md). For proxy or remote-host failures, +Read this after an explicitly approved `canyonos deploy` fails during build, +startup, or a request. For mechanisms, read +[runtime-contract.md](runtime-contract.md). For proxy or remote-host failures, read [llm-proxy.md](llm-proxy.md) or [ec2.md](ec2.md). ## Build or deploy stops early @@ -18,6 +19,7 @@ read [llm-proxy.md](llm-proxy.md) or [ec2.md](ec2.md). | Replica conversion `TypeError` | `replicas` is not an integer | | Policy `AttributeError` | Policy exists but is empty or has null/non-list rules; remove it when unrestricted | | Port or container name already in use | A previous deployment did not complete cleanup | +| Build phase reports `No space left on device` | Docker layers and the package-manager cache exceeded the host budget; inspect `df -h` and `docker system df`, report the pressure, and ask before deleting caches or images | ## Container exits or serves nothing @@ -69,6 +71,7 @@ read [llm-proxy.md](llm-proxy.md) or [ec2.md](ec2.md). | Symptom | Likely cause | |---|---| -| `ventis clean` succeeds but containers remain | The command removes generated directories only | -| `ventis clean` succeeds but images remain | Image deletion is separate and requires exact tags | -| Next deployment collides with old resources | Foreground deploy was killed or crashed before controller cleanup | +| Ctrl+C leaves the deployment running | `canyonos deploy` follows logs; Ctrl+C stops monitoring, not the deployment. Ask before running `canyonos stop` | +| `canyonos clean` succeeds but containers remain | The command removes generated directories only | +| `canyonos clean` succeeds but images remain | Image deletion is separate and requires exact tags | +| Next deployment collides with old resources | The deployment was killed or crashed before controller cleanup | From 0e315939765763fb057142fe0af955b883cefea4 Mon Sep 17 00:00:00 2001 From: Nick Huo Date: Fri, 4 Sep 2026 10:32:46 -0700 Subject: [PATCH 5/7] Keep the porting skill focused on validation --- .claude/skills/porting-to-canyonos/SKILL.md | 10 ++-------- .../porting-to-canyonos/references/troubleshooting.md | 1 - 2 files changed, 2 insertions(+), 9 deletions(-) diff --git a/.claude/skills/porting-to-canyonos/SKILL.md b/.claude/skills/porting-to-canyonos/SKILL.md index 2f8f02a..bdd8890 100644 --- a/.claude/skills/porting-to-canyonos/SKILL.md +++ b/.claude/skills/porting-to-canyonos/SKILL.md @@ -163,10 +163,6 @@ Then survey the copy. Identify: error already present in the source is a source defect, not permission to repair behavior silently. Report it and obtain approval before fixing only the `.car/app` copy. -11. Suspicious module-level behavior before any import or execution: obfuscated - payloads, network downloads, shell/process calls, credential harvesting, or - destructive filesystem operations. Stop and ask the user when found; do not - import, build, or deploy untrusted code merely because it is a port target. Run the validator now, and again after every change until it reports 0 errors. Execute it; do not read it. Its header detects capabilities directly from the @@ -375,7 +371,5 @@ If the user approves, run from the application root: canyonos deploy ``` -Report build or deployment failures without silently changing source behavior, -dependencies, provider, or deployment settings. `canyonos deploy` follows the -controller logs; Ctrl+C stops log monitoring, not necessarily the deployment. -Use `canyonos stop` only when the user asks to stop it. +Do not add build, probe, deployment-debugging, or cleanup work to this skill's +porting flow. diff --git a/.claude/skills/porting-to-canyonos/references/troubleshooting.md b/.claude/skills/porting-to-canyonos/references/troubleshooting.md index ef40ecd..213163c 100644 --- a/.claude/skills/porting-to-canyonos/references/troubleshooting.md +++ b/.claude/skills/porting-to-canyonos/references/troubleshooting.md @@ -19,7 +19,6 @@ read [llm-proxy.md](llm-proxy.md) or [ec2.md](ec2.md). | Replica conversion `TypeError` | `replicas` is not an integer | | Policy `AttributeError` | Policy exists but is empty or has null/non-list rules; remove it when unrestricted | | Port or container name already in use | A previous deployment did not complete cleanup | -| Build phase reports `No space left on device` | Docker layers and the package-manager cache exceeded the host budget; inspect `df -h` and `docker system df`, report the pressure, and ask before deleting caches or images | ## Container exits or serves nothing From eb1fa105b31be4330a48ed4421af03ce8517c663 Mon Sep 17 00:00:00 2001 From: Nick Huo Date: Fri, 4 Sep 2026 13:02:07 -0700 Subject: [PATCH 6/7] Push the survey and the refresh flow out of SKILL.md, and split the validator SKILL.md was carrying three things that only matter once: the step-1 survey list, the rules for refreshing an existing `.car/app`, and adapter prose that adapter.md already gates. All three move out, and SKILL.md keeps the decision and the pointer. - references/source-survey.md: the survey, read after preparing the copy and before choosing service boundaries. - references/refresh.md: read when `.car/app` exists and the source moved on. prepare.py grows `--refresh`, which updates source-owned files while preserving port edits and stops atomically when both sides changed one path -- the case that silently loses an adapter. It now also rejects symbolic links, which either escape the self-contained artifact or are skipped by the runtime sweep, so neither outcome is worth carrying. validate.py splits into a validation package rather than growing further: core.py result and YAML primitives manifest.py the public artifact contract, fail-closed python_source.py static source discovery for adapter and packaging checks runtime.py runtime capabilities and dependency facts references/manifest.md gains the `canyonos config` View/Change review flow and re-scopes its opening: the validator checks YAML structure and the artifact contract before an approved deploy, so this reference explains how to derive the values rather than what the build will reject. (cherry picked from commit d2c82d6bfab4e0ffcb1d12b64b2470eb529be981) --- .claude/skills/porting-to-canyonos/SKILL.md | 209 ++----- .claude/skills/porting-to-canyonos/prepare.py | 239 +++++++- .../references/example-port.md | 2 +- .../references/manifest.md | 35 +- .../references/packaging.md | 2 +- .../porting-to-canyonos/references/refresh.md | 38 ++ .../references/source-survey.md | 29 + .../skills/porting-to-canyonos/validate.py | 535 ++---------------- .../validation/__init__.py | 2 + .../porting-to-canyonos/validation/core.py | 99 ++++ .../validation/manifest.py | 295 ++++++++++ .../validation/python_source.py | 152 +++++ .../porting-to-canyonos/validation/runtime.py | 123 ++++ 13 files changed, 1095 insertions(+), 665 deletions(-) create mode 100644 .claude/skills/porting-to-canyonos/references/refresh.md create mode 100644 .claude/skills/porting-to-canyonos/references/source-survey.md create mode 100644 .claude/skills/porting-to-canyonos/validation/__init__.py create mode 100644 .claude/skills/porting-to-canyonos/validation/core.py create mode 100644 .claude/skills/porting-to-canyonos/validation/manifest.py create mode 100644 .claude/skills/porting-to-canyonos/validation/python_source.py create mode 100644 .claude/skills/porting-to-canyonos/validation/runtime.py diff --git a/.claude/skills/porting-to-canyonos/SKILL.md b/.claude/skills/porting-to-canyonos/SKILL.md index bdd8890..1dcfb8f 100644 --- a/.claude/skills/porting-to-canyonos/SKILL.md +++ b/.claude/skills/porting-to-canyonos/SKILL.md @@ -25,10 +25,10 @@ Port progress: - [ ] 2. Survey the copy and choose service boundaries - [ ] 3. Read adapter.md + manifest.md; write declarations, adapters, workflow; use the `canyonos config` flow to review deployment choices, then write config -- [ ] 4. validate.py reports 0 errors; report readiness and stop +- [ ] 4. validate.py exits 0; report readiness and stop ``` -Do not skip step 4. The porting workflow ends when validation reports 0 errors: +Do not skip step 4. The porting workflow ends when validation exits 0: report the files created, warnings and unresolved runtime blockers, then stop. Never build or deploy as an implicit continuation of the port. @@ -53,6 +53,10 @@ deployment, all three knowable at step 1. - [references/packaging.md](references/packaging.md) -- read when a source import does not resolve from `/app`, the source is nested, packaging metadata is involved, or the source reads non-Python files at runtime. +- [references/source-survey.md](references/source-survey.md) -- read after + preparing the copy and before choosing service boundaries. +- [references/refresh.md](references/refresh.md) -- read when `.car/app` already + exists and the source has changed; preserve port edits while refreshing it. - [references/llm-proxy.md](references/llm-proxy.md) -- read when the target includes `llm_proxy`. - [references/ec2.md](references/ec2.md) -- read when any config entry uses @@ -122,59 +126,29 @@ Once the import root is known, use the skill's preparation script rather than assembling `.car` with ad hoc copy commands: ```bash -python /prepare.py .car +python3 /prepare.py .car ``` The script creates `.car/config/` and copies the import root's **contents** into `.car/app/`, preserving its structure. It excludes version-control data, `.car`, virtualenvs, caches, build outputs, bytecode, and credential-bearing `.env*` files while retaining `.env.example`, `.env.sample`, and -`.env.template`. It refuses to merge into an existing `.car/app`; use `--force` -only when intentionally replacing the entire source copy. `--force` leaves an -existing `.car/config/` unchanged. +`.env.template`. It rejects symbolic links because they either escape the +self-contained artifact or are skipped by the runtime source sweep. + +If `.car/app` already exists, read [references/refresh.md](references/refresh.md) +and use `--refresh`. It updates source-owned files while preserving port edits, +and stops atomically when both sides changed one path. Use `--force` only to +discard every edit in `.car/app`; it leaves `.car/config/` unchanged. Choosing the import root remains a porter decision; the script standardizes only directory creation and copying. After it runs, every edit is inside `.car`. The application source outside it is read-only for the rest of the port -- `git status` at the end shows `.car/` and nothing else. -Then survey the copy. Identify: - -1. The source entry point and callable input/output. When the repository has - several plausible implementations, trace the imports from the production - route, CLI, or documented launch path; do not choose the most convenient - graph by filename. -2. Framework-owned control flow (`StateGraph`, `Crew`, `GroupChat`, routing, - `Send`, `Command`, interrupts). -3. Runtime-injected services nodes read: stores, context, memory, sessions, or - callback managers. -4. Sync versus async boundaries. -5. Imports and declared runtime dependencies. -6. Model provider, credential names, streaming use, and optional `llm_proxy`. -7. Whether independent work fans out and benefits from separate replicas. -8. Whether source imports resolve from `.car/app`, the root that becomes - `/app`. This is the check the validator turns into V031. -9. Every non-Python file reached at runtime: prompts, CrewAI - `agents.yaml`/`tasks.yaml`, PDFs, templates, schemas, and local corpora. If - `sweeps_all_files` is unavailable, these are runtime blockers even though - `prepare.py` copied them. Do not invent base64 embedding or rewrite hardcoded - paths; report and stop. -10. Whether every Python file on the selected import graph parses. A syntax - error already present in the source is a source defect, not permission to - repair behavior silently. Report it and obtain approval before fixing only - the `.car/app` copy. - -Run the validator now, and again after every change until it reports 0 errors. -Execute it; do not read it. Its header detects capabilities directly from the -importable runtime rather than from release history: - -```bash -python /validate.py .car -``` - -If config or agent yaml is malformed, the validator defers that failure to the -build phase inside a later `canyonos deploy`. Capability-gated findings say -which runtime behavior is available. +Read [references/source-survey.md](references/source-survey.md), survey the +copy, and run the validator. The survey determines the source facts used in the +next two steps; do not infer them from framework conventions. ## 2. Choose service boundaries @@ -214,143 +188,52 @@ For each service, keep these names aligned: config entry name == yaml agent.name == entrypoint class name ``` -### Adapter - -Write the adapter where the code it wraps already lives. *Which* module -`entrypoint` then names is the decision adapter.md gates: the build writes that -agent's stub over that path in every other image, so it is the one module in -the copy the port destroys. - -The entrypoint exposes a module-level class named exactly `agent.name`. It -constructs with no arguments and its declared methods are synchronous. Read -configuration from the environment in `__init__`. Serialize framework objects -with their own JSON-safe serializer before returning. - -Do not duplicate source prompts, tools, schemas, or model calls. Keep the source -provider and SDK. +Write a no-argument, synchronous adapter class at the entrypoint selected by +`adapter.md`. Import source-owned behavior instead of duplicating it. Expose +`main(query: str)` in the workflow, import every service from its exact +entrypoint module, and call `deploy(main, port=...)` at module scope. -### Workflow - -Expose `main(query: str)` and call `deploy(main, port=...)` at module scope. -Import each agent from its own `entrypoint`, exactly where the source copy keeps -it -- that is the one module the build replaces with a stub: +For parallel remote calls, dispatch before resolving: ```python -from deploy import deploy -from . import # the agent's entrypoint path +futures = [agent.work(item=item) for item in items] +results = [json.loads(future.value()) for future in futures] ``` -Any other route to the class -- a flat name, a package re-export, a second copy -of the module -- reaches the real class and runs the agent in the workflow -process with none of the deployment behind it. That import needs no rewriting -when the source already imported the agent from there. +Do not fuse dispatch and `.value()` in one comprehension. Do not add a main +guard; the workflow executes as `__main__` in production. -The deployment platform sends `{query: string}` to `/main`. Pack richer input -inside `query`; any additional workflow parameter has a default. +Build declarations and per-image requirements from the copied import graph. +Then use the View/Change flow in `manifest.md` (and `canyonos config` when +interactive) to review developer-owned deployment choices. Write only the +reviewed candidate and rerun validation. -Dispatch every remote call before resolving any future: +## Source-integrity boundary -```python -futures = [agent.work(item=item) for item in items] -results = [json.loads(future.value()) for future in futures] -``` +The validator owns mechanical runtime rules; do not duplicate its check list in +the prompt. The porter owns the rules static analysis cannot prove: + +- Never edit outside `.car` or copy source-owned prompts, tools, schemas, model + calls, and node bodies into an adapter. +- Never swap the source provider, invent runtime configuration, or silently + move, drop, or reclassify a dependency. +- Rewrite framework control flow only where it crosses a service boundary; + preserve it inside a service. +- Never hardcode or bake a real credential into `.car`. -Do not fuse dispatch and `.value()` in one comprehension; that silently -serializes fan-out. Do not add an `if __name__ == "__main__":` block: the -workflow is executed with `__name__ == "__main__"` in production. - -### Config - -`entrypoint` and `workflow_file` are relative to `.car/app/` and may not escape -it. `provider` is lowercase `local`, `replicas` is an integer, and -`requirements` is a per-entry list of distribution names -- the source's own -`requirements.txt` is never installed into any image. Omit `policy.yaml` unless -access must be restricted; if present, give it a non-empty `rules` list. -manifest.md carries the complete manifest and how to build each `requirements` -list. - -Part of the manifest is derived from the copy; the rest is the developer's -deployment choice, and no reading of the source produces it. Derive what the -source decides. Do not derive what it does not. - -Use the interaction exposed by `cli/canyonos/config.py` as the configuration UX. -Before writing `.car/config/global_controller.yaml`, present the same two -choices -- **View** and **Change** -- rather than silently choosing deployment -settings: - -1. Build the complete candidate manifest in memory from derived values plus the - defaults in manifest.md's ownership table. -2. **View** prints the whole candidate, not a summary. Clearly annotate defaults - and values constrained by the source, such as `replicas: 1` for stateful - in-memory services. -3. **Change** asks, in one batch, only for developer-owned values: provider and - its EC2 block, replicas that are not constrained, resources, ports, secret - file location, and whether access needs restricting. Show the current/default - value for every choice. Apply the answers and show the resulting manifest. -4. Write the reviewed candidate, then run the validator. - -If the coding environment can launch an interactive command, prefer running -`canyonos config` for this review. Otherwise reproduce its View/Change flow in -the conversation; do not skip the review merely because the CLI has no TTY. -Do not ask the developer for derived values such as `entrypoint` or -`requirements`: walking the copied source gives a more reliable answer. - -Never block an unattended port on this interaction. `canyonos integrate` -launches this skill from a prompt, and a corpus run has no one at the terminal. -If no answer is available, write the displayed defaults, name them in the final -report, and keep going. The one exception is a value with no safe default: -`provider: EC2` needs infrastructure identifiers that are wrong to invent, so -an unanswered EC2 choice leaves the entry `local` and says so. - -## Hard rules - -Capitalized **MUST** and **NEVER** are reserved for port-breaking or -source-integrity rules. The owner column states where each is decided. - -| ID | Rule | Owner | -|---|---|---| -| M1 | Entrypoint MUST define a class named exactly `agent.name` | V006 | -| M2 | The class MUST construct with no arguments | V007 | -| M3 | yaml argument names MUST match Python parameter names | V008 | -| M4 | yaml argument types MUST be bare builtins | V010 | -| M5 | Declared adapter methods MUST be synchronous | V009 | -| M6 | Config names MUST match yaml agent names | `canyonos deploy` build phase | -| M7 | Config names MUST not collide after lowercase normalization | `canyonos deploy` build output | -| M8 | Local provider MUST be lowercase `local` | `canyonos deploy` preflight | -| M9 | `replicas` MUST be an integer | `canyonos deploy` preflight | -| M10 | `requirements` MUST be a list of strings | `canyonos deploy` build phase | -| M11 | Workflow MUST expose `main(query)`; extra parameters MUST default | V016 | -| M12 | Workflow MUST NEVER contain a main guard | V017 | -| M13 | Fan-out MUST dispatch all calls before resolving any | V018 | -| M14 | A module at the root of the copy MUST not take a runtime flat name | V019 | -| M15 | Workflow MUST import each agent from its own `entrypoint` module | V023 | -| M16 | Policy MUST be absent or contain a non-empty `rules` list | `canyonos deploy` preflight | -| M17 | EC2 entries MUST satisfy the EC2 deployment contract | `canyonos deploy` preflight | -| M18 | NEVER copy source prompts, tools, schemas, or model calls | review | -| M19 | NEVER hardcode or bake a real credential into an image | W003 | -| M20 | NEVER write outside `.car`; the application source stays untouched | `git status` | -| M21 | NEVER swap the source LLM provider | review | -| M22 | NEVER silently move, drop, or reclassify source dependencies | review | -| M23 | Framework control flow MUST be rewritten; source node logic MUST be imported | review | -| M24 | A non-resolving source import MUST have usable packaging metadata at the root of the copy when editable install is supported | V031 | -| M25 | Two agents MUST NOT share one `entrypoint` | V020 | -| M26 | `.car` MUST hold `config/` beside `app/`, the source copy | V032 | -| M27 | `.car/app` MUST be rooted at the source's import root | V031 | -| M28 | `requirements` MUST cover every distribution that entry's import graph reaches, transitively | W006, dependency review | -| M29 | The entrypoint MUST NOT be a module another image imports for its real contents | adapter.md review | -| M30 | The entrypoint's package `__init__.py` MUST NOT re-export from it | V033 | -| M31 | Every segment of the `entrypoint` path MUST be a Python identifier | V034 | -| M32 | The entrypoint's own imports MUST be absolute | V035 | +When a source defect or unsupported runtime capability requires breaking one of +these boundaries, report the blocker and obtain approval for that specific +change. Do not broaden approval to unrelated source edits. ## 4. Validate and stop Run static preflight from the application root: ```bash -python /validate.py .car +python3 /validate.py .car ``` -Fix every ERROR and re-run until it reports 0 errors. Warnings and capability +Fix every ERROR and re-run until it exits 0. Warnings and capability limitations are not permission to hide risk: list each one in the handoff and say whether it blocks this source. Confirm that `git status` outside `.car` shows no change to a file the developer owns. diff --git a/.claude/skills/porting-to-canyonos/prepare.py b/.claude/skills/porting-to-canyonos/prepare.py index 9f282b7..963b0c5 100755 --- a/.claude/skills/porting-to-canyonos/prepare.py +++ b/.claude/skills/porting-to-canyonos/prepare.py @@ -8,10 +8,13 @@ """ import argparse +import hashlib +import json +import os import shutil import sys import uuid -from pathlib import Path +from pathlib import Path, PurePosixPath EXCLUDED_DIRECTORIES = frozenset( { @@ -35,6 +38,7 @@ ) EXCLUDED_FILE_SUFFIXES = (".pyc", ".pyo") ENV_TEMPLATES = frozenset({".env.example", ".env.sample", ".env.template"}) +STATE_FILENAME = ".porting-state.json" def _is_relative_to(path: Path, parent: Path) -> bool: @@ -45,6 +49,17 @@ def _is_relative_to(path: Path, parent: Path) -> bool: return True +def _reject_symlinks(import_root: Path) -> None: + """Keep the copied artifact independent of paths outside the copy.""" + for path in import_root.rglob("*"): + if path.is_symlink(): + relative = path.relative_to(import_root) + raise ValueError( + f"import root contains a symbolic link: {relative}; replace it " + "with the intended file or directory before preparing the port" + ) + + def _ignore_factory(artifact_root: Path): def ignore(directory: str, names: list[str]) -> set[str]: directory_path = Path(directory) @@ -64,7 +79,168 @@ def ignore(directory: str, names: list[str]) -> set[str]: return ignore -def prepare(import_root: Path, artifact_root: Path, force: bool = False) -> None: +def _copy_source(import_root: Path, destination: Path, artifact_root: Path) -> None: + shutil.copytree( + import_root, + destination, + ignore=_ignore_factory(artifact_root), + copy_function=shutil.copy2, + symlinks=True, + ) + + +def _hash_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def _file_hashes(root: Path) -> dict[str, str]: + return { + path.relative_to(root).as_posix(): _hash_file(path) + for path in sorted(root.rglob("*")) + if path.is_file() and not path.is_symlink() + } + + +def _load_state(config_dir: Path) -> dict[str, str]: + state_path = config_dir / STATE_FILENAME + if not state_path.is_file(): + raise ValueError( + f"{state_path} is missing; this artifact predates refresh tracking. " + "Use --force only if discarding all edits in .car/app is intentional" + ) + try: + state = json.loads(state_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as error: + raise ValueError(f"cannot read refresh state {state_path}: {error}") from error + files = state.get("source_files") if isinstance(state, dict) else None + if ( + not isinstance(state, dict) + or state.get("version") != 1 + or not isinstance(files, dict) + or not all(_valid_state_entry(path, digest) for path, digest in files.items()) + ): + raise ValueError(f"invalid refresh state: {state_path}") + return files + + +def _valid_state_entry(path: object, digest: object) -> bool: + if not isinstance(path, str) or not isinstance(digest, str): + return False + relative = PurePosixPath(path) + return ( + path == relative.as_posix() + and not relative.is_absolute() + and path not in ("", ".") + and ".." not in relative.parts + and len(digest) == 64 + and all(character in "0123456789abcdef" for character in digest) + ) + + +def _write_state(path: Path, source_hashes: dict[str, str]) -> None: + path.write_text( + json.dumps( + {"version": 1, "source_files": source_hashes}, + indent=2, + sort_keys=True, + ) + + "\n", + encoding="utf-8", + ) + + +def _refresh_app( + app_dir: Path, + source_copy: Path, + staged_app: Path, + previous_source: dict[str, str], +) -> None: + current = _file_hashes(app_dir) + incoming = _file_hashes(source_copy) + conflicts = [] + + for relative in sorted(previous_source.keys() | incoming.keys() | current.keys()): + old = previous_source.get(relative) + new = incoming.get(relative) + edited = current.get(relative) + source_changed = new != old + app_changed = edited != old + if old is None and new is not None and edited not in (None, new): + conflicts.append(relative) + elif old is not None and source_changed and app_changed and edited != new: + conflicts.append(relative) + + # File/directory replacements need the same three-way protection. A new + # source file at `pkg` must not erase a port-only `pkg/adapter.py`, and a + # new source directory must not silently replace a port-authored file at + # `pkg`. + for incoming_path in incoming: + prefix = incoming_path + "/" + for current_path, edited in current.items(): + if not current_path.startswith(prefix): + continue + if edited != previous_source.get(current_path): + conflicts.append(current_path) + for current_path, edited in current.items(): + prefix = current_path + "/" + if any(incoming_path.startswith(prefix) for incoming_path in incoming): + if edited != previous_source.get(current_path): + conflicts.append(current_path) + + if conflicts: + conflicts = sorted(set(conflicts)) + shown = "\n ".join(conflicts[:20]) + suffix = "" if len(conflicts) <= 20 else f"\n ... and {len(conflicts) - 20} more" + raise ValueError( + "refresh found files changed in both the source and .car/app:\n " + f"{shown}{suffix}\nResolve them in .car/app, then update the source " + "or use --force only to discard all port edits" + ) + + shutil.copytree(app_dir, staged_app, copy_function=shutil.copy2, symlinks=True) + + # Apply safe source deletions first so file-to-directory changes have room. + for relative, old in previous_source.items(): + if relative in incoming or current.get(relative) != old: + continue + target = staged_app / relative + if target.is_file(): + target.unlink() + + for directory in sorted( + (path for path in staged_app.rglob("*") if path.is_dir()), + key=lambda path: len(path.parts), + reverse=True, + ): + try: + directory.rmdir() + except OSError: + pass + + for relative, new in incoming.items(): + old = previous_source.get(relative) + edited = current.get(relative) + if new == old or edited == new: + continue + if old is not None and edited != old: + continue + target = staged_app / relative + if target.is_dir(): + shutil.rmtree(target) + target.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(source_copy / relative, target) + + +def prepare( + import_root: Path, + artifact_root: Path, + force: bool = False, + refresh: bool = False, +) -> None: import_root = import_root.expanduser().resolve() artifact_root = artifact_root.expanduser().resolve() app_dir = artifact_root / "app" @@ -78,34 +254,54 @@ def prepare(import_root: Path, artifact_root: Path, force: bool = False) -> None raise ValueError("import root cannot be inside the artifact root") if app_dir.exists() and not app_dir.is_dir(): raise ValueError(f"app path exists but is not a directory: {app_dir}") - if app_dir.exists() and not force: + if force and refresh: + raise ValueError("--force and --refresh are mutually exclusive") + if refresh and not app_dir.is_dir(): + raise ValueError(f"cannot refresh because {app_dir} does not exist") + if app_dir.exists() and not force and not refresh: raise FileExistsError( - f"{app_dir} already exists; use --force to replace only the source copy" + f"{app_dir} already exists; use --refresh to preserve port edits, or " + "--force to discard and replace the entire source copy" ) if config_dir.exists() and not config_dir.is_dir(): raise ValueError(f"config path exists but is not a directory: {config_dir}") + _reject_symlinks(import_root) + if refresh: + _reject_symlinks(app_dir) + artifact_root.mkdir(parents=True, exist_ok=True) - temporary_app = artifact_root / f".app-{uuid.uuid4().hex}.tmp" + config_dir.mkdir(exist_ok=True) + transaction_id = uuid.uuid4().hex + source_copy = artifact_root / f".source-{transaction_id}.tmp" + temporary_app = artifact_root / f".app-{transaction_id}.tmp" previous_app = artifact_root / f".app-{uuid.uuid4().hex}.previous" + temporary_state = config_dir / f".{STATE_FILENAME}-{transaction_id}.tmp" + state_path = config_dir / STATE_FILENAME installed_new_app = False try: - shutil.copytree( - import_root, - temporary_app, - ignore=_ignore_factory(artifact_root), - copy_function=shutil.copy2, - symlinks=True, - ) + _copy_source(import_root, source_copy, artifact_root) + source_hashes = _file_hashes(source_copy) + if refresh: + previous_source = _load_state(config_dir) + _refresh_app(app_dir, source_copy, temporary_app, previous_source) + shutil.rmtree(source_copy) + else: + source_copy.rename(temporary_app) + _write_state(temporary_state, source_hashes) if app_dir.exists(): app_dir.rename(previous_app) temporary_app.rename(app_dir) installed_new_app = True - config_dir.mkdir(exist_ok=True) + os.replace(temporary_state, state_path) except Exception: + if source_copy.exists(): + shutil.rmtree(source_copy) if temporary_app.exists(): shutil.rmtree(temporary_app) + if temporary_state.exists(): + temporary_state.unlink() if installed_new_app and app_dir.exists(): shutil.rmtree(app_dir) if previous_app.exists(): @@ -133,7 +329,12 @@ def parse_args(argv=None): parser.add_argument( "--force", action="store_true", - help="replace an existing app/ copy; leave config/ unchanged", + help="discard edits and replace an existing app/ copy; leave config/ unchanged", + ) + parser.add_argument( + "--refresh", + action="store_true", + help="update unchanged source files while preserving port-authored edits", ) return parser.parse_args(argv) @@ -141,14 +342,20 @@ def parse_args(argv=None): def main(argv=None) -> int: args = parse_args(argv) try: - prepare(Path(args.import_root), Path(args.artifact_root), args.force) + prepare( + Path(args.import_root), + Path(args.artifact_root), + force=args.force, + refresh=args.refresh, + ) except (OSError, ValueError) as error: sys.stderr.write(f"prepare.py: {error}\n") return 1 artifact_root = Path(args.artifact_root) print(f"Created {artifact_root / 'config'}") - print(f"Copied {Path(args.import_root)} to {artifact_root / 'app'}") + action = "Refreshed" if args.refresh else "Copied" + print(f"{action} {Path(args.import_root)} to {artifact_root / 'app'}") return 0 diff --git a/.claude/skills/porting-to-canyonos/references/example-port.md b/.claude/skills/porting-to-canyonos/references/example-port.md index 326160a..4da2d00 100644 --- a/.claude/skills/porting-to-canyonos/references/example-port.md +++ b/.claude/skills/porting-to-canyonos/references/example-port.md @@ -34,7 +34,7 @@ until the model calls `Done`. is `src/`, not the repository root: ```bash -python /prepare.py src .car +python3 /prepare.py src .car ``` This creates `.car/config/` and copies the contents of `src/` into `.car/app/` diff --git a/.claude/skills/porting-to-canyonos/references/manifest.md b/.claude/skills/porting-to-canyonos/references/manifest.md index 8dc7e07..1a6a9df 100644 --- a/.claude/skills/porting-to-canyonos/references/manifest.md +++ b/.claude/skills/porting-to-canyonos/references/manifest.md @@ -1,13 +1,13 @@ # Writing what goes into `.car/config` -Read this before writing the manifest or an agent declaration. The build phase -of `canyonos deploy` owns yaml syntax; nothing here is syntax. These are the -values that can build successfully and then decide whether a container imports -its own dependencies. +Read this before writing the manifest or an agent declaration. The validator +checks YAML structure and the public artifact contract before an approved +`canyonos deploy`; this reference explains how to derive the values inside it. ## Contents - Who decides each key +- Review configuration through the CanyonOS CLI flow - Agent yaml - Requirements - The manifest, in full @@ -19,8 +19,8 @@ and the copy holds it; asking the developer can only make it worse. A **developer** key is a deployment choice the source does not contain, and deriving it means guessing and presenting the guess as a reading. -SKILL.md step 3 shows the whole manifest and then asks about the second column -only, in one round, carrying these defaults. +The configuration review shows the whole manifest and asks about the second +column only, in one round, carrying these defaults. | Key | Decided by | Default when unanswered | |---|---|---| @@ -49,6 +49,29 @@ the config rather than asking about it: deploy preflight or, worse, provisions something unreachable. Unanswered means the entry stays `local`. +## Review configuration through the CanyonOS CLI flow + +Use the interaction implemented by `canyonos config` before writing +`.car/config/global_controller.yaml`: + +1. Build the complete candidate manifest in memory from derived values and the + defaults above. +2. **View** prints the whole candidate, annotating defaults and source-imposed + constraints such as `replicas: 1` for in-memory state. +3. **Change** asks in one batch only for developer-owned values: provider and + EC2 fields, unconstrained replicas, resources, ports, secret-file location, + and access restrictions. Show each current/default value, apply answers, and + show the result. +4. Write the reviewed candidate and run the validator. + +Prefer running `canyonos config` when an interactive terminal is available; +otherwise reproduce View/Change in conversation. Do not ask for derived values +such as entrypoints or requirements. + +An unattended `canyonos integrate` run must not block on this interaction. Use +and report the displayed defaults. Never invent EC2 infrastructure identifiers: +without them, keep the entry `local`. + ## Agent yaml Declarations go in `.car/config/`, beside the manifest. The build reads every diff --git a/.claude/skills/porting-to-canyonos/references/packaging.md b/.claude/skills/porting-to-canyonos/references/packaging.md index 2b7b727..e87c002 100644 --- a/.claude/skills/porting-to-canyonos/references/packaging.md +++ b/.claude/skills/porting-to-canyonos/references/packaging.md @@ -52,7 +52,7 @@ Reach for the metadata below only when one copy root cannot serve every import Run: ```bash -python /validate.py .car +python3 /validate.py .car ``` Read the `editable_install` capability. If it is unavailable and the original diff --git a/.claude/skills/porting-to-canyonos/references/refresh.md b/.claude/skills/porting-to-canyonos/references/refresh.md new file mode 100644 index 0000000..eaab51b --- /dev/null +++ b/.claude/skills/porting-to-canyonos/references/refresh.md @@ -0,0 +1,38 @@ +# Refreshing an existing port + +Read this when `.car/app` already exists and the application source has changed. + +Run the same preparation command with `--refresh` and the same import root: + +```bash +python3 /prepare.py .car --refresh +``` + +The initial copy records source-file hashes in +`.car/config/.porting-state.json`. Refresh compares three states: + +```text +previous source hash → current source + ↘ current .car/app +``` + +- Only the source changed: update `.car/app`. +- Only `.car/app` changed: preserve the port edit. +- The source added a path unused by the port: add it. +- The source deleted an unmodified path: delete it from `.car/app`. +- Both sides changed the same path differently: make no changes and report all + conflicts. + +Resolve a conflict in `.car/app`, then either make the source match that result +or intentionally start over. The script does not guess a merge because an +adapter and its source often change for different reasons while sharing one +module. + +`--force` is not refresh. It discards the entire `.car/app` tree and replaces it +with a clean source copy while retaining `.car/config`. Use it only when every +adapter and workflow edit in `.car/app` is intentionally disposable. + +After refresh, survey changed imports, dependencies, runtime assets, and service +boundaries again, then run the full validator. A clean file merge is not proof +that the deployment contract still holds. + diff --git a/.claude/skills/porting-to-canyonos/references/source-survey.md b/.claude/skills/porting-to-canyonos/references/source-survey.md new file mode 100644 index 0000000..b44538a --- /dev/null +++ b/.claude/skills/porting-to-canyonos/references/source-survey.md @@ -0,0 +1,29 @@ +# Surveying the copied source + +Read this after `prepare.py` and before choosing service boundaries. Survey +`.car/app`, not a guessed abstraction of the original repository. + +Identify all of the following: + +1. The production entry point and callable input/output. If several + implementations look plausible, trace imports from the documented route, + CLI, or launch path instead of choosing by filename. +2. Framework-owned control flow: graphs, crews, chats, routing, fan-out, + commands, and interrupts. +3. Runtime-injected stores, context, memory, sessions, and callback managers. +4. Sync/async boundaries and objects tied to an event loop. +5. The transitive import graph and the source's pinned runtime distributions. +6. Model provider, credential names, streaming, and optional `llm_proxy` use. +7. Independent work that benefits from separate resource or replica profiles. +8. Whether imports resolve with `.car/app` as `/app`; read `packaging.md` when + they do not. +9. Non-Python runtime files such as prompts, CrewAI YAML, PDFs, templates, + schemas, and corpora. If the runtime cannot sweep all files, report this as a + blocker; do not embed files or rewrite paths to hide it. +10. Whether every Python file on the selected import graph parses. Existing + syntax errors are source defects; report them and obtain approval before + changing even the copied version. + +Run `python3 /validate.py .car` after the survey and after every +change. Missing or malformed required inputs fail closed. If a required runtime +capability is reported unavailable, stop instead of assuming it exists. diff --git a/.claude/skills/porting-to-canyonos/validate.py b/.claude/skills/porting-to-canyonos/validate.py index a8f38c9..0eb9e93 100755 --- a/.claude/skills/porting-to-canyonos/validate.py +++ b/.claude/skills/porting-to-canyonos/validate.py @@ -1,14 +1,13 @@ #!/usr/bin/env python3 -"""Preflight the runtime traps that `ventis build` cannot see. +"""Preflight a CanyonOS port before an approved deployment. -This deliberately does not duplicate build-time validation such as malformed -YAML, missing entrypoints, or config-to-yaml matching. `ventis build` owns those -checks. This script parses Python without importing it and catches failures that -otherwise stay hidden until a container loads an agent, starts a workflow, or -serves its first request. A replica is not evidence: the controller writes -`healthy` to Redis before `_load_agent` runs. +This checks the public `.car` artifact contract first, then parses Python without +importing it to catch failures that would otherwise stay hidden until a +container loads an agent, starts a workflow, or serves its first request. It +fails closed when the required inputs cannot be checked. A replica is not +evidence: the controller writes `healthy` to Redis before `_load_agent` runs. - python validate.py [artifact_root] [-c config/global_controller.yaml] + python3 validate.py [artifact_root] [-c config/global_controller.yaml] [--json] [--strict] `artifact_root` is the `.car` directory: `config/` beside `app/`, the copy of @@ -28,130 +27,43 @@ import os import re import sys -from typing import ClassVar - -try: - import yaml -except ImportError: # pragma: no cover - pyyaml is a CanyonOS Core dependency - sys.stderr.write("validate.py needs pyyaml: pip install pyyaml\n") - raise SystemExit(2) from None +SKILL_DIR = os.path.dirname(os.path.abspath(__file__)) +if SKILL_DIR not in sys.path: + sys.path.insert(0, SKILL_DIR) + +from validation.core import ERROR, INFO, WARN, Report, line_of, load_yaml # noqa: E402 +from validation.manifest import ( # noqa: E402 + check_declaration_bindings, + check_manifest_structure, + check_policy, + check_self_contained_tree, + discover_agent_declarations, +) +from validation.python_source import ( # noqa: E402 + class_methods, + find_class, + parameter_names, + parse_python, + reachable_imports, + required_parameters, + toplevel_import_names, +) +from validation.runtime import ( # noqa: E402 + BASE_AGENT_REQUIREMENTS, + BASE_WORKFLOW_REQUIREMENTS, + CAPABILITY_SOURCE, + IMPORT_TO_DISTRIBUTION, + NAMESPACE_DISTRIBUTIONS, + RUNTIME_FLAT_NAMES, + STDLIB_MODULE_NAMES, + probe_capabilities, +) DEFAULT_CONFIG_PATH = "config/global_controller.yaml" # ventis/cli.py SOURCE_DIR_NAME -- the duplicated application source. SOURCE_DIR_NAME = "app" -# Copied flat into every image over the swept project tree, so a project module -# landing flat under one of these names is overwritten. -# ventis/stub_generator.py generate_docker / generate_workflow_docker. -RUNTIME_FLAT_NAMES = frozenset( - { - "future.py", - "ventis_context.py", - "local_controller.py", - "local_controller_frontend.py", - "redis_client.py", - "grpc_options.py", - "gpu_metrics.py", - "bedrock.py", - "deploy.py", - "session_logging.py", - "workflow_launcher.py", - } -) - -def _base_requirements(): - """What the generator preinstalls, taken from the importable runtime. - - The literals below are a fallback for a machine where `ventis` is not - importable. They are also the only copy of a runtime fact in this file that - nothing checks at run time, and a copied fact rots: the `sweeps_all_files` - probe named a function that never existed and reported `no` for an entire - 45-repository corpus before a porter caught it. Prefer the live values, and - let tests/test_porting_skill_validate.py hold the fallback to them. - """ - agent = [ - "grpcio", - "grpcio-tools", - "redis", - "pyyaml", - "psutil", - "ipdb", - "ipython", - "boto3", - ] - workflow = [*agent, "flask", "sqlalchemy", "psycopg[binary]"] - try: - from ventis import stub_generator - except Exception: # noqa: BLE001 - a broken install must not crash the check - return agent, workflow - return ( - list(getattr(stub_generator, "BASE_AGENT_REQUIREMENTS", agent)), - list(getattr(stub_generator, "BASE_WORKFLOW_REQUIREMENTS", workflow)), - ) - - -BASE_AGENT_REQUIREMENTS, BASE_WORKFLOW_REQUIREMENTS = _base_requirements() -# Import name -> every distribution that provides it. A tuple rather than a -# string because more than one distribution can ship the same import name, and -# reporting a correct declaration as a gap teaches the reader to dismiss W006. -IMPORT_TO_DISTRIBUTION = { - "attr": ("attrs",), - # `import autogen` is shipped by three unrelated distributions: pyautogen - # (Microsoft's original), ag2 (the community continuation), and a package - # literally named autogen. Any of them satisfies the import. - "autogen": ("pyautogen", "ag2", "autogen", "autogen-agentchat"), - "bs4": ("beautifulsoup4",), - "cv2": ("opencv-python",), - "dateutil": ("python-dateutil",), - "dotenv": ("python-dotenv",), - "grpc": ("grpcio",), - "grpc_tools": ("grpcio-tools",), - "jwt": ("pyjwt",), - "PIL": ("pillow",), - "psycopg": ("psycopg",), - "psycopg2": ("psycopg2-binary",), - "pydantic_settings": ("pydantic-settings",), - "sklearn": ("scikit-learn",), - "typing_extensions": ("typing-extensions",), - "yaml": ("pyyaml",), -} -# Import name -> distribution prefix, where the top-level package is shipped by -# a family of distributions rather than one. `import llama_index.llms.openai` -# collapses to `llama_index`, which no correctly scoped requirements list ever -# names: it declares llama-index-core, llama-index-llms-openai and so on. Any -# member of the family satisfies the import. -NAMESPACE_DISTRIBUTIONS = { - "llama_index": "llama-index", -} - -def _stdlib_names(): - """Module names the interpreter provides without any distribution. - - `sys.stdlib_module_names` exists only from 3.10. Below that, derive the set - from the interpreter's own library directory rather than shipping a list - that rots -- without it every `import os` in the copy becomes a W006, and a - check that cries wolf is a check the reader stops reading. - """ - names = getattr(sys, "stdlib_module_names", None) - if names: - return frozenset(names) - found = set(sys.builtin_module_names) - library = os.path.dirname(os.__file__) - try: - entries = os.listdir(library) - except OSError: - return frozenset(found) - for entry in entries: - if entry.endswith(".py"): - found.add(entry[:-3]) - elif "." not in entry and "-" not in entry: - found.add(entry) - return frozenset(found) - - -STDLIB_MODULE_NAMES = _stdlib_names() - SECRET_PATTERNS = [ (re.compile(r"sk-[A-Za-z0-9_-]{20,}"), "an OpenAI-style secret key"), (re.compile(r"AKIA[0-9A-Z]{16}"), "an AWS access key id"), @@ -160,319 +72,6 @@ def _stdlib_names(): ] SECRET_NAME = re.compile(r"(API_KEY|SECRET|TOKEN|PASSWORD|CREDENTIAL)", re.IGNORECASE) -ERROR = "ERROR" -WARN = "WARN" -INFO = "INFO" - - -# ------------------------------------------------------------------ # -# Capabilities # -# ------------------------------------------------------------------ # -# -# Stable labels for behavior detected from the importable runtime. They contain -# no external development metadata. - -CAPABILITY_SOURCE = { - "env_file": "runtime env-file injection", - "editable_install": "editable project installation", - "sweeps_all_files": "full project-file sweep", -} - - -def probe_capabilities(): - """Ask the importable ventis package what it actually supports.""" - caps = dict.fromkeys(CAPABILITY_SOURCE, False) - caps["ventis"] = False - try: - from ventis import stub_generator - except Exception: # noqa: BLE001 - a broken install must not crash the check - return caps - - caps["ventis"] = True - caps["editable_install"] = hasattr(stub_generator, "_install_step") - caps["sweeps_all_files"] = hasattr(stub_generator, "_sweep_project_files") - - import importlib - - for module_name in ("ventis.controller.utils.env_file", "ventis.utils.env_file"): - try: - module = importlib.import_module(module_name) - except Exception: # noqa: BLE001,S112 - the other path is the live one - continue - if hasattr(module, "resolve_env_file"): - caps["env_file"] = True - break - return caps - - -# ------------------------------------------------------------------ # -# YAML with line numbers # -# ------------------------------------------------------------------ # - - -class LineDict(dict): - """A mapping that remembers where it and each of its keys were written.""" - - line = 0 - key_lines: ClassVar[dict] = {} - - -class LineLoader(yaml.SafeLoader): - pass - - -def _construct_mapping(loader, node): - data = LineDict() - yield data - data.update(loader.construct_mapping(node, deep=False)) - data.line = node.start_mark.line + 1 - data.key_lines = { - key.value: key.start_mark.line + 1 - for key, _ in node.value - if isinstance(key, yaml.ScalarNode) - } - - -LineLoader.add_constructor( - yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, _construct_mapping -) - - -def line_of(mapping, key=None): - """The source line of `key` inside `mapping`, or of the mapping itself.""" - if not isinstance(mapping, LineDict): - return 0 - if key is not None: - return mapping.key_lines.get(key, mapping.line) - return mapping.line - - -def load_yaml(path): - """Parse `path`, returning (data, error). Never raises.""" - try: - with open(path, "r", encoding="utf-8") as handle: - return yaml.load(handle, Loader=LineLoader), None - except Exception as exc: # noqa: BLE001 - any parse failure is a finding - return None, str(exc) - - -# ------------------------------------------------------------------ # -# Findings # -# ------------------------------------------------------------------ # - - -class Report: - def __init__(self, project_dir, capabilities): - self.project_dir = project_dir - self.capabilities = capabilities - self.findings = [] - - def add(self, check, level, path, line, summary, mechanism): - self.findings.append( - { - "check": check, - "level": level, - "path": self.rel(path) if path else "", - "line": line or 0, - "summary": summary, - "mechanism": mechanism, - } - ) - - def error(self, check, path, line, summary, mechanism): - self.add(check, ERROR, path, line, summary, mechanism) - - def warn(self, check, path, line, summary, mechanism): - self.add(check, WARN, path, line, summary, mechanism) - - def unavailable(self, check, summary): - self.add(check, INFO, "", 0, summary, "") - - def rel(self, path): - try: - return os.path.relpath(path, self.project_dir) - except ValueError: - return path - - def counts(self): - errors = sum(1 for f in self.findings if f["level"] == ERROR) - warnings = sum(1 for f in self.findings if f["level"] == WARN) - return errors, warnings - - -# ------------------------------------------------------------------ # -# Python source helpers # -# ------------------------------------------------------------------ # - - -def parse_python(path): - """AST for `path`, or (None, error). The port is never imported.""" - try: - with open(path, "r", encoding="utf-8") as handle: - source = handle.read() - except OSError as exc: - return None, str(exc) - try: - return ast.parse(source, filename=path), None - except SyntaxError as exc: - return None, f"{exc.msg} (line {exc.lineno})" - - -def find_class(tree, name): - for node in tree.body: - if isinstance(node, ast.ClassDef) and node.name == name: - return node - return None - - -def class_methods(class_node): - return { - node.name: node - for node in class_node.body - if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) - } - - -def parameter_names(func_node): - """Every parameter a caller can pass by keyword, minus self.""" - args = func_node.args - positional = [a.arg for a in args.posonlyargs + args.args] - if positional and positional[0] in ("self", "cls"): - positional = positional[1:] - return positional + [a.arg for a in args.kwonlyargs] - - -def required_parameters(func_node): - """Parameters with no default, minus self.""" - args = func_node.args - positional = [a.arg for a in args.posonlyargs + args.args] - if positional and positional[0] in ("self", "cls"): - positional = positional[1:] - if args.defaults: - positional = positional[: len(positional) - len(args.defaults)] - kwonly = [ - arg.arg - for arg, default in zip(args.kwonlyargs, args.kw_defaults) - if default is None - ] - return positional + kwonly - - -def toplevel_import_names(tree): - """Top-level package name of every import in the module, with line numbers.""" - names = {} - for node in ast.walk(tree): - if isinstance(node, ast.Import): - for alias in node.names: - names.setdefault(alias.name.split(".")[0], node.lineno) - elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module: - names.setdefault(node.module.split(".")[0], node.lineno) - return names - - -def dotted_import_names(tree): - """Every absolute import in the module as its full dotted path, with lines.""" - names = {} - for node in ast.walk(tree): - if isinstance(node, ast.Import): - for alias in node.names: - names.setdefault(alias.name, node.lineno) - elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module: - names.setdefault(node.module, node.lineno) - return names - - -def _local_module_files(project_dir, dotted): - """Files inside the copy that `import ` executes, outermost first. - - Python runs every package `__init__.py` on the way down before the leaf - module. That is how an image ends up executing code it never names: the - workflow imports `pkg.agent`, `pkg/__init__.py` runs first, and whatever it - imports runs with it. - """ - parts = dotted.split(".") - found = [] - prefix = project_dir - for depth, part in enumerate(parts): - if os.path.isdir(os.path.join(prefix, part)): - init_path = os.path.join(prefix, part, "__init__.py") - if os.path.isfile(init_path): - found.append(init_path) - prefix = os.path.join(prefix, part) - continue - leaf = os.path.join(prefix, part + ".py") - if depth == len(parts) - 1 and os.path.isfile(leaf): - found.append(leaf) - return found - return found - - -def _relative_import_files(project_dir, path, tree): - """Files a module's own `from .sibling import x` imports execute.""" - root = os.path.realpath(project_dir) - found = [] - for node in ast.walk(tree): - if not isinstance(node, ast.ImportFrom) or not node.level: - continue - base = os.path.dirname(path) - for _ in range(node.level - 1): - base = os.path.dirname(base) - resolved = os.path.realpath(base) - if resolved != root and not resolved.startswith(root + os.sep): - continue - target = os.path.join(base, *(node.module.split(".") if node.module else [])) - candidates = [target + ".py", os.path.join(target, "__init__.py")] - candidates += [ - os.path.join(target, alias.name + ".py") for alias in node.names - ] - candidates += [ - os.path.join(target, alias.name, "__init__.py") for alias in node.names - ] - found += [c for c in candidates if os.path.isfile(c)] - return found - - -def reachable_imports(project_dir, root_path, shadowed_paths=()): - """Every third-party import the built image executes, transitively. - - An image runs far more than the file the config names. `tools/parser.py` is - one hop from an entrypoint and its pdfplumber import is invisible to an AST - walk of the entrypoint alone; a package `__init__.py` three hops up drags in - graphql-core. Both surface only as a ModuleNotFoundError deep inside an - import chain at agent load, long after a green build. - - `shadowed_paths` are real source modules replaced in this image before it - runs. In particular, a workflow image receives generated stubs over every - agent entrypoint, so following the overwritten implementations would report - dependencies that image never imports. - - Returns {dotted import name: (file that imports it, line)} for names that do - not resolve inside the copy. - """ - external = {} - seen = set() - shadowed = {os.path.realpath(path) for path in shadowed_paths} - queue = [os.path.realpath(root_path)] - while queue: - path = queue.pop() - if path in shadowed or path in seen or not os.path.isfile(path): - continue - seen.add(path) - tree, _ = parse_python(path) - if tree is None: - continue - for dotted, lineno in dotted_import_names(tree).items(): - local = _local_module_files(project_dir, dotted) - if local: - queue += [os.path.realpath(p) for p in local] - else: - external.setdefault(dotted, (path, lineno)) - queue += [ - os.path.realpath(p) - for p in _relative_import_files(project_dir, path, tree) - ] - return external - # ------------------------------------------------------------------ # # V006-V010 adapter failures hidden by _load_agent # @@ -654,7 +253,7 @@ def check_stub_imports(report, workflow_path, tree, stub_modules): flat, through a package re-export, or from a second copy of the module -- resolves to the real class instead, and the workflow runs the agent in-process with none of the deployment behind it. The class name is another - trap: `ventis build` prints one with a `Stub` suffix that it never writes. + trap: the deploy build prints one with a `Stub` suffix that it never writes. """ for node in ast.walk(tree): if not isinstance(node, ast.ImportFrom) or not node.module: @@ -1135,7 +734,7 @@ def check_secrets(report, port_paths): for number, line in enumerate(lines, start=1): for pattern, description in SECRET_PATTERNS: if pattern.search(line): - report.warn( + report.error( "W003", path, number, @@ -1279,7 +878,7 @@ def check_requirements_coverage( f"it runs {report.rel(where)} on the way there, and that module " "needs it." ) - report.warn( + report.error( "W006", where, lineno, @@ -1308,44 +907,24 @@ def _normalize_distribution(name): # ------------------------------------------------------------------ # -def find_agent_declarations(config_dir): - """Map agent name -> declaration, for every declaration in `config/`. - - Mirrors ventis/cli.py: declarations sit in `config/` beside the manifest, - which -- like `policy.yaml` -- carries no top-level `agent.name` and so - drops out here. - """ - import glob - - declarations = {} - for path in sorted(glob.glob(os.path.join(config_dir, "*.yaml"))): - data, error = load_yaml(path) - if error is not None or not isinstance(data, dict): - continue - agent = data.get("agent") - name = agent.get("name") if isinstance(agent, dict) else None - if isinstance(name, str) and name: - declarations[name] = (path, agent) - return declarations - - def module_path(entrypoint): """Dotted module name an entrypoint has inside the container.""" return os.path.splitext(entrypoint)[0].replace("\\", "/").replace("/", ".") def validate(artifact_dir, config_path, capabilities): - """Inspect only failures hidden behind a successful image build.""" + """Check the public artifact contract and deeper runtime failure modes.""" report = Report(artifact_dir, capabilities) - # The build owns config/YAML syntax and shape validation. We read only enough - # valid structure to locate code for the deeper checks below. config, error = load_yaml(config_path) if error is not None or not isinstance(config, dict): - report.unavailable( - "BUILD", - "runtime preflight skipped because the config cannot be read; " - "ventis build owns and reports this error.", + report.error( + "V001", + config_path, + 0, + f"the global manifest cannot be read: {error or 'expected a YAML mapping'}", + "Validation finishes before an approved `canyonos deploy`, so an " + "unreadable manifest cannot be deferred to deploy.", ) return report @@ -1363,17 +942,17 @@ def validate(artifact_dir, config_path, capabilities): ) return report - agents_by_name = find_agent_declarations(os.path.dirname(config_path)) + check_self_contained_tree(report, artifact_dir) - entries = config.get("agents") - if not isinstance(entries, list): - report.unavailable( - "BUILD", - "runtime preflight skipped because `agents:` is not a list; " - "ventis build owns and reports this error.", - ) + config_dir = os.path.dirname(config_path) + entries = check_manifest_structure(report, config, config_path, source_dir) + if entries is None: return report + agents_by_name = discover_agent_declarations(report, config_dir, config_path) + check_declaration_bindings(report, entries, agents_by_name, config_path) + check_policy(report, config_dir) + entrypoints = [] for entry in entries: if not isinstance(entry, dict) or entry.get("type", "agent") == "workflow": diff --git a/.claude/skills/porting-to-canyonos/validation/__init__.py b/.claude/skills/porting-to-canyonos/validation/__init__.py new file mode 100644 index 0000000..f6152f8 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/__init__.py @@ -0,0 +1,2 @@ +"""Composable validation checks for the CanyonOS porting skill.""" + diff --git a/.claude/skills/porting-to-canyonos/validation/core.py b/.claude/skills/porting-to-canyonos/validation/core.py new file mode 100644 index 0000000..684e5cc --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/core.py @@ -0,0 +1,99 @@ +"""Shared result and YAML primitives for validation checks.""" + +import os +from typing import ClassVar + +try: + import yaml +except ImportError: # pragma: no cover - pyyaml is a CanyonOS dependency + raise RuntimeError("validate.py needs pyyaml: pip install pyyaml") from None + + +ERROR = "ERROR" +WARN = "WARN" +INFO = "INFO" + + +class LineDict(dict): + """A mapping that remembers where it and each of its keys were written.""" + + line = 0 + key_lines: ClassVar[dict] = {} + + +class LineLoader(yaml.SafeLoader): + pass + + +def _construct_mapping(loader, node): + data = LineDict() + yield data + data.update(loader.construct_mapping(node, deep=False)) + data.line = node.start_mark.line + 1 + data.key_lines = { + key.value: key.start_mark.line + 1 + for key, _ in node.value + if isinstance(key, yaml.ScalarNode) + } + + +LineLoader.add_constructor( + yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, _construct_mapping +) + + +def line_of(mapping, key=None): + if not isinstance(mapping, LineDict): + return 0 + if key is not None: + return mapping.key_lines.get(key, mapping.line) + return mapping.line + + +def load_yaml(path): + """Parse path and return ``(data, error)`` without raising.""" + try: + with open(path, "r", encoding="utf-8") as handle: + return yaml.load(handle, Loader=LineLoader), None + except Exception as exc: # noqa: BLE001 - every parse failure is a finding + return None, str(exc) + + +class Report: + def __init__(self, project_dir, capabilities): + self.project_dir = project_dir + self.capabilities = capabilities + self.findings = [] + + def add(self, check, level, path, line, summary, mechanism): + self.findings.append( + { + "check": check, + "level": level, + "path": self.rel(path) if path else "", + "line": line or 0, + "summary": summary, + "mechanism": mechanism, + } + ) + + def error(self, check, path, line, summary, mechanism): + self.add(check, ERROR, path, line, summary, mechanism) + + def warn(self, check, path, line, summary, mechanism): + self.add(check, WARN, path, line, summary, mechanism) + + def unavailable(self, check, summary): + self.add(check, INFO, "", 0, summary, "") + + def rel(self, path): + try: + return os.path.relpath(path, self.project_dir) + except ValueError: + return path + + def counts(self): + errors = sum(1 for finding in self.findings if finding["level"] == ERROR) + warnings = sum(1 for finding in self.findings if finding["level"] == WARN) + return errors, warnings + diff --git a/.claude/skills/porting-to-canyonos/validation/manifest.py b/.claude/skills/porting-to-canyonos/validation/manifest.py new file mode 100644 index 0000000..df40c23 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/manifest.py @@ -0,0 +1,295 @@ +"""Fail-closed checks for the public CanyonOS artifact contract.""" + +import glob +import os + +from validation.core import line_of, load_yaml + +def _safe_relative_python_path(value): + if not isinstance(value, str) or not value.strip(): + return False + normalized = value.replace("\\", "/") + return ( + not normalized.startswith("/") + and ".." not in normalized.split("/") + and normalized.endswith(".py") + ) + + +def check_manifest_structure(report, config, config_path, source_dir): + """Return entries only when deeper checks can traverse them safely.""" + entries = config.get("agents") + if not isinstance(entries, list): + report.error( + "V001", + config_path, + line_of(config, "agents"), + "`agents:` must be a list", + "The CanyonOS manifest cannot be traversed or built without an agents list.", + ) + return None + + valid = True + agent_count = sum( + 1 + for entry in entries + if isinstance(entry, dict) and entry.get("type", "agent") == "agent" + ) + workflow_count = sum( + 1 + for entry in entries + if isinstance(entry, dict) and entry.get("type", "agent") == "workflow" + ) + if agent_count < 1: + report.error( + "V002", + config_path, + line_of(config, "agents"), + "the manifest must contain at least one agent service", + "A CanyonOS port needs a callable service behind its workflow.", + ) + valid = False + if workflow_count != 1: + report.error( + "V002", + config_path, + line_of(config, "agents"), + f"the manifest must contain exactly one workflow service; found {workflow_count}", + "The deployment exposes one `/main` workflow and builds one workflow image.", + ) + valid = False + for index, entry in enumerate(entries): + if not isinstance(entry, dict): + report.error( + "V002", + config_path, + 0, + f"agents[{index}] must be a mapping", + "CanyonOS reads each agents item as a service declaration.", + ) + valid = False + continue + + name = entry.get("name") + if not isinstance(name, str) or not name.strip(): + report.error( + "V002", + config_path, + line_of(entry, "name"), + f"agents[{index}] has no non-empty string `name`", + "Names bind manifest entries, declarations, generated stubs, and images.", + ) + valid = False + else: + earlier = [ + item.get("name") + for item in entries[:index] + if isinstance(item, dict) and isinstance(item.get("name"), str) + ] + collision = next( + (other for other in earlier if other.lower() == name.lower()), None + ) + if collision is not None: + report.error( + "V002", + config_path, + line_of(entry, "name"), + f"`{name}` collides with `{collision}` after lowercase normalization", + "CanyonOS uses lowercase image and target names, so one " + "service overwrites the other.", + ) + valid = False + + service_type = entry.get("type", "agent") + if service_type not in ("agent", "workflow"): + report.error( + "V002", + config_path, + line_of(entry, "type"), + f"`type: {service_type}` is neither `agent` nor `workflow`", + "Only those two service shapes have a CanyonOS build contract.", + ) + valid = False + + provider = entry.get("provider", "local") + if provider not in ("local", "EC2"): + report.error( + "V002", + config_path, + line_of(entry, "provider"), + f"unsupported provider spelling `{provider}`", + "Use lowercase `local` or uppercase `EC2`; runtime provider " + "handling is case-sensitive.", + ) + valid = False + + replicas = entry.get("replicas", 1) + if isinstance(replicas, bool) or not isinstance(replicas, int) or replicas < 1: + report.error( + "V002", + config_path, + line_of(entry, "replicas"), + "`replicas` must be an integer greater than zero", + "CanyonOS creates one placement per replica and cannot deploy " + "an empty or fractional set.", + ) + valid = False + + requirements = entry.get("requirements", []) + if not isinstance(requirements, list) or not all( + isinstance(item, str) and item.strip() for item in requirements + ): + report.error( + "V002", + config_path, + line_of(entry, "requirements"), + "`requirements` must be a list of non-empty strings", + "CanyonOS writes this list into the image requirements file.", + ) + valid = False + + path_key = "workflow_file" if service_type == "workflow" else "entrypoint" + relative = entry.get(path_key) + if not _safe_relative_python_path(relative): + report.error( + "V002", + config_path, + line_of(entry, path_key), + f"`{path_key}` must be a relative .py path contained by `.car/app`", + "Absolute and parent-relative paths escape the self-contained CanyonOS artifact.", + ) + valid = False + elif not os.path.isfile(os.path.join(source_dir, relative)): + report.error( + "V002", + config_path, + line_of(entry, path_key), + f"`{path_key}: {relative}` does not exist in `.car/app`", + "The deploy build cannot create this service without its Python entry file.", + ) + valid = False + + return entries if valid else None + + +def check_self_contained_tree(report, artifact_dir): + """Reject symlinks because `.car` must not depend on outside state.""" + for root, directories, files in os.walk(artifact_dir, followlinks=False): + for name in [*directories, *files]: + path = os.path.join(root, name) + if not os.path.islink(path): + continue + report.error( + "V036", + path, + 0, + "`.car` contains a symbolic link", + "A symlink can escape the artifact or be skipped by the " + "Python-file sweep. Copy the intended file or directory into " + "the artifact explicitly.", + ) + + +def discover_agent_declarations(report, config_dir, config_path): + """Load declarations without silently discarding malformed or duplicate YAML.""" + declarations = {} + for path in sorted(glob.glob(os.path.join(config_dir, "*.yaml"))): + data, error = load_yaml(path) + if error is not None: + report.error( + "V003", + path, + 0, + f"YAML cannot be parsed: {error}", + "CanyonOS reads every YAML file in the config directory during deploy.", + ) + continue + if not isinstance(data, dict): + if os.path.realpath(path) == os.path.realpath(config_path): + report.error( + "V003", + path, + 0, + "the global manifest must be a mapping", + "A scalar or empty manifest has no CanyonOS configuration contract.", + ) + continue + agent = data.get("agent") + if agent is None: + continue + if ( + not isinstance(agent, dict) + or not isinstance(agent.get("name"), str) + or not agent["name"] + ): + report.error( + "V003", + path, + line_of(data, "agent"), + "`agent` must contain a non-empty string `name`", + "The declaration name is the binding used to generate its stub.", + ) + continue + name = agent["name"] + if name in declarations: + report.error( + "V003", + path, + line_of(agent, "name"), + f"duplicate declaration for `{name}`", + "Filename ordering would otherwise choose one declaration silently.", + ) + continue + declarations[name] = (path, agent) + return declarations + + +def check_declaration_bindings(report, entries, declarations, config_path): + """Require a one-to-one binding for every agent service.""" + configured = { + entry["name"] + for entry in entries + if entry.get("type", "agent") != "workflow" + } + for name in sorted(configured - declarations.keys()): + report.error( + "V004", + config_path, + 0, + f"agent `{name}` has no matching declaration in `.car/config`", + f"Without `agent.name: {name}`, CanyonOS cannot generate the service stub.", + ) + for name in sorted(declarations.keys() - configured): + path, _ = declarations[name] + report.warn( + "V004", + path, + 0, + f"declaration `{name}` has no agent service in the manifest", + "It is stale or unused and will not produce a deployable service.", + ) + + +def check_policy(report, config_dir): + path = os.path.join(config_dir, "policy.yaml") + if not os.path.exists(path): + return + data, error = load_yaml(path) + if error is not None or not isinstance(data, dict): + report.error( + "V005", + path, + 0, + "policy.yaml must be a YAML mapping", + error or "CanyonOS reads `rules` from this mapping during deploy.", + ) + return + rules = data.get("rules") + if not isinstance(rules, list) or not rules: + report.error( + "V005", + path, + line_of(data, "rules"), + "policy.yaml must contain a non-empty `rules` list", + "Remove the file for unrestricted access; an empty policy is not a valid restriction.", + ) diff --git a/.claude/skills/porting-to-canyonos/validation/python_source.py b/.claude/skills/porting-to-canyonos/validation/python_source.py new file mode 100644 index 0000000..7c3eefe --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/python_source.py @@ -0,0 +1,152 @@ +"""Static Python-source discovery used by adapter and packaging checks.""" + +import ast +import os + + +def parse_python(path): + """Return ``(AST, None)`` or ``(None, error)`` without importing the file.""" + try: + with open(path, "r", encoding="utf-8") as handle: + source = handle.read() + except OSError as exc: + return None, str(exc) + try: + return ast.parse(source, filename=path), None + except SyntaxError as exc: + return None, f"{exc.msg} (line {exc.lineno})" + + +def find_class(tree, name): + for node in tree.body: + if isinstance(node, ast.ClassDef) and node.name == name: + return node + return None + + +def class_methods(class_node): + return { + node.name: node + for node in class_node.body + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) + } + + +def parameter_names(func_node): + """Return every keyword-callable parameter, excluding ``self``/``cls``.""" + args = func_node.args + positional = [arg.arg for arg in args.posonlyargs + args.args] + if positional and positional[0] in ("self", "cls"): + positional = positional[1:] + return positional + [arg.arg for arg in args.kwonlyargs] + + +def required_parameters(func_node): + """Return parameters without defaults, excluding ``self``/``cls``.""" + args = func_node.args + positional = [arg.arg for arg in args.posonlyargs + args.args] + if positional and positional[0] in ("self", "cls"): + positional = positional[1:] + if args.defaults: + positional = positional[: len(positional) - len(args.defaults)] + kwonly = [ + arg.arg + for arg, default in zip(args.kwonlyargs, args.kw_defaults) + if default is None + ] + return positional + kwonly + + +def toplevel_import_names(tree): + """Return top-level import names and their first line numbers.""" + names = {} + for node in ast.walk(tree): + if isinstance(node, ast.Import): + for alias in node.names: + names.setdefault(alias.name.split(".")[0], node.lineno) + elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module: + names.setdefault(node.module.split(".")[0], node.lineno) + return names + + +def dotted_import_names(tree): + """Return absolute dotted imports and their first line numbers.""" + names = {} + for node in ast.walk(tree): + if isinstance(node, ast.Import): + for alias in node.names: + names.setdefault(alias.name, node.lineno) + elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module: + names.setdefault(node.module, node.lineno) + return names + + +def _local_module_files(project_dir, dotted): + """Return local files executed by importing ``dotted``, outermost first.""" + parts = dotted.split(".") + found = [] + prefix = project_dir + for depth, part in enumerate(parts): + if os.path.isdir(os.path.join(prefix, part)): + init_path = os.path.join(prefix, part, "__init__.py") + if os.path.isfile(init_path): + found.append(init_path) + prefix = os.path.join(prefix, part) + continue + leaf = os.path.join(prefix, part + ".py") + if depth == len(parts) - 1 and os.path.isfile(leaf): + found.append(leaf) + return found + return found + + +def _relative_import_files(project_dir, path, tree): + """Return local files executed by a module's relative imports.""" + root = os.path.realpath(project_dir) + found = [] + for node in ast.walk(tree): + if not isinstance(node, ast.ImportFrom) or not node.level: + continue + base = os.path.dirname(path) + for _ in range(node.level - 1): + base = os.path.dirname(base) + resolved = os.path.realpath(base) + if resolved != root and not resolved.startswith(root + os.sep): + continue + target = os.path.join(base, *(node.module.split(".") if node.module else [])) + candidates = [target + ".py", os.path.join(target, "__init__.py")] + candidates += [ + os.path.join(target, alias.name + ".py") for alias in node.names + ] + candidates += [ + os.path.join(target, alias.name, "__init__.py") for alias in node.names + ] + found += [candidate for candidate in candidates if os.path.isfile(candidate)] + return found + + +def reachable_imports(project_dir, root_path, shadowed_paths=()): + """Return third-party imports reachable from ``root_path`` transitively.""" + external = {} + seen = set() + shadowed = {os.path.realpath(path) for path in shadowed_paths} + queue = [os.path.realpath(root_path)] + while queue: + path = queue.pop() + if path in shadowed or path in seen or not os.path.isfile(path): + continue + seen.add(path) + tree, _ = parse_python(path) + if tree is None: + continue + for dotted, lineno in dotted_import_names(tree).items(): + local = _local_module_files(project_dir, dotted) + if local: + queue += [os.path.realpath(item) for item in local] + else: + external.setdefault(dotted, (path, lineno)) + queue += [ + os.path.realpath(item) + for item in _relative_import_files(project_dir, path, tree) + ] + return external diff --git a/.claude/skills/porting-to-canyonos/validation/runtime.py b/.claude/skills/porting-to-canyonos/validation/runtime.py new file mode 100644 index 0000000..36b1e55 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/runtime.py @@ -0,0 +1,123 @@ +"""Runtime capabilities and dependency facts used by validation checks.""" + +import importlib +import os +import sys + + +RUNTIME_FLAT_NAMES = frozenset( + { + "future.py", + "ventis_context.py", + "local_controller.py", + "local_controller_frontend.py", + "redis_client.py", + "grpc_options.py", + "gpu_metrics.py", + "bedrock.py", + "deploy.py", + "session_logging.py", + "workflow_launcher.py", + } +) + +IMPORT_TO_DISTRIBUTION = { + "attr": ("attrs",), + "autogen": ("pyautogen", "ag2", "autogen", "autogen-agentchat"), + "bs4": ("beautifulsoup4",), + "cv2": ("opencv-python",), + "dateutil": ("python-dateutil",), + "dotenv": ("python-dotenv",), + "grpc": ("grpcio",), + "grpc_tools": ("grpcio-tools",), + "jwt": ("pyjwt",), + "PIL": ("pillow",), + "psycopg": ("psycopg",), + "psycopg2": ("psycopg2-binary",), + "pydantic_settings": ("pydantic-settings",), + "sklearn": ("scikit-learn",), + "typing_extensions": ("typing-extensions",), + "yaml": ("pyyaml",), +} + +NAMESPACE_DISTRIBUTIONS = {"llama_index": "llama-index"} + +CAPABILITY_SOURCE = { + "env_file": "runtime env-file injection", + "editable_install": "editable project installation", + "sweeps_all_files": "full project-file sweep", +} + + +def _base_requirements(): + agent = [ + "grpcio", + "grpcio-tools", + "redis", + "pyyaml", + "psutil", + "ipdb", + "ipython", + "boto3", + ] + workflow = [*agent, "flask", "sqlalchemy", "psycopg[binary]"] + try: + from ventis import stub_generator + except Exception: # noqa: BLE001 - a broken install must not crash validation + return agent, workflow + return ( + list(getattr(stub_generator, "BASE_AGENT_REQUIREMENTS", agent)), + list(getattr(stub_generator, "BASE_WORKFLOW_REQUIREMENTS", workflow)), + ) + + +def _stdlib_names(): + names = getattr(sys, "stdlib_module_names", None) + if names: + return frozenset(names) + found = set(sys.builtin_module_names) + library = os.path.dirname(os.__file__) + try: + entries = os.listdir(library) + except OSError: + return frozenset(found) + for entry in entries: + if entry.endswith(".py"): + found.add(entry[:-3]) + elif "." not in entry and "-" not in entry: + found.add(entry) + return frozenset(found) + + +BASE_AGENT_REQUIREMENTS, BASE_WORKFLOW_REQUIREMENTS = _base_requirements() +STDLIB_MODULE_NAMES = _stdlib_names() + + +def probe_capabilities(): + """Probe the installed compatibility runtime behind the CanyonOS CLI.""" + capabilities = dict.fromkeys(CAPABILITY_SOURCE, False) + capabilities["ventis"] = False + try: + from ventis import stub_generator + except Exception: # noqa: BLE001 - unavailable runtime is reported, not fatal + return capabilities + + capabilities["ventis"] = True + capabilities["editable_install"] = hasattr(stub_generator, "_install_step") + capabilities["sweeps_all_files"] = hasattr( + stub_generator, "_sweep_project_files" + ) + + for module_name in ( + "ventis.controller.utils.env_file", + "ventis.utils.env_file", + ): + try: + module = importlib.import_module(module_name) + except Exception: # noqa: BLE001 - try the other supported location + continue + if hasattr(module, "resolve_env_file"): + capabilities["env_file"] = True + break + return capabilities + From 279750593f3477835f3dac396d3d8cd5b52d5ce2 Mon Sep 17 00:00:00 2001 From: Nick Huo Date: Fri, 4 Sep 2026 13:02:07 -0700 Subject: [PATCH 7/7] Finish splitting the validator, and stop citing a table that no longer exists validate.py still held nineteen of the twenty-five checks after the last split, so `validation/` was a helper package with one very large caller rather than a set of checks. The remaining check functions move out by the section boundaries the file already had: validation/adapter.py V006-V010 adapter faults _load_agent swallows validation/workflow.py V016-V018, V023 the workflow and how it reaches an agent validation/entrypoint.py V019, V020, V033-V035 traps set by which module the entrypoint names validation/packaging.py V030-V031 credentials and import roots validation/dependencies.py W003, W006 credentials and imports a successful build does not reject `module_path` and the two import resolvers join python_source.py, which is where the other static-source helpers already live; both resolvers were private and are now shared by packaging.py and dependencies.py, so they lose the underscore. validate.py keeps argparse, the driver, and report printing: 1153 lines to 311. This is a pure move. All twenty-four relocated definitions are AST-identical to their originals, and ten fixtures covering every check code that lives outside manifest.py -- V002, V006-V010, V016-V020, V023, V030-V035, W003, W006 -- produce byte-identical `--json` output before and after. Four references still cited hard-rule labels by number: references/adapter.md M18 references/llm-proxy.md M18, M21 references/manifest.md M22, M23 The M1-M32 table went away when SKILL.md gained `## Source-integrity boundary`, which restates the four rules static analysis cannot prove and drops the twenty-eight the validator now checks itself. The rules survived; the labels did not, so each citation named a table a reader could not find. They now name the boundary. Also drops nine dead `# noqa: E402` directives and sorts the new modules' imports, which nets the skill's scripts down from twelve ruff findings to seven. The seven that remain are in prepare.py and runtime.py and predate this change. (cherry picked from commit 7cfbece8cb84a2bb591aa88d346a4742a95fb540) --- .../porting-to-canyonos/references/adapter.md | 5 +- .../references/llm-proxy.md | 6 +- .../references/manifest.md | 11 +- .../skills/porting-to-canyonos/validate.py | 876 +----------------- .../validation/__init__.py | 1 - .../porting-to-canyonos/validation/adapter.py | 175 ++++ .../porting-to-canyonos/validation/core.py | 1 - .../validation/dependencies.py | 207 +++++ .../validation/entrypoint.py | 141 +++ .../validation/manifest.py | 5 +- .../validation/packaging.py | 108 +++ .../validation/python_source.py | 33 +- .../porting-to-canyonos/validation/runtime.py | 5 +- .../validation/workflow.py | 213 +++++ 14 files changed, 906 insertions(+), 881 deletions(-) create mode 100644 .claude/skills/porting-to-canyonos/validation/adapter.py create mode 100644 .claude/skills/porting-to-canyonos/validation/dependencies.py create mode 100644 .claude/skills/porting-to-canyonos/validation/entrypoint.py create mode 100644 .claude/skills/porting-to-canyonos/validation/packaging.py create mode 100644 .claude/skills/porting-to-canyonos/validation/workflow.py diff --git a/.claude/skills/porting-to-canyonos/references/adapter.md b/.claude/skills/porting-to-canyonos/references/adapter.md index 00e8d37..606e4ba 100644 --- a/.claude/skills/porting-to-canyonos/references/adapter.md +++ b/.claude/skills/porting-to-canyonos/references/adapter.md @@ -45,8 +45,9 @@ another. 5. **Does module-level code perform a real run?** A script ending in `result = crew.kickoff(...)` / `print(result)` fires that run whenever the module loads, before a request exists. Delete the - invocation and keep the construction: M18 protects prompts, tools, schemas - and model calls, not a script's own main body. + invocation and keep the construction. SKILL.md's source-integrity boundary + protects prompts, tools, schemas, model calls and node bodies -- not a + script's own main body. ## Bridging async diff --git a/.claude/skills/porting-to-canyonos/references/llm-proxy.md b/.claude/skills/porting-to-canyonos/references/llm-proxy.md index f660672..d2047bb 100644 --- a/.claude/skills/porting-to-canyonos/references/llm-proxy.md +++ b/.claude/skills/porting-to-canyonos/references/llm-proxy.md @@ -57,9 +57,9 @@ credentials in the separate proxy process, not in the port's `env_file`. Some sources build the HTTP call themselves -- `urllib.request` against a module constant like `API = "https://api.openai.com/v1/responses"` -- and read no -base-URL variable at all. Editing that constant is a source edit M18 and M21 -forbid, so the `env_file` is inert and the container can only ever reach the -real provider. Report this as a proxy blocker and stop. Do not hand the +base-URL variable at all. Editing that constant swaps the source provider's +endpoint, which SKILL.md's source-integrity boundary forbids, so the `env_file` +is inert and the container can only ever reach the real provider. Report this as a proxy blocker and stop. Do not hand the container a real upstream credential instead. Detect it before deploying: grep the source for the provider hostname. A literal diff --git a/.claude/skills/porting-to-canyonos/references/manifest.md b/.claude/skills/porting-to-canyonos/references/manifest.md index 1a6a9df..e90188b 100644 --- a/.claude/skills/porting-to-canyonos/references/manifest.md +++ b/.claude/skills/porting-to-canyonos/references/manifest.md @@ -102,9 +102,9 @@ you wrote: In a peer image the entrypoint is a stub, but its package `__init__` and its siblings are real, so that image still installs what they import. 3. Omit distributions reachable only from source files no image imports, such as - a Gradio or Streamlit UI beside the agent. M22 forbids reclassifying a - declared dependency, not declining to ship an unreachable one; name what you - left out in the report. + a Gradio or Streamlit UI beside the agent. The source-integrity boundary + forbids reclassifying a declared dependency, not declining to ship an + unreachable one; name what you left out in the report. `validate.py` walks the same graph and reports what is missing as W006. @@ -121,8 +121,9 @@ today: - **The source predates a known SDK break**: pin contemporaneous with its last commit. A 2023 AutoGen script passing `request_timeout=` needs `pyautogen==0.1.14`, which depends on `openai<1`, not `autogen==0.7.5`, which - floors on `openai>=1.58` where that kwarg is `timeout`. M23 forbids rewriting - the source call, so the pin has to absorb the difference. Compare the source's + floors on `openai>=1.58` where that kwarg is `timeout`. The source-integrity + boundary forbids rewriting that call, so the pin has to absorb the + difference. Compare the source's commit date against the pin's release date whenever the source hardcodes SDK kwargs. diff --git a/.claude/skills/porting-to-canyonos/validate.py b/.claude/skills/porting-to-canyonos/validate.py index 0eb9e93..61788e8 100755 --- a/.claude/skills/porting-to-canyonos/validate.py +++ b/.claude/skills/porting-to-canyonos/validate.py @@ -21,885 +21,48 @@ """ import argparse -import ast -import builtins import json import os -import re import sys SKILL_DIR = os.path.dirname(os.path.abspath(__file__)) if SKILL_DIR not in sys.path: sys.path.insert(0, SKILL_DIR) -from validation.core import ERROR, INFO, WARN, Report, line_of, load_yaml # noqa: E402 -from validation.manifest import ( # noqa: E402 +from validation.adapter import check_adapter +from validation.core import ERROR, INFO, WARN, Report, load_yaml +from validation.dependencies import ( + check_requirements_coverage, + check_secrets, +) +from validation.entrypoint import ( + check_entrypoint_module, + check_flat_collisions, +) +from validation.manifest import ( check_declaration_bindings, check_manifest_structure, check_policy, check_self_contained_tree, discover_agent_declarations, ) -from validation.python_source import ( # noqa: E402 - class_methods, - find_class, - parameter_names, - parse_python, - reachable_imports, - required_parameters, - toplevel_import_names, -) -from validation.runtime import ( # noqa: E402 +from validation.packaging import check_env_file, check_import_root +from validation.python_source import module_path +from validation.runtime import ( BASE_AGENT_REQUIREMENTS, BASE_WORKFLOW_REQUIREMENTS, CAPABILITY_SOURCE, - IMPORT_TO_DISTRIBUTION, - NAMESPACE_DISTRIBUTIONS, - RUNTIME_FLAT_NAMES, - STDLIB_MODULE_NAMES, probe_capabilities, ) +from validation.workflow import check_workflow DEFAULT_CONFIG_PATH = "config/global_controller.yaml" # ventis/cli.py SOURCE_DIR_NAME -- the duplicated application source. SOURCE_DIR_NAME = "app" -SECRET_PATTERNS = [ - (re.compile(r"sk-[A-Za-z0-9_-]{20,}"), "an OpenAI-style secret key"), - (re.compile(r"AKIA[0-9A-Z]{16}"), "an AWS access key id"), - (re.compile(r"gh[pousr]_[A-Za-z0-9]{20,}"), "a GitHub token"), - (re.compile(r"AIza[0-9A-Za-z_-]{30,}"), "a Google API key"), -] -SECRET_NAME = re.compile(r"(API_KEY|SECRET|TOKEN|PASSWORD|CREDENTIAL)", re.IGNORECASE) - - -# ------------------------------------------------------------------ # -# V006-V010 adapter failures hidden by _load_agent # -# ------------------------------------------------------------------ # - -BUILTIN_TYPE_NAMES = frozenset( - name - for name in dir(builtins) - if isinstance(getattr(builtins, name), type) and not name[0].isupper() -) - - -def check_adapter(report, agent_yaml_path, agent_block, entry, project_dir): - """V006 V007 V008 V009 V010.""" - name = agent_block["name"] - functions = agent_block.get("functions") or [] - check_argument_types(report, agent_yaml_path, functions) - - entrypoint = entry.get("entrypoint") - if not entrypoint: - return - entrypoint_path = os.path.join(project_dir, entrypoint) - if not os.path.isfile(entrypoint_path): - return - - tree, error = parse_python(entrypoint_path) - if tree is None: - report.error( - "V006", - entrypoint_path, - 0, - f"the entrypoint does not parse: {error}", - "_load_agent exec_module's it and swallows the exception; the first " - "request answers 'No agent loaded'.", - ) - return - - class_node = find_class(tree, name) - if class_node is None: - classes = [n.name for n in tree.body if isinstance(n, ast.ClassDef)] - found = ", ".join(classes) if classes else "no classes at all" - report.error( - "V006", - entrypoint_path, - 1, - f"no class named `{name}` at module level (found: {found})", - "_load_agent does getattr(module, VENTIS_AGENT_NAME) and swallows " - "the AttributeError. The class name must equal agent.name exactly.", - ) - return - - methods = class_methods(class_node) - check_constructor(report, entrypoint_path, name, methods) - - for func in functions: - if not isinstance(func, dict) or not isinstance(func.get("name"), str): - continue - check_method(report, entrypoint_path, agent_yaml_path, name, func, methods) - - -def check_argument_types(report, agent_yaml_path, functions): - """V010 -- the type string is pasted into an ast.Name, never checked.""" - for func in functions: - if not isinstance(func, dict): - continue - for arg in func.get("arguments") or []: - if not isinstance(arg, dict) or "type" not in arg: - continue - declared = arg.get("type") - if not isinstance(declared, str): - continue # stub generation reports malformed type values - if declared in BUILTIN_TYPE_NAMES: - continue - report.error( - "V010", - agent_yaml_path, - line_of(arg, "type"), - f"`type: {declared}` is not a builtin", - "stub_generator pastes it verbatim into the generated " - "annotation, and the stub module imports only Future and " - "inspect. Anything else raises NameError when the stub is " - "imported -- after a green build. Use str int float bool dict " - "list.", - ) - - -def check_constructor(report, entrypoint_path, name, methods): - """V007 -- _load_agent calls agent_class() with no arguments.""" - init = methods.get("__init__") - if init is None: - return - required = required_parameters(init) - if required: - report.error( - "V007", - entrypoint_path, - init.lineno, - f"`{name}.__init__` requires {', '.join(required)}", - "_load_agent calls agent_class() with no arguments; the TypeError is " - "swallowed and the first request answers 'No agent loaded'. Read " - "configuration from the environment inside __init__ instead.", - ) - - -def check_method(report, entrypoint_path, agent_yaml_path, class_name, func, methods): - """V008 V009.""" - func_name = func["name"] - method = methods.get(func_name) - if method is None: - report.error( - "V008", - entrypoint_path, - 0, - f"`{class_name}` has no method `{func_name}`", - "The yaml declares it, so callers get a stub for it; the controller " - f"then answers \"Agent {class_name} has no method '{func_name}'\".", - ) - return - - # V009 -- nothing on the execution path awaits. - if isinstance(method, ast.AsyncFunctionDef): - report.error( - "V009", - entrypoint_path, - method.lineno, - f"`{class_name}.{func_name}` is `async def`", - "The executor calls method(**args) with no await, so Redis receives " - "''. Keep the signature synchronous and call " - "asyncio.run(...) inside the body.", - ) - - # V008 -- the controller calls method(**args) with the yaml's names. - declared = [ - arg["name"] - for arg in func.get("arguments") or [] - if isinstance(arg, dict) and isinstance(arg.get("name"), str) - ] - actual = parameter_names(method) - required = required_parameters(method) - - missing = [arg for arg in declared if arg not in actual] - if missing: - report.error( - "V008", - entrypoint_path, - method.lineno, - f"`{class_name}.{func_name}` has no parameter " - f"{', '.join(repr(m) for m in missing)}, but the yaml declares it", - "LocalController does method(**args) with the yaml's argument names. " - "A mismatch is TypeError: unexpected keyword argument, at request " - f"time. See {os.path.basename(agent_yaml_path)}.", - ) - - unfilled = [arg for arg in required if arg not in declared] - if unfilled: - report.error( - "V008", - entrypoint_path, - method.lineno, - f"`{class_name}.{func_name}` requires {', '.join(unfilled)}, which " - "the yaml does not declare", - "Only declared arguments are ever sent, and the generated stub gives " - "none of them a default. Declare them in the yaml or default them " - "in the signature.", - ) - - - -# ------------------------------------------------------------------ # -# V016-V018 the workflow # -# ------------------------------------------------------------------ # - - -def check_stub_imports(report, workflow_path, tree, stub_modules): - """V023 -- the workflow must import each agent from its own entrypoint module. - - The build writes a stub over exactly one path: the agent's `entrypoint` - inside the source copy. An import that reaches the class any other way -- - flat, through a package re-export, or from a second copy of the module -- - resolves to the real class instead, and the workflow runs the agent - in-process with none of the deployment behind it. The class name is another - trap: the deploy build prints one with a `Stub` suffix that it never writes. - """ - for node in ast.walk(tree): - if not isinstance(node, ast.ImportFrom) or not node.module: - continue - for alias in node.names: - name = alias.name - base = name.removesuffix("Stub") - expected = stub_modules.get(base) - if expected is None: - continue - if name.endswith("Stub"): - report.error( - "V023", workflow_path, node.lineno, - f"`{name}` is the name the build prints, not the class it " - f"writes", - "generate_stub sets class_name = agent_config['name'] and " - "then recomputes it with a 'Stub' suffix for the log line " - "only. The message names a class that does not exist; the " - f"class is `{base}`.", - ) - elif node.module != expected: - report.error( - "V023", workflow_path, node.lineno, - f"`from {node.module} import {name}` -- the stub for {name} " - f"is written to {expected.replace('.', '/')}.py", - "The build replaces the module at the agent's entrypoint " - "and nothing else, so this import reaches the real class " - "and runs the agent in this process instead of over gRPC. " - f"Import it from `{expected}`, where the source already " - "keeps it.", - ) - - -def check_workflow(report, workflow_path, stub_modules=None): - """V016 V017 V018 V023.""" - tree, error = parse_python(workflow_path) - if tree is None: - report.error("V016", workflow_path, 0, f"does not parse: {error}", "") - return - - main = None - for node in tree.body: - if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and ( - node.name == "main" - ): - main = node - break - if main is None: - defined = [ - n.name - for n in tree.body - if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef)) - ] - found = ", ".join(defined) if defined else "no top-level functions" - report.error( - "V016", - workflow_path, - 1, - f"no top-level function named `main` (found: {found})", - "CanyonOS Core serves POST /, but the deployment platform's " - "test endpoint posts to a hardcoded /main. A differently named " - "workflow builds, deploys and stays unreachable -- 404, container " - "healthy.", - ) - else: - check_main_signature(report, workflow_path, main) - - if stub_modules: - check_stub_imports(report, workflow_path, tree, stub_modules) - - # V016 -- deploy() is what starts Flask. - if not any( - isinstance(node, ast.Call) - and isinstance(node.func, ast.Name) - and node.func.id == "deploy" - for node in ast.walk(tree) - ): - report.error( - "V016", - workflow_path, - 1, - "the workflow never calls `deploy(...)`", - "workflow_launcher.py exec's this file and nothing else starts the " - "HTTP server; the container comes up serving nothing.", - ) - - check_main_guard(report, workflow_path, tree) - check_fused_fanout(report, workflow_path, tree) - - -def check_main_signature(report, workflow_path, main): - """V016 -- the platform sends exactly {"query": ...}.""" - if isinstance(main, ast.AsyncFunctionDef): - report.error( - "V016", - workflow_path, - main.lineno, - "`main` is `async def`", - "deploy() calls workflow_fn(**kwargs) on a Flask worker thread with " - "no await; the response body would be a coroutine repr.", - ) - params = parameter_names(main) - if not params: - report.error( - "V016", - workflow_path, - main.lineno, - "`main` takes no arguments", - 'The platform posts {"query": "..."} and deploy() splats the ' - "body in as kwargs -- TypeError on every request.", - ) - return - if params[0] != "query": - report.error( - "V016", - workflow_path, - main.lineno, - f"`main`'s first parameter is `{params[0]}`, not `query`", - "The platform's body schema is strictly validated as " - "{query: string}; any other key is rejected with 400 in the control " - "plane, before the request reaches the host.", - ) - extra = [p for p in required_parameters(main) if p != "query"] - if extra: - report.error( - "V016", - workflow_path, - main.lineno, - f"`main` requires {', '.join(extra)} beyond `query`", - "Only `query` is ever sent, so every other parameter needs a " - "default or the call raises on every request. Pack richer input " - "into `query`.", - ) - - -def check_main_guard(report, workflow_path, tree): - """V017 -- the workflow is exec'd, so __name__ == "__main__".""" - for node in tree.body: - if not isinstance(node, ast.If): - continue - test = node.test - if ( - isinstance(test, ast.Compare) - and isinstance(test.left, ast.Name) - and test.left.id == "__name__" - and any( - isinstance(c, ast.Constant) and c.value == "__main__" - for c in test.comparators - ) - ): - report.error( - "V017", - workflow_path, - node.lineno, - '`if __name__ == "__main__":` block in the workflow', - "workflow_launcher.py runs exec(open().read()), so " - "__name__ IS '__main__' here and this block executes in " - "production, at container start.", - ) - - -def check_fused_fanout(report, workflow_path, tree): - """V018 -- .value() blocks, so dispatching and resolving in one - comprehension runs the fan-out one call at a time.""" - comprehensions = (ast.ListComp, ast.SetComp, ast.GeneratorExp) - for node in ast.walk(tree): - if not isinstance(node, comprehensions + (ast.DictComp,)): - continue - elements = ( - [node.key, node.value] if isinstance(node, ast.DictComp) else [node.elt] - ) - for element in elements: - for inner in ast.walk(element): - if ( - isinstance(inner, ast.Call) - and isinstance(inner.func, ast.Attribute) - and inner.func.attr == "value" - and isinstance(inner.func.value, ast.Call) - ): - report.error( - "V018", - workflow_path, - node.lineno, - "one comprehension both dispatches a call and resolves " - "it with .value()", - ".value() blocks, so each call completes before the " - "next is dispatched. It does not error -- the fan-out " - "is just silently serial, and with it the reason to be " - "on CanyonOS Core. Dispatch every call first, then resolve: " - "futures = [a.work(i) for i in items] then " - "[f.value() for f in futures].", - ) - return - - -# ------------------------------------------------------------------ # -# V019-V020 what the copy order overwrites # -# ------------------------------------------------------------------ # - - -def check_flat_collisions(report, source_dir, entrypoints): - """V019 V020 -- later copies land on earlier ones at the context root.""" - for entry in sorted(os.listdir(source_dir)): - path = os.path.join(source_dir, entry) - if not os.path.isfile(path) or not entry.endswith(".py"): - continue - - # V019 -- the shared runtime is copied flat, after the project sweep. - if entry in RUNTIME_FLAT_NAMES: - report.error( - "V019", - path, - 1, - f"a project module named `{entry}` sits at the root of the " - "source copy", - "The shared CanyonOS Core runtime is copied flat into the image after " - "the project sweep, so this file is overwritten by CanyonOS Core's own " - f"{entry}. Rename it or move it into a package directory.", - ) - - # V020 -- one module cannot be the entrypoint of two agents. - owners = {} - for name, entrypoint in entrypoints: - owners.setdefault(entrypoint, []).append(name) - for entrypoint, names in sorted(owners.items()): - if len(names) < 2: - continue - report.error( - "V020", - os.path.join(source_dir, entrypoint), - 1, - f"{' and '.join(sorted(names))} both declare `{entrypoint}` as their " - "entrypoint", - "Each agent's stub is written over its own entrypoint, so the two " - "land on one path and the last one built wins. Every caller then " - "reaches whichever agent that was. Give each agent its own module.", - ) - - -# ------------------------------------------------------------------ # -# V030-V031 capability-gated rules # -# ------------------------------------------------------------------ # - - -def check_env_file(report, config, config_path, artifact_dir): - """V030 -- gated on detected env-file injection support.""" - declared = config.get("env_file") - supported = report.capabilities.get("env_file") - - if not supported: - if declared: - report.error( - "V030", - config_path, - line_of(config, "env_file"), - f"`env_file: {declared}` is set, but this CanyonOS Core never reads it", - "No resolve_env_file in the importable ventis package, so the " - "key is silently dropped and the container answers a provider " - "credential error on the first request. This port requires the " - "`env_file` runtime capability.", - ) - else: - report.unavailable( - "V030", - "env_file is not supported by the importable `ventis` runtime. " - "Credentials have no declared path into a container on this tree.", - ) - return - - if not declared: - report.warn( - "V030", - config_path, - line_of(config), - "no `env_file:` in the config", - "Only runtime-managed VENTIS_* variables are guaranteed without it. " - "If the source reads credentials from the environment, the first " - "request fails on a provider error.", - ) - return - - # Path existence and readability are deploy-preflight checks. Do not - # duplicate them here. - - -def check_import_root(report, source_dir, entrypoint_paths): - """V031 -- gated on detected editable-install support.""" - supported = report.capabilities.get("editable_install") - has_metadata = any( - os.path.isfile(os.path.join(source_dir, name)) - for name in ("pyproject.toml", "setup.py", "setup.cfg") - ) - - non_flat = [] - for path in entrypoint_paths: - tree, _ = parse_python(path) - if tree is None: - continue - for name, lineno in toplevel_import_names(tree).items(): - if f"{name}.py" in RUNTIME_FLAT_NAMES: - continue - if _resolves_flat(source_dir, name): - continue - location = _resolves_nested(source_dir, name) - if location: - non_flat.append((path, lineno, name, location)) - - if not supported: - report.unavailable( - "V031", - "the editable install (`-e .`) is not supported by the importable " - "`ventis` runtime. Only names rooted at /app import inside a container.", - ) - for path, lineno, name, location in non_flat: - report.error( - "V031", - path, - lineno, - f"`import {name}` resolves to {location}, which is not at the " - "root of the source copy", - "sys.path[0] is /app and this CanyonOS Core runs no editable install, " - "so only modules swept to the root import. The adapter raises " - "ModuleNotFoundError inside _load_agent and the first request " - "answers 'No agent loaded'.", - ) - return - - if non_flat and not has_metadata: - for path, lineno, name, location in non_flat: - report.error( - "V031", - path, - lineno, - f"`import {name}` resolves to {location}, and the source copy's " - "root has no packaging metadata", - "A pyproject.toml, setup.py or setup.cfg at the root of the " - "source copy is what adds `-e .`; metadata nested deeper in the " - "tree is ignored. Add minimal root metadata pointing at the " - "existing package directory. Without it the install is skipped " - "silently.", - ) - - -# ------------------------------------------------------------------ # -# V033-V035 traps set by where the entrypoint sits # -# ------------------------------------------------------------------ # - - -def check_entrypoint_module(report, source_dir, name, entrypoint): - """V033 V034 V035. - - Two runtime facts collide here. The build writes this agent's stub over - `entrypoint` in every image except this agent's own, and the controller - loads the real file by path rather than by import. Each breaks a module - layout that is correct everywhere else in Python. - """ - path = os.path.join(source_dir, entrypoint) - if not os.path.isfile(path): - return - - segments = os.path.splitext(entrypoint)[0].replace("\\", "/").split("/") - invalid = [part for part in segments if not part.isidentifier()] - if invalid: - report.error( - "V034", - path, - 0, - f"`{invalid[0]}` in the entrypoint path is not a Python identifier", - "The controller loads the entrypoint by file path, so this file runs " - "-- but the workflow has to import the class from " - f"`{module_path(entrypoint)}` (V023), and that is a SyntaxError, not " - "an ImportError. Rename the file inside the copy, or point " - "`entrypoint` at a normally-named sibling that loads this file by " - "path and re-exposes the class.", - ) - - tree, _ = parse_python(path) - if tree is not None: - for node in ast.walk(tree): - if isinstance(node, ast.ImportFrom) and node.level: - spelling = "." * node.level + (node.module or "") - report.error( - "V035", - path, - node.lineno, - f"the entrypoint's own `from {spelling} import ...` is relative", - "_load_agent loads this file with spec_from_file_location(" - "VENTIS_AGENT_FILE.replace('.py', ''), path). That name keeps " - "the entrypoint's directory separator, so it has no parent " - "package and __package__ is empty: every relative import in " - "this file raises 'attempted relative import with no known " - "parent package' at agent load, behind 'No agent loaded'. " - "Make this file's own top-level imports absolute; modules it " - "imports may keep theirs.", - ) - break - - directory = os.path.dirname(entrypoint) - if not directory: - return - init_path = os.path.join(source_dir, directory, "__init__.py") - if not os.path.isfile(init_path): - return - module = os.path.splitext(os.path.basename(entrypoint))[0] - package = directory.replace("\\", "/").replace("/", ".") - init_tree, _ = parse_python(init_path) - if init_tree is None: - return - for node in ast.walk(init_tree): - if not isinstance(node, ast.ImportFrom): - continue - target = node.module or "" - hit = target == module if node.level else target in ( - module, - f"{package}.{module}", - ) - if not hit and node.level and not node.module: - hit = any(alias.name == module for alias in node.names) - if not hit: - continue - report.error( - "V033", - init_path, - node.lineno, - f"`{package}/__init__.py` re-exports from `{module}`, the entrypoint " - f"for {name}", - "Python runs a package's __init__.py before any of its submodules, " - "and in every image except this agent's own the module at the " - "entrypoint is the generated stub, which defines the agent class and " - f"nothing else. Any peer image that imports anything from `{package}` " - "-- the workflow importing the agent class included -- re-runs this " - "re-export against the stub and dies at container startup with " - f"ImportError. Point `entrypoint` at a module `{package}/__init__.py` " - "does not re-export from; add one that imports the real module if " - "every existing module is re-exported.", - ) - break - - -def _resolves_flat(project_dir, name): - """Whether Python can resolve `name` with /app as its import root. - - A directory does not need __init__.py: PEP 420 namespace packages resolve - from sys.path just like regular packages. - """ - return os.path.isfile(os.path.join(project_dir, f"{name}.py")) or os.path.isdir( - os.path.join(project_dir, name) - ) - - -def _resolves_nested(project_dir, name): - """Where below /app `name` lives but cannot resolve as a top-level name.""" - for root, dirs, files in os.walk(project_dir): - dirs[:] = [d for d in dirs if not d.startswith(".") and d != "__pycache__"] - if root == project_dir: - continue - if f"{name}.py" in files: - return os.path.relpath(os.path.join(root, f"{name}.py"), project_dir) - if name in dirs: - return os.path.relpath(os.path.join(root, name), project_dir) - return None - - -# ------------------------------------------------------------------ # -# W003, W006 secrets and imports a green build does not reject # -# ------------------------------------------------------------------ # - - -def check_secrets(report, port_paths): - """W003 -- env_file is the way in; nothing else is.""" - for path in port_paths: - try: - with open(path, "r", encoding="utf-8") as handle: - lines = handle.read().splitlines() - except OSError: - continue - for number, line in enumerate(lines, start=1): - for pattern, description in SECRET_PATTERNS: - if pattern.search(line): - report.error( - "W003", - path, - number, - f"this line looks like {description}", - "Never put a secret in the source tree or the build " - "context. The build sweeps the project into every " - "image.", - ) - break - - tree, _ = parse_python(path) - if tree is None: - continue - for node in ast.walk(tree): - if not isinstance(node, ast.Assign): - continue - if not ( - isinstance(node.value, ast.Constant) - and isinstance(node.value.value, str) - and node.value.value.strip() - ): - continue - for target in node.targets: - if isinstance(target, ast.Name) and SECRET_NAME.search(target.id): - report.warn( - "W003", - path, - node.lineno, - f"`{target.id}` is assigned a literal string", - "Read it from the environment instead; the build sweeps " - "this file into every image.", - ) - - -def _pyproject_dependencies(project_dir): - """What `-e .` installs alongside `requirements:`, or None if unreadable. - - None and the empty set mean different things here: empty means the project - declares no dependencies, None means we could not find out -- a setup.py, or - a tomllib this interpreter does not have. The caller must not treat the - second as the first, or it warns about imports the install would satisfy. - """ - path = os.path.join(project_dir, "pyproject.toml") - if not os.path.isfile(path): - return None - try: - import tomllib - except ImportError: # < 3.11 - return None - try: - with open(path, "rb") as handle: - data = tomllib.load(handle) - except Exception: # noqa: BLE001 - malformed metadata is uv's error to give - return None - deps = (data.get("project") or {}).get("dependencies") - if not isinstance(deps, list): - return None - return {_normalize_distribution(d) for d in deps if isinstance(d, str)} - - -def check_requirements_coverage( - report, - project_dir, - entry, - root_path, - config_path, - base_requirements, - shadowed_paths=(), -): - """W006 -- an import the container cannot satisfy. - - Walks the whole import graph the image executes from `root_path`, not just - that one file: a distribution reached through a local module or a package - __init__ is exactly as missing, and exactly as invisible until the container - starts. - """ - declared = { - _normalize_distribution(item) - for item in (entry.get("requirements") or []) - if isinstance(item, str) - } - - # Where the editable install exists, `-e .` resolves the project's own - # [project.dependencies] in the same pass as `requirements:`. Warning about - # those is a false positive, and a false warning about a dependency is worse - # than none: it teaches the reader to dismiss this check. - editable = report.capabilities.get("editable_install") - metadata = any( - os.path.isfile(os.path.join(project_dir, name)) - for name in ("pyproject.toml", "setup.py", "setup.cfg") - ) - unreadable_metadata = False - if editable and metadata: - project_deps = _pyproject_dependencies(project_dir) - if project_deps is None: - unreadable_metadata = True - else: - declared |= project_deps - base = {_normalize_distribution(item) for item in base_requirements} - satisfied = base | declared - - external = reachable_imports(project_dir, root_path, shadowed_paths) - for dotted, (where, lineno) in sorted(external.items()): - name = dotted.split(".")[0] - if name in STDLIB_MODULE_NAMES or name == "ventis": - continue - # Provided by the image itself: the shared runtime is copied flat over - # the swept tree. A stub is not listed here -- it replaces a module the - # source copy already carries, so the tree checks below cover it. - if f"{name}.py" in RUNTIME_FLAT_NAMES: - continue - if _resolves_flat(project_dir, name) or _resolves_nested(project_dir, name): - continue - prefix = NAMESPACE_DISTRIBUTIONS.get(name) - if prefix and any( - item == prefix or item.startswith(prefix + "-") for item in declared - ): - continue - if _candidate_distributions(name) & satisfied: - continue - if unreadable_metadata: - mechanism = ( - "The container installs the base list, `requirements:`, and -- " - "since this project declares packaging metadata -- whatever " - "`-e .` resolves from it. That metadata could not be read here, " - f"so if it already requires `{name}` this line is noise; " - "otherwise it is a ModuleNotFoundError inside _load_agent and " - "'No agent loaded' on the first request." - ) - else: - mechanism = ( - "The container installs the base list plus `requirements:` and " - "nothing else, so this is a ModuleNotFoundError inside " - "_load_agent and 'No agent loaded' on the first request. If the " - f"distribution is named something other than `{name}`, declare " - f"that name in {report.rel(config_path)}." - ) - if os.path.realpath(where) != os.path.realpath(root_path): - mechanism += ( - f" This image never names `{name}` in {report.rel(root_path)}; " - f"it runs {report.rel(where)} on the way there, and that module " - "needs it." - ) - report.error( - "W006", - where, - lineno, - f"`import {dotted}` is in neither the runtime's base list nor " - f"{entry.get('name') or 'this entry'}'s `requirements:`", - mechanism, - ) - - -def _candidate_distributions(name): - """Every distribution name that would satisfy `import `.""" - return { - _normalize_distribution(item) - for item in IMPORT_TO_DISTRIBUTION.get(name, (name,)) - } - - -def _normalize_distribution(name): - return re.split(r"[<>=!\[;\s]", name.strip().lower(), maxsplit=1)[0].replace( - "_", "-" - ) +# Path existence and readability are deploy-preflight checks. Do not +# duplicate them here. # ------------------------------------------------------------------ # @@ -907,11 +70,6 @@ def _normalize_distribution(name): # ------------------------------------------------------------------ # -def module_path(entrypoint): - """Dotted module name an entrypoint has inside the container.""" - return os.path.splitext(entrypoint)[0].replace("\\", "/").replace("/", ".") - - def validate(artifact_dir, config_path, capabilities): """Check the public artifact contract and deeper runtime failure modes.""" report = Report(artifact_dir, capabilities) diff --git a/.claude/skills/porting-to-canyonos/validation/__init__.py b/.claude/skills/porting-to-canyonos/validation/__init__.py index f6152f8..aa7a1a3 100644 --- a/.claude/skills/porting-to-canyonos/validation/__init__.py +++ b/.claude/skills/porting-to-canyonos/validation/__init__.py @@ -1,2 +1 @@ """Composable validation checks for the CanyonOS porting skill.""" - diff --git a/.claude/skills/porting-to-canyonos/validation/adapter.py b/.claude/skills/porting-to-canyonos/validation/adapter.py new file mode 100644 index 0000000..445ff10 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/adapter.py @@ -0,0 +1,175 @@ +"""V006-V010 -- adapter faults the controller swallows inside _load_agent.""" + +import ast +import builtins +import os + +from validation.core import line_of +from validation.python_source import ( + class_methods, + find_class, + parameter_names, + parse_python, + required_parameters, +) + +BUILTIN_TYPE_NAMES = frozenset( + name + for name in dir(builtins) + if isinstance(getattr(builtins, name), type) and not name[0].isupper() +) + + +def check_adapter(report, agent_yaml_path, agent_block, entry, project_dir): + """V006 V007 V008 V009 V010.""" + name = agent_block["name"] + functions = agent_block.get("functions") or [] + check_argument_types(report, agent_yaml_path, functions) + + entrypoint = entry.get("entrypoint") + if not entrypoint: + return + entrypoint_path = os.path.join(project_dir, entrypoint) + if not os.path.isfile(entrypoint_path): + return + + tree, error = parse_python(entrypoint_path) + if tree is None: + report.error( + "V006", + entrypoint_path, + 0, + f"the entrypoint does not parse: {error}", + "_load_agent exec_module's it and swallows the exception; the first " + "request answers 'No agent loaded'.", + ) + return + + class_node = find_class(tree, name) + if class_node is None: + classes = [n.name for n in tree.body if isinstance(n, ast.ClassDef)] + found = ", ".join(classes) if classes else "no classes at all" + report.error( + "V006", + entrypoint_path, + 1, + f"no class named `{name}` at module level (found: {found})", + "_load_agent does getattr(module, VENTIS_AGENT_NAME) and swallows " + "the AttributeError. The class name must equal agent.name exactly.", + ) + return + + methods = class_methods(class_node) + check_constructor(report, entrypoint_path, name, methods) + + for func in functions: + if not isinstance(func, dict) or not isinstance(func.get("name"), str): + continue + check_method(report, entrypoint_path, agent_yaml_path, name, func, methods) + + +def check_argument_types(report, agent_yaml_path, functions): + """V010 -- the type string is pasted into an ast.Name, never checked.""" + for func in functions: + if not isinstance(func, dict): + continue + for arg in func.get("arguments") or []: + if not isinstance(arg, dict) or "type" not in arg: + continue + declared = arg.get("type") + if not isinstance(declared, str): + continue # stub generation reports malformed type values + if declared in BUILTIN_TYPE_NAMES: + continue + report.error( + "V010", + agent_yaml_path, + line_of(arg, "type"), + f"`type: {declared}` is not a builtin", + "stub_generator pastes it verbatim into the generated " + "annotation, and the stub module imports only Future and " + "inspect. Anything else raises NameError when the stub is " + "imported -- after a green build. Use str int float bool dict " + "list.", + ) + + +def check_constructor(report, entrypoint_path, name, methods): + """V007 -- _load_agent calls agent_class() with no arguments.""" + init = methods.get("__init__") + if init is None: + return + required = required_parameters(init) + if required: + report.error( + "V007", + entrypoint_path, + init.lineno, + f"`{name}.__init__` requires {', '.join(required)}", + "_load_agent calls agent_class() with no arguments; the TypeError is " + "swallowed and the first request answers 'No agent loaded'. Read " + "configuration from the environment inside __init__ instead.", + ) + + +def check_method(report, entrypoint_path, agent_yaml_path, class_name, func, methods): + """V008 V009.""" + func_name = func["name"] + method = methods.get(func_name) + if method is None: + report.error( + "V008", + entrypoint_path, + 0, + f"`{class_name}` has no method `{func_name}`", + "The yaml declares it, so callers get a stub for it; the controller " + f"then answers \"Agent {class_name} has no method '{func_name}'\".", + ) + return + + # V009 -- nothing on the execution path awaits. + if isinstance(method, ast.AsyncFunctionDef): + report.error( + "V009", + entrypoint_path, + method.lineno, + f"`{class_name}.{func_name}` is `async def`", + "The executor calls method(**args) with no await, so Redis receives " + "''. Keep the signature synchronous and call " + "asyncio.run(...) inside the body.", + ) + + # V008 -- the controller calls method(**args) with the yaml's names. + declared = [ + arg["name"] + for arg in func.get("arguments") or [] + if isinstance(arg, dict) and isinstance(arg.get("name"), str) + ] + actual = parameter_names(method) + required = required_parameters(method) + + missing = [arg for arg in declared if arg not in actual] + if missing: + report.error( + "V008", + entrypoint_path, + method.lineno, + f"`{class_name}.{func_name}` has no parameter " + f"{', '.join(repr(m) for m in missing)}, but the yaml declares it", + "LocalController does method(**args) with the yaml's argument names. " + "A mismatch is TypeError: unexpected keyword argument, at request " + f"time. See {os.path.basename(agent_yaml_path)}.", + ) + + unfilled = [arg for arg in required if arg not in declared] + if unfilled: + report.error( + "V008", + entrypoint_path, + method.lineno, + f"`{class_name}.{func_name}` requires {', '.join(unfilled)}, which " + "the yaml does not declare", + "Only declared arguments are ever sent, and the generated stub gives " + "none of them a default. Declare them in the yaml or default them " + "in the signature.", + ) diff --git a/.claude/skills/porting-to-canyonos/validation/core.py b/.claude/skills/porting-to-canyonos/validation/core.py index 684e5cc..490b949 100644 --- a/.claude/skills/porting-to-canyonos/validation/core.py +++ b/.claude/skills/porting-to-canyonos/validation/core.py @@ -96,4 +96,3 @@ def counts(self): errors = sum(1 for finding in self.findings if finding["level"] == ERROR) warnings = sum(1 for finding in self.findings if finding["level"] == WARN) return errors, warnings - diff --git a/.claude/skills/porting-to-canyonos/validation/dependencies.py b/.claude/skills/porting-to-canyonos/validation/dependencies.py new file mode 100644 index 0000000..4060319 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/dependencies.py @@ -0,0 +1,207 @@ +"""W003, W006 -- credentials and imports a successful build does not reject.""" + +import ast +import os +import re + +from validation.python_source import ( + parse_python, + reachable_imports, + resolves_flat, + resolves_nested, +) +from validation.runtime import ( + IMPORT_TO_DISTRIBUTION, + NAMESPACE_DISTRIBUTIONS, + RUNTIME_FLAT_NAMES, + STDLIB_MODULE_NAMES, +) + +SECRET_PATTERNS = [ + (re.compile(r"sk-[A-Za-z0-9_-]{20,}"), "an OpenAI-style secret key"), + (re.compile(r"AKIA[0-9A-Z]{16}"), "an AWS access key id"), + (re.compile(r"gh[pousr]_[A-Za-z0-9]{20,}"), "a GitHub token"), + (re.compile(r"AIza[0-9A-Za-z_-]{30,}"), "a Google API key"), +] + + +SECRET_NAME = re.compile(r"(API_KEY|SECRET|TOKEN|PASSWORD|CREDENTIAL)", re.IGNORECASE) + + +def check_secrets(report, port_paths): + """W003 -- env_file is the way in; nothing else is.""" + for path in port_paths: + try: + with open(path, "r", encoding="utf-8") as handle: + lines = handle.read().splitlines() + except OSError: + continue + for number, line in enumerate(lines, start=1): + for pattern, description in SECRET_PATTERNS: + if pattern.search(line): + report.error( + "W003", + path, + number, + f"this line looks like {description}", + "Never put a secret in the source tree or the build " + "context. The build sweeps the project into every " + "image.", + ) + break + + tree, _ = parse_python(path) + if tree is None: + continue + for node in ast.walk(tree): + if not isinstance(node, ast.Assign): + continue + if not ( + isinstance(node.value, ast.Constant) + and isinstance(node.value.value, str) + and node.value.value.strip() + ): + continue + for target in node.targets: + if isinstance(target, ast.Name) and SECRET_NAME.search(target.id): + report.warn( + "W003", + path, + node.lineno, + f"`{target.id}` is assigned a literal string", + "Read it from the environment instead; the build sweeps " + "this file into every image.", + ) + + +def _pyproject_dependencies(project_dir): + """What `-e .` installs alongside `requirements:`, or None if unreadable. + + None and the empty set mean different things here: empty means the project + declares no dependencies, None means we could not find out -- a setup.py, or + a tomllib this interpreter does not have. The caller must not treat the + second as the first, or it warns about imports the install would satisfy. + """ + path = os.path.join(project_dir, "pyproject.toml") + if not os.path.isfile(path): + return None + try: + import tomllib + except ImportError: # < 3.11 + return None + try: + with open(path, "rb") as handle: + data = tomllib.load(handle) + except Exception: # noqa: BLE001 - malformed metadata is uv's error to give + return None + deps = (data.get("project") or {}).get("dependencies") + if not isinstance(deps, list): + return None + return {_normalize_distribution(d) for d in deps if isinstance(d, str)} + + +def check_requirements_coverage( + report, + project_dir, + entry, + root_path, + config_path, + base_requirements, + shadowed_paths=(), +): + """W006 -- an import the container cannot satisfy. + + Walks the whole import graph the image executes from `root_path`, not just + that one file: a distribution reached through a local module or a package + __init__ is exactly as missing, and exactly as invisible until the container + starts. + """ + declared = { + _normalize_distribution(item) + for item in (entry.get("requirements") or []) + if isinstance(item, str) + } + + # Where the editable install exists, `-e .` resolves the project's own + # [project.dependencies] in the same pass as `requirements:`. Warning about + # those is a false positive, and a false warning about a dependency is worse + # than none: it teaches the reader to dismiss this check. + editable = report.capabilities.get("editable_install") + metadata = any( + os.path.isfile(os.path.join(project_dir, name)) + for name in ("pyproject.toml", "setup.py", "setup.cfg") + ) + unreadable_metadata = False + if editable and metadata: + project_deps = _pyproject_dependencies(project_dir) + if project_deps is None: + unreadable_metadata = True + else: + declared |= project_deps + base = {_normalize_distribution(item) for item in base_requirements} + satisfied = base | declared + + external = reachable_imports(project_dir, root_path, shadowed_paths) + for dotted, (where, lineno) in sorted(external.items()): + name = dotted.split(".")[0] + if name in STDLIB_MODULE_NAMES or name == "ventis": + continue + # Provided by the image itself: the shared runtime is copied flat over + # the swept tree. A stub is not listed here -- it replaces a module the + # source copy already carries, so the tree checks below cover it. + if f"{name}.py" in RUNTIME_FLAT_NAMES: + continue + if resolves_flat(project_dir, name) or resolves_nested(project_dir, name): + continue + prefix = NAMESPACE_DISTRIBUTIONS.get(name) + if prefix and any( + item == prefix or item.startswith(prefix + "-") for item in declared + ): + continue + if _candidate_distributions(name) & satisfied: + continue + if unreadable_metadata: + mechanism = ( + "The container installs the base list, `requirements:`, and -- " + "since this project declares packaging metadata -- whatever " + "`-e .` resolves from it. That metadata could not be read here, " + f"so if it already requires `{name}` this line is noise; " + "otherwise it is a ModuleNotFoundError inside _load_agent and " + "'No agent loaded' on the first request." + ) + else: + mechanism = ( + "The container installs the base list plus `requirements:` and " + "nothing else, so this is a ModuleNotFoundError inside " + "_load_agent and 'No agent loaded' on the first request. If the " + f"distribution is named something other than `{name}`, declare " + f"that name in {report.rel(config_path)}." + ) + if os.path.realpath(where) != os.path.realpath(root_path): + mechanism += ( + f" This image never names `{name}` in {report.rel(root_path)}; " + f"it runs {report.rel(where)} on the way there, and that module " + "needs it." + ) + report.error( + "W006", + where, + lineno, + f"`import {dotted}` is in neither the runtime's base list nor " + f"{entry.get('name') or 'this entry'}'s `requirements:`", + mechanism, + ) + + +def _candidate_distributions(name): + """Every distribution name that would satisfy `import `.""" + return { + _normalize_distribution(item) + for item in IMPORT_TO_DISTRIBUTION.get(name, (name,)) + } + + +def _normalize_distribution(name): + return re.split(r"[<>=!\[;\s]", name.strip().lower(), maxsplit=1)[0].replace( + "_", "-" + ) diff --git a/.claude/skills/porting-to-canyonos/validation/entrypoint.py b/.claude/skills/porting-to-canyonos/validation/entrypoint.py new file mode 100644 index 0000000..902c3f8 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/entrypoint.py @@ -0,0 +1,141 @@ +"""V019, V020, V033-V035 -- traps set by which module the entrypoint names.""" + +import ast +import os + +from validation.python_source import module_path, parse_python +from validation.runtime import RUNTIME_FLAT_NAMES + + +def check_flat_collisions(report, source_dir, entrypoints): + """V019 V020 -- later copies land on earlier ones at the context root.""" + for entry in sorted(os.listdir(source_dir)): + path = os.path.join(source_dir, entry) + if not os.path.isfile(path) or not entry.endswith(".py"): + continue + + # V019 -- the shared runtime is copied flat, after the project sweep. + if entry in RUNTIME_FLAT_NAMES: + report.error( + "V019", + path, + 1, + f"a project module named `{entry}` sits at the root of the source copy", + "The shared CanyonOS Core runtime is copied flat into the image after " + "the project sweep, so this file is overwritten by CanyonOS Core's own " + f"{entry}. Rename it or move it into a package directory.", + ) + + # V020 -- one module cannot be the entrypoint of two agents. + owners = {} + for name, entrypoint in entrypoints: + owners.setdefault(entrypoint, []).append(name) + for entrypoint, names in sorted(owners.items()): + if len(names) < 2: + continue + report.error( + "V020", + os.path.join(source_dir, entrypoint), + 1, + f"{' and '.join(sorted(names))} both declare `{entrypoint}` as their " + "entrypoint", + "Each agent's stub is written over its own entrypoint, so the two " + "land on one path and the last one built wins. Every caller then " + "reaches whichever agent that was. Give each agent its own module.", + ) + + +def check_entrypoint_module(report, source_dir, name, entrypoint): + """V033 V034 V035. + + Two runtime facts collide here. The build writes this agent's stub over + `entrypoint` in every image except this agent's own, and the controller + loads the real file by path rather than by import. Each breaks a module + layout that is correct everywhere else in Python. + """ + path = os.path.join(source_dir, entrypoint) + if not os.path.isfile(path): + return + + segments = os.path.splitext(entrypoint)[0].replace("\\", "/").split("/") + invalid = [part for part in segments if not part.isidentifier()] + if invalid: + report.error( + "V034", + path, + 0, + f"`{invalid[0]}` in the entrypoint path is not a Python identifier", + "The controller loads the entrypoint by file path, so this file runs " + "-- but the workflow has to import the class from " + f"`{module_path(entrypoint)}` (V023), and that is a SyntaxError, not " + "an ImportError. Rename the file inside the copy, or point " + "`entrypoint` at a normally-named sibling that loads this file by " + "path and re-exposes the class.", + ) + + tree, _ = parse_python(path) + if tree is not None: + for node in ast.walk(tree): + if isinstance(node, ast.ImportFrom) and node.level: + spelling = "." * node.level + (node.module or "") + report.error( + "V035", + path, + node.lineno, + f"the entrypoint's own `from {spelling} import ...` is relative", + "_load_agent loads this file with spec_from_file_location(" + "VENTIS_AGENT_FILE.replace('.py', ''), path). That name keeps " + "the entrypoint's directory separator, so it has no parent " + "package and __package__ is empty: every relative import in " + "this file raises 'attempted relative import with no known " + "parent package' at agent load, behind 'No agent loaded'. " + "Make this file's own top-level imports absolute; modules it " + "imports may keep theirs.", + ) + break + + directory = os.path.dirname(entrypoint) + if not directory: + return + init_path = os.path.join(source_dir, directory, "__init__.py") + if not os.path.isfile(init_path): + return + module = os.path.splitext(os.path.basename(entrypoint))[0] + package = directory.replace("\\", "/").replace("/", ".") + init_tree, _ = parse_python(init_path) + if init_tree is None: + return + for node in ast.walk(init_tree): + if not isinstance(node, ast.ImportFrom): + continue + target = node.module or "" + hit = ( + target == module + if node.level + else target + in ( + module, + f"{package}.{module}", + ) + ) + if not hit and node.level and not node.module: + hit = any(alias.name == module for alias in node.names) + if not hit: + continue + report.error( + "V033", + init_path, + node.lineno, + f"`{package}/__init__.py` re-exports from `{module}`, the entrypoint " + f"for {name}", + "Python runs a package's __init__.py before any of its submodules, " + "and in every image except this agent's own the module at the " + "entrypoint is the generated stub, which defines the agent class and " + f"nothing else. Any peer image that imports anything from `{package}` " + "-- the workflow importing the agent class included -- re-runs this " + "re-export against the stub and dies at container startup with " + f"ImportError. Point `entrypoint` at a module `{package}/__init__.py` " + "does not re-export from; add one that imports the real module if " + "every existing module is re-exported.", + ) + break diff --git a/.claude/skills/porting-to-canyonos/validation/manifest.py b/.claude/skills/porting-to-canyonos/validation/manifest.py index df40c23..a69b7d4 100644 --- a/.claude/skills/porting-to-canyonos/validation/manifest.py +++ b/.claude/skills/porting-to-canyonos/validation/manifest.py @@ -5,6 +5,7 @@ from validation.core import line_of, load_yaml + def _safe_relative_python_path(value): if not isinstance(value, str) or not value.strip(): return False @@ -247,9 +248,7 @@ def discover_agent_declarations(report, config_dir, config_path): def check_declaration_bindings(report, entries, declarations, config_path): """Require a one-to-one binding for every agent service.""" configured = { - entry["name"] - for entry in entries - if entry.get("type", "agent") != "workflow" + entry["name"] for entry in entries if entry.get("type", "agent") != "workflow" } for name in sorted(configured - declarations.keys()): report.error( diff --git a/.claude/skills/porting-to-canyonos/validation/packaging.py b/.claude/skills/porting-to-canyonos/validation/packaging.py new file mode 100644 index 0000000..d0682fc --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/packaging.py @@ -0,0 +1,108 @@ +"""V030-V031 -- capability-gated rules about credentials and import roots.""" + +import os + +from validation.core import line_of +from validation.python_source import ( + parse_python, + resolves_flat, + resolves_nested, + toplevel_import_names, +) +from validation.runtime import RUNTIME_FLAT_NAMES + + +def check_env_file(report, config, config_path, artifact_dir): + """V030 -- gated on detected env-file injection support.""" + declared = config.get("env_file") + supported = report.capabilities.get("env_file") + + if not supported: + if declared: + report.error( + "V030", + config_path, + line_of(config, "env_file"), + f"`env_file: {declared}` is set, but this CanyonOS Core never reads it", + "No resolve_env_file in the importable ventis package, so the " + "key is silently dropped and the container answers a provider " + "credential error on the first request. This port requires the " + "`env_file` runtime capability.", + ) + else: + report.unavailable( + "V030", + "env_file is not supported by the importable `ventis` runtime. " + "Credentials have no declared path into a container on this tree.", + ) + return + + if not declared: + report.warn( + "V030", + config_path, + line_of(config), + "no `env_file:` in the config", + "Only runtime-managed VENTIS_* variables are guaranteed without it. " + "If the source reads credentials from the environment, the first " + "request fails on a provider error.", + ) + return + + +def check_import_root(report, source_dir, entrypoint_paths): + """V031 -- gated on detected editable-install support.""" + supported = report.capabilities.get("editable_install") + has_metadata = any( + os.path.isfile(os.path.join(source_dir, name)) + for name in ("pyproject.toml", "setup.py", "setup.cfg") + ) + + non_flat = [] + for path in entrypoint_paths: + tree, _ = parse_python(path) + if tree is None: + continue + for name, lineno in toplevel_import_names(tree).items(): + if f"{name}.py" in RUNTIME_FLAT_NAMES: + continue + if resolves_flat(source_dir, name): + continue + location = resolves_nested(source_dir, name) + if location: + non_flat.append((path, lineno, name, location)) + + if not supported: + report.unavailable( + "V031", + "the editable install (`-e .`) is not supported by the importable " + "`ventis` runtime. Only names rooted at /app import inside a container.", + ) + for path, lineno, name, location in non_flat: + report.error( + "V031", + path, + lineno, + f"`import {name}` resolves to {location}, which is not at the " + "root of the source copy", + "sys.path[0] is /app and this CanyonOS Core runs no editable install, " + "so only modules swept to the root import. The adapter raises " + "ModuleNotFoundError inside _load_agent and the first request " + "answers 'No agent loaded'.", + ) + return + + if non_flat and not has_metadata: + for path, lineno, name, location in non_flat: + report.error( + "V031", + path, + lineno, + f"`import {name}` resolves to {location}, and the source copy's " + "root has no packaging metadata", + "A pyproject.toml, setup.py or setup.cfg at the root of the " + "source copy is what adds `-e .`; metadata nested deeper in the " + "tree is ignored. Add minimal root metadata pointing at the " + "existing package directory. Without it the install is skipped " + "silently.", + ) diff --git a/.claude/skills/porting-to-canyonos/validation/python_source.py b/.claude/skills/porting-to-canyonos/validation/python_source.py index 7c3eefe..50df1e6 100644 --- a/.claude/skills/porting-to-canyonos/validation/python_source.py +++ b/.claude/skills/porting-to-canyonos/validation/python_source.py @@ -115,9 +115,7 @@ def _relative_import_files(project_dir, path, tree): continue target = os.path.join(base, *(node.module.split(".") if node.module else [])) candidates = [target + ".py", os.path.join(target, "__init__.py")] - candidates += [ - os.path.join(target, alias.name + ".py") for alias in node.names - ] + candidates += [os.path.join(target, alias.name + ".py") for alias in node.names] candidates += [ os.path.join(target, alias.name, "__init__.py") for alias in node.names ] @@ -150,3 +148,32 @@ def reachable_imports(project_dir, root_path, shadowed_paths=()): for item in _relative_import_files(project_dir, path, tree) ] return external + + +def module_path(entrypoint): + """Dotted module name an entrypoint has inside the container.""" + return os.path.splitext(entrypoint)[0].replace("\\", "/").replace("/", ".") + + +def resolves_flat(project_dir, name): + """Whether Python can resolve `name` with /app as its import root. + + A directory does not need __init__.py: PEP 420 namespace packages resolve + from sys.path just like regular packages. + """ + return os.path.isfile(os.path.join(project_dir, f"{name}.py")) or os.path.isdir( + os.path.join(project_dir, name) + ) + + +def resolves_nested(project_dir, name): + """Where below /app `name` lives but cannot resolve as a top-level name.""" + for root, dirs, files in os.walk(project_dir): + dirs[:] = [d for d in dirs if not d.startswith(".") and d != "__pycache__"] + if root == project_dir: + continue + if f"{name}.py" in files: + return os.path.relpath(os.path.join(root, f"{name}.py"), project_dir) + if name in dirs: + return os.path.relpath(os.path.join(root, name), project_dir) + return None diff --git a/.claude/skills/porting-to-canyonos/validation/runtime.py b/.claude/skills/porting-to-canyonos/validation/runtime.py index 36b1e55..4ec438a 100644 --- a/.claude/skills/porting-to-canyonos/validation/runtime.py +++ b/.claude/skills/porting-to-canyonos/validation/runtime.py @@ -104,9 +104,7 @@ def probe_capabilities(): capabilities["ventis"] = True capabilities["editable_install"] = hasattr(stub_generator, "_install_step") - capabilities["sweeps_all_files"] = hasattr( - stub_generator, "_sweep_project_files" - ) + capabilities["sweeps_all_files"] = hasattr(stub_generator, "_sweep_project_files") for module_name in ( "ventis.controller.utils.env_file", @@ -120,4 +118,3 @@ def probe_capabilities(): capabilities["env_file"] = True break return capabilities - diff --git a/.claude/skills/porting-to-canyonos/validation/workflow.py b/.claude/skills/porting-to-canyonos/validation/workflow.py new file mode 100644 index 0000000..613c4ce --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/workflow.py @@ -0,0 +1,213 @@ +"""V016-V018, V023 -- the workflow module and how it reaches an agent.""" + +import ast + +from validation.python_source import parameter_names, parse_python, required_parameters + + +def check_stub_imports(report, workflow_path, tree, stub_modules): + """V023 -- the workflow must import each agent from its own entrypoint module. + + The build writes a stub over exactly one path: the agent's `entrypoint` + inside the source copy. An import that reaches the class any other way -- + flat, through a package re-export, or from a second copy of the module -- + resolves to the real class instead, and the workflow runs the agent + in-process with none of the deployment behind it. The class name is another + trap: the deploy build prints one with a `Stub` suffix that it never writes. + """ + for node in ast.walk(tree): + if not isinstance(node, ast.ImportFrom) or not node.module: + continue + for alias in node.names: + name = alias.name + base = name.removesuffix("Stub") + expected = stub_modules.get(base) + if expected is None: + continue + if name.endswith("Stub"): + report.error( + "V023", + workflow_path, + node.lineno, + f"`{name}` is the name the build prints, not the class it writes", + "generate_stub sets class_name = agent_config['name'] and " + "then recomputes it with a 'Stub' suffix for the log line " + "only. The message names a class that does not exist; the " + f"class is `{base}`.", + ) + elif node.module != expected: + report.error( + "V023", + workflow_path, + node.lineno, + f"`from {node.module} import {name}` -- the stub for {name} " + f"is written to {expected.replace('.', '/')}.py", + "The build replaces the module at the agent's entrypoint " + "and nothing else, so this import reaches the real class " + "and runs the agent in this process instead of over gRPC. " + f"Import it from `{expected}`, where the source already " + "keeps it.", + ) + + +def check_workflow(report, workflow_path, stub_modules=None): + """V016 V017 V018 V023.""" + tree, error = parse_python(workflow_path) + if tree is None: + report.error("V016", workflow_path, 0, f"does not parse: {error}", "") + return + + main = None + for node in tree.body: + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and ( + node.name == "main" + ): + main = node + break + + if main is None: + defined = [ + n.name + for n in tree.body + if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef)) + ] + found = ", ".join(defined) if defined else "no top-level functions" + report.error( + "V016", + workflow_path, + 1, + f"no top-level function named `main` (found: {found})", + "CanyonOS Core serves POST /, but the deployment platform's " + "test endpoint posts to a hardcoded /main. A differently named " + "workflow builds, deploys and stays unreachable -- 404, container " + "healthy.", + ) + else: + check_main_signature(report, workflow_path, main) + + if stub_modules: + check_stub_imports(report, workflow_path, tree, stub_modules) + + # V016 -- deploy() is what starts Flask. + if not any( + isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "deploy" + for node in ast.walk(tree) + ): + report.error( + "V016", + workflow_path, + 1, + "the workflow never calls `deploy(...)`", + "workflow_launcher.py exec's this file and nothing else starts the " + "HTTP server; the container comes up serving nothing.", + ) + + check_main_guard(report, workflow_path, tree) + check_fused_fanout(report, workflow_path, tree) + + +def check_main_signature(report, workflow_path, main): + """V016 -- the platform sends exactly {"query": ...}.""" + if isinstance(main, ast.AsyncFunctionDef): + report.error( + "V016", + workflow_path, + main.lineno, + "`main` is `async def`", + "deploy() calls workflow_fn(**kwargs) on a Flask worker thread with " + "no await; the response body would be a coroutine repr.", + ) + params = parameter_names(main) + if not params: + report.error( + "V016", + workflow_path, + main.lineno, + "`main` takes no arguments", + 'The platform posts {"query": "..."} and deploy() splats the ' + "body in as kwargs -- TypeError on every request.", + ) + return + if params[0] != "query": + report.error( + "V016", + workflow_path, + main.lineno, + f"`main`'s first parameter is `{params[0]}`, not `query`", + "The platform's body schema is strictly validated as " + "{query: string}; any other key is rejected with 400 in the control " + "plane, before the request reaches the host.", + ) + extra = [p for p in required_parameters(main) if p != "query"] + if extra: + report.error( + "V016", + workflow_path, + main.lineno, + f"`main` requires {', '.join(extra)} beyond `query`", + "Only `query` is ever sent, so every other parameter needs a " + "default or the call raises on every request. Pack richer input " + "into `query`.", + ) + + +def check_main_guard(report, workflow_path, tree): + """V017 -- the workflow is exec'd, so __name__ == "__main__".""" + for node in tree.body: + if not isinstance(node, ast.If): + continue + test = node.test + if ( + isinstance(test, ast.Compare) + and isinstance(test.left, ast.Name) + and test.left.id == "__name__" + and any( + isinstance(c, ast.Constant) and c.value == "__main__" + for c in test.comparators + ) + ): + report.error( + "V017", + workflow_path, + node.lineno, + '`if __name__ == "__main__":` block in the workflow', + "workflow_launcher.py runs exec(open().read()), so " + "__name__ IS '__main__' here and this block executes in " + "production, at container start.", + ) + + +def check_fused_fanout(report, workflow_path, tree): + """V018 -- .value() blocks, so dispatching and resolving in one + comprehension runs the fan-out one call at a time.""" + comprehensions = (ast.ListComp, ast.SetComp, ast.GeneratorExp) + for node in ast.walk(tree): + if not isinstance(node, comprehensions + (ast.DictComp,)): + continue + elements = ( + [node.key, node.value] if isinstance(node, ast.DictComp) else [node.elt] + ) + for element in elements: + for inner in ast.walk(element): + if ( + isinstance(inner, ast.Call) + and isinstance(inner.func, ast.Attribute) + and inner.func.attr == "value" + and isinstance(inner.func.value, ast.Call) + ): + report.error( + "V018", + workflow_path, + node.lineno, + "one comprehension both dispatches a call and resolves " + "it with .value()", + ".value() blocks, so each call completes before the " + "next is dispatched. It does not error -- the fan-out " + "is just silently serial, and with it the reason to be " + "on CanyonOS Core. Dispatch every call first, then resolve: " + "futures = [a.work(i) for i in items] then " + "[f.value() for f in futures].", + ) + return