diff --git a/.claude/skills/porting-to-canyonos-core/SKILL.md b/.claude/skills/porting-to-canyonos-core/SKILL.md deleted file mode 100644 index 3b8f261..0000000 --- a/.claude/skills/porting-to-canyonos-core/SKILL.md +++ /dev/null @@ -1,375 +0,0 @@ ---- -name: porting-to-canyonos-core -description: Ports existing LangChain, LangGraph, CrewAI, AutoGen, and hand-rolled Python agent projects onto CanyonOS Core, whose CLI is `ventis` and whose artifacts live in a `.car` directory. Writes the `.car/config` manifest and declarations, duplicates the source into `.car/app`, writes adapters and the workflow, then validates, builds, deploys and probes. Use when converting, migrating, adapting, packaging, building or deploying an existing agent or multi-agent project onto CanyonOS Core or ventis, when running `ventis build` or `ventis deploy`, or when a `.car` port fails to build, load an agent, or answer a request. ---- - -# Port an agent project to CanyonOS Core - -Requires Python, Docker, and the `ventis` CLI. `validate.py` in this skill needs -Python 3 and `pyyaml`. - -CanyonOS Core is the product name. Its compatibility executable and Python -package remain `ventis`; environment variables and Docker resources retain the -`VENTIS_*` and `ventis-*` prefixes. These are protocol identifiers, not branding -strings. Do not rename them. - -## Port checklist - -Copy this into your response and check items off as you go. Every step below -maps to one line here. - -``` -Port progress: -- [ ] 1. Copy the source into .car/app, rooted at its import root -- [ ] 2. Survey the copy and choose service boundaries -- [ ] 3. Read adapter.md + manifest.md; write declarations, adapters, workflow, config -- [ ] 4. validate.py reports 0 errors -- [ ] 5. ventis build succeeds -- [ ] 6. Every image passes its probes, including the peer import -- [ ] 7. A real request returns through /status -- [ ] 8. Clean up; git status shows nothing outside .car -``` - -Do not skip step 4. Every failure mode it reports survives a green build and a -healthy replica, and then costs a deploy cycle to rediscover. - -## References - -Every reference is linked from here and read whole when its trigger fires. What -differs between the groups is the *kind* of trigger. - -**Before you write.** Triggered by the step, not by a symptom: a porter cannot -look up a rule whose violation builds green and fails in a container. Neither is -optional. - -- [references/adapter.md](references/adapter.md) -- choosing the entrypoint, - bridging async, session state. Read before writing into `.car/app`. -- [references/manifest.md](references/manifest.md) -- the agent yaml, the - complete manifest, and how to build a `requirements` list. Read before writing - into `.car/config`. - -**When the target has this shape.** Triggered by a fact about the source or the -deployment, all three knowable at step 1. - -- [references/packaging.md](references/packaging.md) -- read when a source - import does not resolve from `/app`, the source is nested, or packaging - metadata is involved. -- [references/llm-proxy.md](references/llm-proxy.md) -- read when the target - includes `llm_proxy`. -- [references/ec2.md](references/ec2.md) -- read when any config entry uses - `provider: EC2`. - -**After something failed.** Triggered by a symptom. - -- [references/troubleshooting.md](references/troubleshooting.md) -- read after a - failed build, image probe, deploy, or request; symptom-to-cause tables. -- [references/runtime-contract.md](references/runtime-contract.md) -- read when - a validator finding needs explanation or the runtime mechanism is unclear. - -**For orientation.** - -- [references/example-port.md](references/example-port.md) -- one LangGraph port - end to end: the decisions, the files, and the evidence that closed it. - -## Goal: a self-contained `.car`, and a source tree that never learns about it - -The port lives entirely inside `.car/`, next to the application source and -never inside it: - -```text -.car/config/global_controller.yaml deployment manifest -.car/config/policy.yaml optional access restriction -.car/config/.yaml one callable surface per service -.car/app/ a copy of the application source -.car/app//.py the adapter, written where the code it wraps lives -.car/app//_workflow.py HTTP entry point; calls deploy() -.car/app/pyproject.toml conditional nested-import scaffolding -/ the developer's tree, untouched and unaware -``` - -`.car` has exactly two authored directories: `config/`, which holds every -declaration Canyon owns, and `app/`, the copy that becomes `/app` in every -container. The container keeps the directory structure the application already -had. Write adapters into that copy, in the module the code they wrap already -lives in -- not into new `agents/` and `workflow/` directories. `ventis` -commands run from the application root and read `.car` below it. - -Nothing under `.car` points back out at the application source, and nothing in -the application source points at `.car`. Deleting `.car` returns the project to -exactly where it started; regenerating it touches no file the developer owns. - -The file count follows the deployment. A multi-agent port has one yaml/adapter -pair per service that is worth deploying separately. If a source class in the -copy already satisfies the runtime contract, point its config entry at that -file and do not write an adapter beside it. - -Everything the source already owns—prompts, tools, schemas, parsing, retries, -model clients, and node bodies—is imported from where the copy keeps it. The -port re-expresses only the CanyonOS Core boundary and framework-owned -orchestration. - -## 1. Duplicate the source, then survey it - -Copy the application source into `.car/app/`, preserving its structure. Leave -out only what no container should carry: `.git/`, `.car/` itself, virtualenvs, -caches, build outputs, and `.env` files holding real credentials. - -**Root the copy at the source's import root, which is not always its repository -root.** `/app` is the copy, and without the editable-install capability it is -the only entry on `sys.path`, so a source under `src/` that imports -`from tools import ...` needs the *contents* of `src/` at `.car/app/`. Read the -source's own imports, not its directory names, to decide. Getting it wrong -builds green and answers `No agent loaded` on the first request; -[references/packaging.md](references/packaging.md) works the case through. - -```bash -mkdir -p .car/config -rsync -a --exclude '.git' --exclude '.car' --exclude '.venv' --exclude 'venv' \ - --exclude '__pycache__' --exclude '.env' / .car/app/ -``` - -Every edit from here on is inside `.car`. The application source outside it is -read-only for the rest of the port -- `git status` at the end shows `.car/` and -nothing else. - -Then survey the copy. Identify: - -1. The source entry point and callable input/output. -2. Framework-owned control flow (`StateGraph`, `Crew`, `GroupChat`, routing, - `Send`, `Command`, interrupts). -3. Runtime-injected services nodes read: stores, context, memory, sessions, or - callback managers. -4. Sync versus async boundaries. -5. Imports and declared runtime dependencies. -6. Model provider, credential names, streaming use, and optional `llm_proxy`. -7. Whether independent work fans out and benefits from separate replicas. -8. Whether source imports resolve from `.car/app`, the root that becomes - `/app`. This is the check the validator turns into V031, and it is the one - most likely to survive a green build and a healthy replica. - -Run the validator now, and again after every change until it reports 0 errors. -Execute it; do not read it. Its header detects capabilities directly from the -importable runtime rather than from release history: - -```bash -python /validate.py .car -``` - -If config or agent yaml is malformed, the validator defers to `ventis build`. -Capability-gated findings say which runtime behavior is available. - -## 2. Choose service boundaries - -Start with one service. Split only when it creates independent parallel work or -a distinct resource/replica profile. - -- Keep a ReAct loop together; every turn needs shared message history. -- Hoist supervisor task lists and `Send`-style fan-out into the workflow. -- Do not create a one-replica service with no distinct resource profile merely - to mirror every source graph node. - -Rewrite framework-owned edges as ordinary Python **only where they cross a -service boundary you chose**. A graph whose nodes all land in one agent has no -boundary to express: keep `graph.compile().invoke(...)` and wrap it. Rewriting -it anyway restates control flow the source already had working and buys no -deployment. Import the connected node functions unchanged wherever you do -rewrite. - -Construct runtime-injected service objects from source configuration; do not -invent models, dimensions, stores, or defaults silently. Report any choice the -source does not specify. A service object that holds state across requests -- a -vector store, a memory, a checkpointer built in `__init__` -- makes -`replicas: 1` a correctness requirement rather than a sizing choice, because -the controller picks a replica per call and the others cannot see that state. -Say so in the report; do not leave it implied. - -## 3. Write declarations and adapters - -Read [references/adapter.md](references/adapter.md) before writing an adapter, -and [references/manifest.md](references/manifest.md) before writing -`.car/config`. Neither is optional and neither is triggered by a symptom: every -rule in them builds green and fails inside a container. - -For each service, keep these names aligned: - -```text -config entry name == yaml agent.name == entrypoint class name -``` - -### Adapter - -Write the adapter where the code it wraps already lives. *Which* module -`entrypoint` then names is the decision adapter.md gates: the build writes that -agent's stub over that path in every other image, so it is the one module in -the copy the port destroys. - -The entrypoint exposes a module-level class named exactly `agent.name`. It -constructs with no arguments and its declared methods are synchronous. Read -configuration from the environment in `__init__`. Serialize framework objects -with their own JSON-safe serializer before returning. - -Do not duplicate source prompts, tools, schemas, or model calls. Keep the source -provider and SDK. - -### Workflow - -Expose `main(query: str)` and call `deploy(main, port=...)` at module scope. -Import each agent from its own `entrypoint`, exactly where the source copy keeps -it -- that is the one module the build replaces with a stub: - -```python -from deploy import deploy -from . import # the agent's entrypoint path -``` - -Any other route to the class -- a flat name, a package re-export, a second copy -of the module -- reaches the real class and runs the agent in the workflow -process with none of the deployment behind it. That import needs no rewriting -when the source already imported the agent from there. - -The deployment platform sends `{query: string}` to `/main`. Pack richer input -inside `query`; any additional workflow parameter has a default. - -Dispatch every remote call before resolving any future: - -```python -futures = [agent.work(item=item) for item in items] -results = [json.loads(future.value()) for future in futures] -``` - -Do not fuse dispatch and `.value()` in one comprehension; that silently -serializes fan-out. Do not add an `if __name__ == "__main__":` block: the -workflow is executed with `__name__ == "__main__"` in production. - -### Config - -`entrypoint` and `workflow_file` are relative to `.car/app/` and may not escape -it. `provider` is lowercase `local`, `replicas` is an integer, and -`requirements` is a per-entry list of distribution names -- the source's own -`requirements.txt` is never installed into any image. Omit `policy.yaml` unless -access must be restricted; if present, give it a non-empty `rules` list. -manifest.md carries the complete manifest and how to build each `requirements` -list. - -## Hard rules - -Capitalized **MUST** and **NEVER** are reserved for port-breaking or -source-integrity rules. The owner column states where each is decided. - -| ID | Rule | Owner | -|---|---|---| -| M1 | Entrypoint MUST define a class named exactly `agent.name` | V006 | -| M2 | The class MUST construct with no arguments | V007 | -| M3 | yaml argument names MUST match Python parameter names | V008 | -| M4 | yaml argument types MUST be bare builtins | V010 | -| M5 | Declared adapter methods MUST be synchronous | V009 | -| M6 | Config names MUST match yaml agent names | build | -| M7 | Config names MUST not collide after lowercase normalization | build output | -| M8 | Local provider MUST be lowercase `local` | deploy preflight | -| M9 | `replicas` MUST be an integer | deploy preflight | -| M10 | `requirements` MUST be a list of strings | build | -| M11 | Workflow MUST expose `main(query)`; extra parameters MUST default | V016 | -| M12 | Workflow MUST NEVER contain a main guard | V017 | -| M13 | Fan-out MUST dispatch all calls before resolving any | V018 | -| M14 | A module at the root of the copy MUST not take a runtime flat name | V019 | -| M15 | Workflow MUST import each agent from its own `entrypoint` module | V023 | -| M16 | Policy MUST be absent or contain a non-empty `rules` list | deploy preflight | -| M17 | EC2 entries MUST satisfy the EC2 deployment contract | deploy preflight | -| M18 | NEVER copy source prompts, tools, schemas, or model calls | review | -| M19 | NEVER hardcode or bake a real credential into an image | W003 | -| M20 | NEVER write outside `.car`; the application source stays untouched | `git status` | -| M21 | NEVER swap the source LLM provider | review | -| M22 | NEVER silently move, drop, or reclassify source dependencies | review | -| M23 | Framework control flow MUST be rewritten; source node logic MUST be imported | review | -| M24 | A non-resolving source import MUST have usable packaging metadata at the root of the copy when editable install is supported | V031 | -| M25 | Two agents MUST NOT share one `entrypoint` | V020 | -| M26 | `.car` MUST hold `config/` beside `app/`, the source copy | V032 | -| M27 | `.car/app` MUST be rooted at the source's import root | V031 | -| M28 | `requirements` MUST cover every distribution that entry's import graph reaches, transitively | W006, probe 2 | -| M29 | The entrypoint MUST NOT be a module another image imports for its real contents | probe 3 | -| M30 | The entrypoint's package `__init__.py` MUST NOT re-export from it | V033 | -| M31 | Every segment of the `entrypoint` path MUST be a Python identifier | V034 | -| M32 | The entrypoint's own imports MUST be absolute | V035 | - -## 4. Validate, build, and probe - -Run static preflight, then let the build own build-time validation. Both run -from the application root: - -```bash -python /validate.py .car -ventis build -``` - -Fix every ERROR and re-run the validator until it reports 0 errors before -running `ventis build`. A build that skips this passes, and the port then fails -at `docker run` or on the first request, where the message names a container -rather than the mistake. - -A green build never imports the adapter. Probe in this order: - -```bash -# 1. Runtime startup path. Every image, agent and workflow alike. - docker run --rm ventis- \ - python -c "import local_controller" - -# 2. Agent load path. Name the module after the entrypoint's own path, exactly -# as _load_agent does -- spec_from_file_location('m', ...) sets a __name__ the -# runtime never uses and hides relative-import failures until deploy. - docker run --rm --env-file ventis- \ - python -c "import importlib.util,sys; \ -p='';n=p[:-3]; \ -s=importlib.util.spec_from_file_location(n,p); \ -m=importlib.util.module_from_spec(s);sys.modules[n]=m;s.loader.exec_module(m); \ -m.();print('ok')" - -# 3. Peer-import path, in the workflow image: here the stub stands where the -# entrypoint was and the package around it is real. - docker run --rm ventis- \ - python -c "from import ;print('ok')" -``` - -Probe 2 needs `--env-file` in the ordinary case, not the exceptional one: a -source that builds its client at module scope (`client = Anthropic()`) fails at -import without it, and the SDK raises on a missing key even when the key is a -placeholder pointed at a proxy. - -Probe 3 is the only one that exercises what the workflow container does at -startup. It is what catches a package `__init__` re-export against a stub, and a -distribution the workflow image needs only because the entrypoint's package -siblings import it. Also probe the workflow image with -`python -c "import local_controller"`; it has its own dependency resolve. - -Then deploy, send a representative request, and poll its status: - -```bash -ventis deploy -curl -X POST http://localhost:8080/main \ - -H 'Content-Type: application/json' -d '{"query":""}' -curl http://localhost:8080/status/ -``` - -A successful outer request with a source-level failure still proves the port -reached and returned the source behavior. Record the distinction. - -## 5. Clean up - -After collecting evidence, stop foreground deploy with Ctrl+C and wait for -controller cleanup. Remove exact leftovers if startup crashed. Then remove build -products and exact images from this config: - -```bash -ventis clean -docker image rm ventis- \ - ventis- - -test ! -e .car/stubs && test ! -e .car/grpc_stubs && test ! -e .car/docker_container -docker ps -a --format '{{.Names}}' -``` - -`ventis clean` removes only `.car/stubs/`, `.car/grpc_stubs/`, and -`.car/docker_container/`; it does not remove containers or images. Keep -`.car/config`, `.car/app`, and requested logs or reports. - -Finally, confirm the decoupling held: `git status` outside `.car` reports no -change to any file the developer owns. diff --git a/.claude/skills/porting-to-canyonos-core/validate.py b/.claude/skills/porting-to-canyonos-core/validate.py deleted file mode 100755 index edc21c6..0000000 --- a/.claude/skills/porting-to-canyonos-core/validate.py +++ /dev/null @@ -1,1556 +0,0 @@ -#!/usr/bin/env python3 -"""Preflight the runtime traps that `ventis build` cannot see. - -This deliberately does not duplicate build-time validation such as malformed -YAML, missing entrypoints, or config-to-yaml matching. `ventis build` owns those -checks. This script parses Python without importing it and catches failures that -otherwise stay hidden until a container loads an agent, starts a workflow, or -serves its first request. A replica is not evidence: the controller writes -`healthy` to Redis before `_load_agent` runs. - - python validate.py [artifact_root] [-c config/global_controller.yaml] - [--json] [--strict] - -`artifact_root` is the `.car` directory: `config/` beside `app/`, the copy of -the application source that becomes /app inside every container. - -Exit 1 if any ERROR was reported, 0 otherwise. --strict also fails on warnings. - -Runtime capabilities vary across CanyonOS Core installations. This script probes -the importable `ventis` package directly. A capability-gated check reports -UNAVAILABLE when its behavior cannot be proven. -""" - -import argparse -import ast -import builtins -import json -import os -import re -import sys -from typing import ClassVar - -try: - import yaml -except ImportError: # pragma: no cover - pyyaml is a CanyonOS Core dependency - sys.stderr.write("validate.py needs pyyaml: pip install pyyaml\n") - raise SystemExit(2) from None - - -DEFAULT_CONFIG_PATH = "config/global_controller.yaml" -# ventis/cli.py SOURCE_DIR_NAME -- the duplicated application source. -SOURCE_DIR_NAME = "app" - -# Copied flat into every image over the swept project tree, so a project module -# landing flat under one of these names is overwritten. -# ventis/stub_generator.py generate_docker / generate_workflow_docker. -RUNTIME_FLAT_NAMES = frozenset( - { - "future.py", - "ventis_context.py", - "local_controller.py", - "local_controller_frontend.py", - "redis_client.py", - "grpc_options.py", - "gpu_metrics.py", - "bedrock.py", - "deploy.py", - "session_logging.py", - "workflow_launcher.py", - } -) - -def _base_requirements(): - """What the generator preinstalls, taken from the importable runtime. - - The literals below are a fallback for a machine where `ventis` is not - importable. They are also the only copy of a runtime fact in this file that - nothing checks at run time, and a copied fact rots: the `sweeps_all_files` - probe named a function that never existed and reported `no` for an entire - 45-repository corpus before a porter caught it. Prefer the live values, and - let tests/test_porting_skill_validate.py hold the fallback to them. - """ - agent = [ - "grpcio", - "grpcio-tools", - "redis", - "pyyaml", - "psutil", - "ipdb", - "ipython", - "boto3", - ] - workflow = [*agent, "flask", "sqlalchemy", "psycopg[binary]"] - try: - from ventis import stub_generator - except Exception: # noqa: BLE001 - a broken install must not crash the check - return agent, workflow - return ( - list(getattr(stub_generator, "BASE_AGENT_REQUIREMENTS", agent)), - list(getattr(stub_generator, "BASE_WORKFLOW_REQUIREMENTS", workflow)), - ) - - -BASE_AGENT_REQUIREMENTS, BASE_WORKFLOW_REQUIREMENTS = _base_requirements() -# Import name -> every distribution that provides it. A tuple rather than a -# string because more than one distribution can ship the same import name, and -# reporting a correct declaration as a gap teaches the reader to dismiss W006. -IMPORT_TO_DISTRIBUTION = { - "attr": ("attrs",), - # `import autogen` is shipped by three unrelated distributions: pyautogen - # (Microsoft's original), ag2 (the community continuation), and a package - # literally named autogen. Any of them satisfies the import. - "autogen": ("pyautogen", "ag2", "autogen", "autogen-agentchat"), - "bs4": ("beautifulsoup4",), - "cv2": ("opencv-python",), - "dateutil": ("python-dateutil",), - "dotenv": ("python-dotenv",), - "grpc": ("grpcio",), - "grpc_tools": ("grpcio-tools",), - "jwt": ("pyjwt",), - "PIL": ("pillow",), - "psycopg": ("psycopg",), - "psycopg2": ("psycopg2-binary",), - "pydantic_settings": ("pydantic-settings",), - "sklearn": ("scikit-learn",), - "typing_extensions": ("typing-extensions",), - "yaml": ("pyyaml",), -} -# Import name -> distribution prefix, where the top-level package is shipped by -# a family of distributions rather than one. `import llama_index.llms.openai` -# collapses to `llama_index`, which no correctly scoped requirements list ever -# names: it declares llama-index-core, llama-index-llms-openai and so on. Any -# member of the family satisfies the import. -NAMESPACE_DISTRIBUTIONS = { - "llama_index": "llama-index", -} - -def _stdlib_names(): - """Module names the interpreter provides without any distribution. - - `sys.stdlib_module_names` exists only from 3.10. Below that, derive the set - from the interpreter's own library directory rather than shipping a list - that rots -- without it every `import os` in the copy becomes a W006, and a - check that cries wolf is a check the reader stops reading. - """ - names = getattr(sys, "stdlib_module_names", None) - if names: - return frozenset(names) - found = set(sys.builtin_module_names) - library = os.path.dirname(os.__file__) - try: - entries = os.listdir(library) - except OSError: - return frozenset(found) - for entry in entries: - if entry.endswith(".py"): - found.add(entry[:-3]) - elif "." not in entry and "-" not in entry: - found.add(entry) - return frozenset(found) - - -STDLIB_MODULE_NAMES = _stdlib_names() - -SECRET_PATTERNS = [ - (re.compile(r"sk-[A-Za-z0-9_-]{20,}"), "an OpenAI-style secret key"), - (re.compile(r"AKIA[0-9A-Z]{16}"), "an AWS access key id"), - (re.compile(r"gh[pousr]_[A-Za-z0-9]{20,}"), "a GitHub token"), - (re.compile(r"AIza[0-9A-Za-z_-]{30,}"), "a Google API key"), -] -SECRET_NAME = re.compile(r"(API_KEY|SECRET|TOKEN|PASSWORD|CREDENTIAL)", re.IGNORECASE) - -ERROR = "ERROR" -WARN = "WARN" -INFO = "INFO" - - -# ------------------------------------------------------------------ # -# Capabilities # -# ------------------------------------------------------------------ # -# -# Stable labels for behavior detected from the importable runtime. They contain -# no external development metadata. - -CAPABILITY_SOURCE = { - "env_file": "runtime env-file injection", - "editable_install": "editable project installation", - "sweeps_all_files": "full project-file sweep", -} - - -def probe_capabilities(): - """Ask the importable ventis package what it actually supports.""" - caps = dict.fromkeys(CAPABILITY_SOURCE, False) - caps["ventis"] = False - try: - from ventis import stub_generator - except Exception: # noqa: BLE001 - a broken install must not crash the check - return caps - - caps["ventis"] = True - caps["editable_install"] = hasattr(stub_generator, "_install_step") - caps["sweeps_all_files"] = hasattr(stub_generator, "_sweep_project_files") - - import importlib - - for module_name in ("ventis.controller.utils.env_file", "ventis.utils.env_file"): - try: - module = importlib.import_module(module_name) - except Exception: # noqa: BLE001,S112 - the other path is the live one - continue - if hasattr(module, "resolve_env_file"): - caps["env_file"] = True - break - return caps - - -# ------------------------------------------------------------------ # -# YAML with line numbers # -# ------------------------------------------------------------------ # - - -class LineDict(dict): - """A mapping that remembers where it and each of its keys were written.""" - - line = 0 - key_lines: ClassVar[dict] = {} - - -class LineLoader(yaml.SafeLoader): - pass - - -def _construct_mapping(loader, node): - data = LineDict() - yield data - data.update(loader.construct_mapping(node, deep=False)) - data.line = node.start_mark.line + 1 - data.key_lines = { - key.value: key.start_mark.line + 1 - for key, _ in node.value - if isinstance(key, yaml.ScalarNode) - } - - -LineLoader.add_constructor( - yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, _construct_mapping -) - - -def line_of(mapping, key=None): - """The source line of `key` inside `mapping`, or of the mapping itself.""" - if not isinstance(mapping, LineDict): - return 0 - if key is not None: - return mapping.key_lines.get(key, mapping.line) - return mapping.line - - -def load_yaml(path): - """Parse `path`, returning (data, error). Never raises.""" - try: - with open(path, "r", encoding="utf-8") as handle: - return yaml.load(handle, Loader=LineLoader), None - except Exception as exc: # noqa: BLE001 - any parse failure is a finding - return None, str(exc) - - -# ------------------------------------------------------------------ # -# Findings # -# ------------------------------------------------------------------ # - - -class Report: - def __init__(self, project_dir, capabilities): - self.project_dir = project_dir - self.capabilities = capabilities - self.findings = [] - - def add(self, check, level, path, line, summary, mechanism): - self.findings.append( - { - "check": check, - "level": level, - "path": self.rel(path) if path else "", - "line": line or 0, - "summary": summary, - "mechanism": mechanism, - } - ) - - def error(self, check, path, line, summary, mechanism): - self.add(check, ERROR, path, line, summary, mechanism) - - def warn(self, check, path, line, summary, mechanism): - self.add(check, WARN, path, line, summary, mechanism) - - def unavailable(self, check, summary): - self.add(check, INFO, "", 0, summary, "") - - def rel(self, path): - try: - return os.path.relpath(path, self.project_dir) - except ValueError: - return path - - def counts(self): - errors = sum(1 for f in self.findings if f["level"] == ERROR) - warnings = sum(1 for f in self.findings if f["level"] == WARN) - return errors, warnings - - -# ------------------------------------------------------------------ # -# Python source helpers # -# ------------------------------------------------------------------ # - - -def parse_python(path): - """AST for `path`, or (None, error). The port is never imported.""" - try: - with open(path, "r", encoding="utf-8") as handle: - source = handle.read() - except OSError as exc: - return None, str(exc) - try: - return ast.parse(source, filename=path), None - except SyntaxError as exc: - return None, f"{exc.msg} (line {exc.lineno})" - - -def find_class(tree, name): - for node in tree.body: - if isinstance(node, ast.ClassDef) and node.name == name: - return node - return None - - -def class_methods(class_node): - return { - node.name: node - for node in class_node.body - if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) - } - - -def parameter_names(func_node): - """Every parameter a caller can pass by keyword, minus self.""" - args = func_node.args - positional = [a.arg for a in args.posonlyargs + args.args] - if positional and positional[0] in ("self", "cls"): - positional = positional[1:] - return positional + [a.arg for a in args.kwonlyargs] - - -def required_parameters(func_node): - """Parameters with no default, minus self.""" - args = func_node.args - positional = [a.arg for a in args.posonlyargs + args.args] - if positional and positional[0] in ("self", "cls"): - positional = positional[1:] - if args.defaults: - positional = positional[: len(positional) - len(args.defaults)] - kwonly = [ - arg.arg - for arg, default in zip(args.kwonlyargs, args.kw_defaults) - if default is None - ] - return positional + kwonly - - -def toplevel_import_names(tree): - """Top-level package name of every import in the module, with line numbers.""" - names = {} - for node in ast.walk(tree): - if isinstance(node, ast.Import): - for alias in node.names: - names.setdefault(alias.name.split(".")[0], node.lineno) - elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module: - names.setdefault(node.module.split(".")[0], node.lineno) - return names - - -def dotted_import_names(tree): - """Every absolute import in the module as its full dotted path, with lines.""" - names = {} - for node in ast.walk(tree): - if isinstance(node, ast.Import): - for alias in node.names: - names.setdefault(alias.name, node.lineno) - elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module: - names.setdefault(node.module, node.lineno) - return names - - -def _local_module_files(project_dir, dotted): - """Files inside the copy that `import ` executes, outermost first. - - Python runs every package `__init__.py` on the way down before the leaf - module. That is how an image ends up executing code it never names: the - workflow imports `pkg.agent`, `pkg/__init__.py` runs first, and whatever it - imports runs with it. - """ - parts = dotted.split(".") - found = [] - prefix = project_dir - for depth, part in enumerate(parts): - if os.path.isdir(os.path.join(prefix, part)): - init_path = os.path.join(prefix, part, "__init__.py") - if os.path.isfile(init_path): - found.append(init_path) - prefix = os.path.join(prefix, part) - continue - leaf = os.path.join(prefix, part + ".py") - if depth == len(parts) - 1 and os.path.isfile(leaf): - found.append(leaf) - return found - return found - - -def _relative_import_files(project_dir, path, tree): - """Files a module's own `from .sibling import x` imports execute.""" - root = os.path.realpath(project_dir) - found = [] - for node in ast.walk(tree): - if not isinstance(node, ast.ImportFrom) or not node.level: - continue - base = os.path.dirname(path) - for _ in range(node.level - 1): - base = os.path.dirname(base) - resolved = os.path.realpath(base) - if resolved != root and not resolved.startswith(root + os.sep): - continue - target = os.path.join(base, *(node.module.split(".") if node.module else [])) - candidates = [target + ".py", os.path.join(target, "__init__.py")] - candidates += [ - os.path.join(target, alias.name + ".py") for alias in node.names - ] - candidates += [ - os.path.join(target, alias.name, "__init__.py") for alias in node.names - ] - found += [c for c in candidates if os.path.isfile(c)] - return found - - -def reachable_imports(project_dir, root_path): - """Every third-party import the image executes from `root_path`, transitively. - - An image runs far more than the file the config names. `tools/parser.py` is - one hop from an entrypoint and its pdfplumber import is invisible to an AST - walk of the entrypoint alone; a package `__init__.py` three hops up drags in - graphql-core. Both surface only as a ModuleNotFoundError deep inside an - import chain at agent load, long after a green build. - - Returns {dotted import name: (file that imports it, line)} for names that do - not resolve inside the copy. - """ - external = {} - seen = set() - queue = [os.path.realpath(root_path)] - while queue: - path = queue.pop() - if path in seen or not os.path.isfile(path): - continue - seen.add(path) - tree, _ = parse_python(path) - if tree is None: - continue - for dotted, lineno in dotted_import_names(tree).items(): - local = _local_module_files(project_dir, dotted) - if local: - queue += [os.path.realpath(p) for p in local] - else: - external.setdefault(dotted, (path, lineno)) - queue += [ - os.path.realpath(p) - for p in _relative_import_files(project_dir, path, tree) - ] - return external - - -# ------------------------------------------------------------------ # -# V006-V010 adapter failures hidden by _load_agent # -# ------------------------------------------------------------------ # - -BUILTIN_TYPE_NAMES = frozenset( - name - for name in dir(builtins) - if isinstance(getattr(builtins, name), type) and not name[0].isupper() -) - - -def check_adapter(report, agent_yaml_path, agent_block, entry, project_dir): - """V006 V007 V008 V009 V010.""" - name = agent_block["name"] - functions = agent_block.get("functions") or [] - check_argument_types(report, agent_yaml_path, functions) - - entrypoint = entry.get("entrypoint") - if not entrypoint: - return - entrypoint_path = os.path.join(project_dir, entrypoint) - if not os.path.isfile(entrypoint_path): - return - - tree, error = parse_python(entrypoint_path) - if tree is None: - report.error( - "V006", - entrypoint_path, - 0, - f"the entrypoint does not parse: {error}", - "_load_agent exec_module's it and swallows the exception; the first " - "request answers 'No agent loaded'.", - ) - return - - class_node = find_class(tree, name) - if class_node is None: - classes = [n.name for n in tree.body if isinstance(n, ast.ClassDef)] - found = ", ".join(classes) if classes else "no classes at all" - report.error( - "V006", - entrypoint_path, - 1, - f"no class named `{name}` at module level (found: {found})", - "_load_agent does getattr(module, VENTIS_AGENT_NAME) and swallows " - "the AttributeError. The class name must equal agent.name exactly.", - ) - return - - methods = class_methods(class_node) - check_constructor(report, entrypoint_path, name, methods) - - for func in functions: - if not isinstance(func, dict) or not isinstance(func.get("name"), str): - continue - check_method(report, entrypoint_path, agent_yaml_path, name, func, methods) - - -def check_argument_types(report, agent_yaml_path, functions): - """V010 -- the type string is pasted into an ast.Name, never checked.""" - for func in functions: - if not isinstance(func, dict): - continue - for arg in func.get("arguments") or []: - if not isinstance(arg, dict) or "type" not in arg: - continue - declared = arg.get("type") - if not isinstance(declared, str): - continue # stub generation reports malformed type values - if declared in BUILTIN_TYPE_NAMES: - continue - report.error( - "V010", - agent_yaml_path, - line_of(arg, "type"), - f"`type: {declared}` is not a builtin", - "stub_generator pastes it verbatim into the generated " - "annotation, and the stub module imports only Future and " - "inspect. Anything else raises NameError when the stub is " - "imported -- after a green build. Use str int float bool dict " - "list.", - ) - - -def check_constructor(report, entrypoint_path, name, methods): - """V007 -- _load_agent calls agent_class() with no arguments.""" - init = methods.get("__init__") - if init is None: - return - required = required_parameters(init) - if required: - report.error( - "V007", - entrypoint_path, - init.lineno, - f"`{name}.__init__` requires {', '.join(required)}", - "_load_agent calls agent_class() with no arguments; the TypeError is " - "swallowed and the first request answers 'No agent loaded'. Read " - "configuration from the environment inside __init__ instead.", - ) - - -def check_method(report, entrypoint_path, agent_yaml_path, class_name, func, methods): - """V008 V009.""" - func_name = func["name"] - method = methods.get(func_name) - if method is None: - report.error( - "V008", - entrypoint_path, - 0, - f"`{class_name}` has no method `{func_name}`", - "The yaml declares it, so callers get a stub for it; the controller " - f"then answers \"Agent {class_name} has no method '{func_name}'\".", - ) - return - - # V009 -- nothing on the execution path awaits. - if isinstance(method, ast.AsyncFunctionDef): - report.error( - "V009", - entrypoint_path, - method.lineno, - f"`{class_name}.{func_name}` is `async def`", - "The executor calls method(**args) with no await, so Redis receives " - "''. Keep the signature synchronous and call " - "asyncio.run(...) inside the body.", - ) - - # V008 -- the controller calls method(**args) with the yaml's names. - declared = [ - arg["name"] - for arg in func.get("arguments") or [] - if isinstance(arg, dict) and isinstance(arg.get("name"), str) - ] - actual = parameter_names(method) - required = required_parameters(method) - - missing = [arg for arg in declared if arg not in actual] - if missing: - report.error( - "V008", - entrypoint_path, - method.lineno, - f"`{class_name}.{func_name}` has no parameter " - f"{', '.join(repr(m) for m in missing)}, but the yaml declares it", - "LocalController does method(**args) with the yaml's argument names. " - "A mismatch is TypeError: unexpected keyword argument, at request " - f"time. See {os.path.basename(agent_yaml_path)}.", - ) - - unfilled = [arg for arg in required if arg not in declared] - if unfilled: - report.error( - "V008", - entrypoint_path, - method.lineno, - f"`{class_name}.{func_name}` requires {', '.join(unfilled)}, which " - "the yaml does not declare", - "Only declared arguments are ever sent, and the generated stub gives " - "none of them a default. Declare them in the yaml or default them " - "in the signature.", - ) - - - -# ------------------------------------------------------------------ # -# V016-V018 the workflow # -# ------------------------------------------------------------------ # - - -def check_stub_imports(report, workflow_path, tree, stub_modules): - """V023 -- the workflow must import each agent from its own entrypoint module. - - The build writes a stub over exactly one path: the agent's `entrypoint` - inside the source copy. An import that reaches the class any other way -- - flat, through a package re-export, or from a second copy of the module -- - resolves to the real class instead, and the workflow runs the agent - in-process with none of the deployment behind it. The class name is another - trap: `ventis build` prints one with a `Stub` suffix that it never writes. - """ - for node in ast.walk(tree): - if not isinstance(node, ast.ImportFrom) or not node.module: - continue - for alias in node.names: - name = alias.name - base = name.removesuffix("Stub") - expected = stub_modules.get(base) - if expected is None: - continue - if name.endswith("Stub"): - report.error( - "V023", workflow_path, node.lineno, - f"`{name}` is the name the build prints, not the class it " - f"writes", - "generate_stub sets class_name = agent_config['name'] and " - "then recomputes it with a 'Stub' suffix for the log line " - "only. The message names a class that does not exist; the " - f"class is `{base}`.", - ) - elif node.module != expected: - report.error( - "V023", workflow_path, node.lineno, - f"`from {node.module} import {name}` -- the stub for {name} " - f"is written to {expected.replace('.', '/')}.py", - "The build replaces the module at the agent's entrypoint " - "and nothing else, so this import reaches the real class " - "and runs the agent in this process instead of over gRPC. " - f"Import it from `{expected}`, where the source already " - "keeps it.", - ) - - -def check_workflow(report, workflow_path, stub_modules=None): - """V016 V017 V018 V023.""" - tree, error = parse_python(workflow_path) - if tree is None: - report.error("V016", workflow_path, 0, f"does not parse: {error}", "") - return - - main = None - for node in tree.body: - if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and ( - node.name == "main" - ): - main = node - break - - if main is None: - defined = [ - n.name - for n in tree.body - if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef)) - ] - found = ", ".join(defined) if defined else "no top-level functions" - report.error( - "V016", - workflow_path, - 1, - f"no top-level function named `main` (found: {found})", - "CanyonOS Core serves POST /, but the deployment platform's " - "test endpoint posts to a hardcoded /main. A differently named " - "workflow builds, deploys and stays unreachable -- 404, container " - "healthy.", - ) - else: - check_main_signature(report, workflow_path, main) - - if stub_modules: - check_stub_imports(report, workflow_path, tree, stub_modules) - - # V016 -- deploy() is what starts Flask. - if not any( - isinstance(node, ast.Call) - and isinstance(node.func, ast.Name) - and node.func.id == "deploy" - for node in ast.walk(tree) - ): - report.error( - "V016", - workflow_path, - 1, - "the workflow never calls `deploy(...)`", - "workflow_launcher.py exec's this file and nothing else starts the " - "HTTP server; the container comes up serving nothing.", - ) - - check_main_guard(report, workflow_path, tree) - check_fused_fanout(report, workflow_path, tree) - - -def check_main_signature(report, workflow_path, main): - """V016 -- the platform sends exactly {"query": ...}.""" - if isinstance(main, ast.AsyncFunctionDef): - report.error( - "V016", - workflow_path, - main.lineno, - "`main` is `async def`", - "deploy() calls workflow_fn(**kwargs) on a Flask worker thread with " - "no await; the response body would be a coroutine repr.", - ) - params = parameter_names(main) - if not params: - report.error( - "V016", - workflow_path, - main.lineno, - "`main` takes no arguments", - 'The platform posts {"query": "..."} and deploy() splats the ' - "body in as kwargs -- TypeError on every request.", - ) - return - if params[0] != "query": - report.error( - "V016", - workflow_path, - main.lineno, - f"`main`'s first parameter is `{params[0]}`, not `query`", - "The platform's body schema is strictly validated as " - "{query: string}; any other key is rejected with 400 in the control " - "plane, before the request reaches the host.", - ) - extra = [p for p in required_parameters(main) if p != "query"] - if extra: - report.error( - "V016", - workflow_path, - main.lineno, - f"`main` requires {', '.join(extra)} beyond `query`", - "Only `query` is ever sent, so every other parameter needs a " - "default or the call raises on every request. Pack richer input " - "into `query`.", - ) - - -def check_main_guard(report, workflow_path, tree): - """V017 -- the workflow is exec'd, so __name__ == "__main__".""" - for node in tree.body: - if not isinstance(node, ast.If): - continue - test = node.test - if ( - isinstance(test, ast.Compare) - and isinstance(test.left, ast.Name) - and test.left.id == "__name__" - and any( - isinstance(c, ast.Constant) and c.value == "__main__" - for c in test.comparators - ) - ): - report.error( - "V017", - workflow_path, - node.lineno, - '`if __name__ == "__main__":` block in the workflow', - "workflow_launcher.py runs exec(open().read()), so " - "__name__ IS '__main__' here and this block executes in " - "production, at container start.", - ) - - -def check_fused_fanout(report, workflow_path, tree): - """V018 -- .value() blocks, so dispatching and resolving in one - comprehension runs the fan-out one call at a time.""" - comprehensions = (ast.ListComp, ast.SetComp, ast.GeneratorExp) - for node in ast.walk(tree): - if not isinstance(node, comprehensions + (ast.DictComp,)): - continue - elements = ( - [node.key, node.value] if isinstance(node, ast.DictComp) else [node.elt] - ) - for element in elements: - for inner in ast.walk(element): - if ( - isinstance(inner, ast.Call) - and isinstance(inner.func, ast.Attribute) - and inner.func.attr == "value" - and isinstance(inner.func.value, ast.Call) - ): - report.error( - "V018", - workflow_path, - node.lineno, - "one comprehension both dispatches a call and resolves " - "it with .value()", - ".value() blocks, so each call completes before the " - "next is dispatched. It does not error -- the fan-out " - "is just silently serial, and with it the reason to be " - "on CanyonOS Core. Dispatch every call first, then resolve: " - "futures = [a.work(i) for i in items] then " - "[f.value() for f in futures].", - ) - return - - -# ------------------------------------------------------------------ # -# V019-V020 what the copy order overwrites # -# ------------------------------------------------------------------ # - - -def check_flat_collisions(report, source_dir, entrypoints): - """V019 V020 -- later copies land on earlier ones at the context root.""" - for entry in sorted(os.listdir(source_dir)): - path = os.path.join(source_dir, entry) - if not os.path.isfile(path) or not entry.endswith(".py"): - continue - - # V019 -- the shared runtime is copied flat, after the project sweep. - if entry in RUNTIME_FLAT_NAMES: - report.error( - "V019", - path, - 1, - f"a project module named `{entry}` sits at the root of the " - "source copy", - "The shared CanyonOS Core runtime is copied flat into the image after " - "the project sweep, so this file is overwritten by CanyonOS Core's own " - f"{entry}. Rename it or move it into a package directory.", - ) - - # V020 -- one module cannot be the entrypoint of two agents. - owners = {} - for name, entrypoint in entrypoints: - owners.setdefault(entrypoint, []).append(name) - for entrypoint, names in sorted(owners.items()): - if len(names) < 2: - continue - report.error( - "V020", - os.path.join(source_dir, entrypoint), - 1, - f"{' and '.join(sorted(names))} both declare `{entrypoint}` as their " - "entrypoint", - "Each agent's stub is written over its own entrypoint, so the two " - "land on one path and the last one built wins. Every caller then " - "reaches whichever agent that was. Give each agent its own module.", - ) - - -# ------------------------------------------------------------------ # -# V030-V031 capability-gated rules # -# ------------------------------------------------------------------ # - - -def check_env_file(report, config, config_path, artifact_dir): - """V030 -- gated on detected env-file injection support.""" - declared = config.get("env_file") - supported = report.capabilities.get("env_file") - - if not supported: - if declared: - report.error( - "V030", - config_path, - line_of(config, "env_file"), - f"`env_file: {declared}` is set, but this CanyonOS Core never reads it", - "No resolve_env_file in the importable ventis package, so the " - "key is silently dropped and the container answers a provider " - "credential error on the first request. This port requires the " - "`env_file` runtime capability.", - ) - else: - report.unavailable( - "V030", - "env_file is not supported by the importable `ventis` runtime. " - "Credentials have no declared path into a container on this tree.", - ) - return - - if not declared: - report.warn( - "V030", - config_path, - line_of(config), - "no `env_file:` in the config", - "Only runtime-managed VENTIS_* variables are guaranteed without it. " - "If the source reads credentials from the environment, the first " - "request fails on a provider error.", - ) - return - - # Path existence and readability are deploy-preflight checks. Do not - # duplicate them here. - - -def check_import_root(report, source_dir, entrypoint_paths): - """V031 -- gated on detected editable-install support.""" - supported = report.capabilities.get("editable_install") - has_metadata = any( - os.path.isfile(os.path.join(source_dir, name)) - for name in ("pyproject.toml", "setup.py", "setup.cfg") - ) - - non_flat = [] - for path in entrypoint_paths: - tree, _ = parse_python(path) - if tree is None: - continue - for name, lineno in toplevel_import_names(tree).items(): - if f"{name}.py" in RUNTIME_FLAT_NAMES: - continue - if _resolves_flat(source_dir, name): - continue - location = _resolves_nested(source_dir, name) - if location: - non_flat.append((path, lineno, name, location)) - - if not supported: - report.unavailable( - "V031", - "the editable install (`-e .`) is not supported by the importable " - "`ventis` runtime. Only names rooted at /app import inside a container.", - ) - for path, lineno, name, location in non_flat: - report.error( - "V031", - path, - lineno, - f"`import {name}` resolves to {location}, which is not at the " - "root of the source copy", - "sys.path[0] is /app and this CanyonOS Core runs no editable install, " - "so only modules swept to the root import. The adapter raises " - "ModuleNotFoundError inside _load_agent and the first request " - "answers 'No agent loaded'.", - ) - return - - if non_flat and not has_metadata: - for path, lineno, name, location in non_flat: - report.error( - "V031", - path, - lineno, - f"`import {name}` resolves to {location}, and the source copy's " - "root has no packaging metadata", - "A pyproject.toml, setup.py or setup.cfg at the root of the " - "source copy is what adds `-e .`; metadata nested deeper in the " - "tree is ignored. Add minimal root metadata pointing at the " - "existing package directory. Without it the install is skipped " - "silently.", - ) - - -# ------------------------------------------------------------------ # -# V033-V035 traps set by where the entrypoint sits # -# ------------------------------------------------------------------ # - - -def check_entrypoint_module(report, source_dir, name, entrypoint): - """V033 V034 V035. - - Two runtime facts collide here. The build writes this agent's stub over - `entrypoint` in every image except this agent's own, and the controller - loads the real file by path rather than by import. Each breaks a module - layout that is correct everywhere else in Python. - """ - path = os.path.join(source_dir, entrypoint) - if not os.path.isfile(path): - return - - segments = os.path.splitext(entrypoint)[0].replace("\\", "/").split("/") - invalid = [part for part in segments if not part.isidentifier()] - if invalid: - report.error( - "V034", - path, - 0, - f"`{invalid[0]}` in the entrypoint path is not a Python identifier", - "The controller loads the entrypoint by file path, so this file runs " - "-- but the workflow has to import the class from " - f"`{module_path(entrypoint)}` (V023), and that is a SyntaxError, not " - "an ImportError. Rename the file inside the copy, or point " - "`entrypoint` at a normally-named sibling that loads this file by " - "path and re-exposes the class.", - ) - - tree, _ = parse_python(path) - if tree is not None: - for node in ast.walk(tree): - if isinstance(node, ast.ImportFrom) and node.level: - spelling = "." * node.level + (node.module or "") - report.error( - "V035", - path, - node.lineno, - f"the entrypoint's own `from {spelling} import ...` is relative", - "_load_agent loads this file with spec_from_file_location(" - "VENTIS_AGENT_FILE.replace('.py', ''), path). That name keeps " - "the entrypoint's directory separator, so it has no parent " - "package and __package__ is empty: every relative import in " - "this file raises 'attempted relative import with no known " - "parent package' at agent load, behind 'No agent loaded'. " - "Make this file's own top-level imports absolute; modules it " - "imports may keep theirs.", - ) - break - - directory = os.path.dirname(entrypoint) - if not directory: - return - init_path = os.path.join(source_dir, directory, "__init__.py") - if not os.path.isfile(init_path): - return - module = os.path.splitext(os.path.basename(entrypoint))[0] - package = directory.replace("\\", "/").replace("/", ".") - init_tree, _ = parse_python(init_path) - if init_tree is None: - return - for node in ast.walk(init_tree): - if not isinstance(node, ast.ImportFrom): - continue - target = node.module or "" - hit = target == module if node.level else target in ( - module, - f"{package}.{module}", - ) - if not hit and node.level and not node.module: - hit = any(alias.name == module for alias in node.names) - if not hit: - continue - report.error( - "V033", - init_path, - node.lineno, - f"`{package}/__init__.py` re-exports from `{module}`, the entrypoint " - f"for {name}", - "Python runs a package's __init__.py before any of its submodules, " - "and in every image except this agent's own the module at the " - "entrypoint is the generated stub, which defines the agent class and " - f"nothing else. Any peer image that imports anything from `{package}` " - "-- the workflow importing the agent class included -- re-runs this " - "re-export against the stub and dies at container startup with " - f"ImportError. Point `entrypoint` at a module `{package}/__init__.py` " - "does not re-export from; add one that imports the real module if " - "every existing module is re-exported.", - ) - break - - -def _resolves_flat(project_dir, name): - """Whether Python can resolve `name` with /app as its import root. - - A directory does not need __init__.py: PEP 420 namespace packages resolve - from sys.path just like regular packages. - """ - return os.path.isfile(os.path.join(project_dir, f"{name}.py")) or os.path.isdir( - os.path.join(project_dir, name) - ) - - -def _resolves_nested(project_dir, name): - """Where below /app `name` lives but cannot resolve as a top-level name.""" - for root, dirs, files in os.walk(project_dir): - dirs[:] = [d for d in dirs if not d.startswith(".") and d != "__pycache__"] - if root == project_dir: - continue - if f"{name}.py" in files: - return os.path.relpath(os.path.join(root, f"{name}.py"), project_dir) - if name in dirs: - return os.path.relpath(os.path.join(root, name), project_dir) - return None - - -# ------------------------------------------------------------------ # -# W003, W006 secrets and imports a green build does not reject # -# ------------------------------------------------------------------ # - - -def check_secrets(report, port_paths): - """W003 -- env_file is the way in; nothing else is.""" - for path in port_paths: - try: - with open(path, "r", encoding="utf-8") as handle: - lines = handle.read().splitlines() - except OSError: - continue - for number, line in enumerate(lines, start=1): - for pattern, description in SECRET_PATTERNS: - if pattern.search(line): - report.warn( - "W003", - path, - number, - f"this line looks like {description}", - "Never put a secret in the source tree or the build " - "context. The build sweeps the project into every " - "image.", - ) - break - - tree, _ = parse_python(path) - if tree is None: - continue - for node in ast.walk(tree): - if not isinstance(node, ast.Assign): - continue - if not ( - isinstance(node.value, ast.Constant) - and isinstance(node.value.value, str) - and node.value.value.strip() - ): - continue - for target in node.targets: - if isinstance(target, ast.Name) and SECRET_NAME.search(target.id): - report.warn( - "W003", - path, - node.lineno, - f"`{target.id}` is assigned a literal string", - "Read it from the environment instead; the build sweeps " - "this file into every image.", - ) - - -def _pyproject_dependencies(project_dir): - """What `-e .` installs alongside `requirements:`, or None if unreadable. - - None and the empty set mean different things here: empty means the project - declares no dependencies, None means we could not find out -- a setup.py, or - a tomllib this interpreter does not have. The caller must not treat the - second as the first, or it warns about imports the install would satisfy. - """ - path = os.path.join(project_dir, "pyproject.toml") - if not os.path.isfile(path): - return None - try: - import tomllib - except ImportError: # < 3.11 - return None - try: - with open(path, "rb") as handle: - data = tomllib.load(handle) - except Exception: # noqa: BLE001 - malformed metadata is uv's error to give - return None - deps = (data.get("project") or {}).get("dependencies") - if not isinstance(deps, list): - return None - return {_normalize_distribution(d) for d in deps if isinstance(d, str)} - - -def check_requirements_coverage( - report, project_dir, entry, root_path, config_path, base_requirements -): - """W006 -- an import the container cannot satisfy. - - Walks the whole import graph the image executes from `root_path`, not just - that one file: a distribution reached through a local module or a package - __init__ is exactly as missing, and exactly as invisible until the container - starts. - """ - declared = { - _normalize_distribution(item) - for item in (entry.get("requirements") or []) - if isinstance(item, str) - } - - # Where the editable install exists, `-e .` resolves the project's own - # [project.dependencies] in the same pass as `requirements:`. Warning about - # those is a false positive, and a false warning about a dependency is worse - # than none: it teaches the reader to dismiss this check. - editable = report.capabilities.get("editable_install") - metadata = any( - os.path.isfile(os.path.join(project_dir, name)) - for name in ("pyproject.toml", "setup.py", "setup.cfg") - ) - unreadable_metadata = False - if editable and metadata: - project_deps = _pyproject_dependencies(project_dir) - if project_deps is None: - unreadable_metadata = True - else: - declared |= project_deps - base = {_normalize_distribution(item) for item in base_requirements} - satisfied = base | declared - - external = reachable_imports(project_dir, root_path) - for dotted, (where, lineno) in sorted(external.items()): - name = dotted.split(".")[0] - if name in STDLIB_MODULE_NAMES or name == "ventis": - continue - # Provided by the image itself: the shared runtime is copied flat over - # the swept tree. A stub is not listed here -- it replaces a module the - # source copy already carries, so the tree checks below cover it. - if f"{name}.py" in RUNTIME_FLAT_NAMES: - continue - if _resolves_flat(project_dir, name) or _resolves_nested(project_dir, name): - continue - prefix = NAMESPACE_DISTRIBUTIONS.get(name) - if prefix and any( - item == prefix or item.startswith(prefix + "-") for item in declared - ): - continue - if _candidate_distributions(name) & satisfied: - continue - if unreadable_metadata: - mechanism = ( - "The container installs the base list, `requirements:`, and -- " - "since this project declares packaging metadata -- whatever " - "`-e .` resolves from it. That metadata could not be read here, " - f"so if it already requires `{name}` this line is noise; " - "otherwise it is a ModuleNotFoundError inside _load_agent and " - "'No agent loaded' on the first request." - ) - else: - mechanism = ( - "The container installs the base list plus `requirements:` and " - "nothing else, so this is a ModuleNotFoundError inside " - "_load_agent and 'No agent loaded' on the first request. If the " - f"distribution is named something other than `{name}`, declare " - f"that name in {report.rel(config_path)}." - ) - if os.path.realpath(where) != os.path.realpath(root_path): - mechanism += ( - f" This image never names `{name}` in {report.rel(root_path)}; " - f"it runs {report.rel(where)} on the way there, and that module " - "needs it." - ) - report.warn( - "W006", - where, - lineno, - f"`import {dotted}` is in neither the runtime's base list nor " - f"{entry.get('name') or 'this entry'}'s `requirements:`", - mechanism, - ) - - -def _candidate_distributions(name): - """Every distribution name that would satisfy `import `.""" - return { - _normalize_distribution(item) - for item in IMPORT_TO_DISTRIBUTION.get(name, (name,)) - } - - -def _normalize_distribution(name): - return re.split(r"[<>=!\[;\s]", name.strip().lower(), maxsplit=1)[0].replace( - "_", "-" - ) - - -# ------------------------------------------------------------------ # -# Driver # -# ------------------------------------------------------------------ # - - -def find_agent_declarations(config_dir): - """Map agent name -> declaration, for every declaration in `config/`. - - Mirrors ventis/cli.py: declarations sit in `config/` beside the manifest, - which -- like `policy.yaml` -- carries no top-level `agent.name` and so - drops out here. - """ - import glob - - declarations = {} - for path in sorted(glob.glob(os.path.join(config_dir, "*.yaml"))): - data, error = load_yaml(path) - if error is not None or not isinstance(data, dict): - continue - agent = data.get("agent") - name = agent.get("name") if isinstance(agent, dict) else None - if isinstance(name, str) and name: - declarations[name] = (path, agent) - return declarations - - -def module_path(entrypoint): - """Dotted module name an entrypoint has inside the container.""" - return os.path.splitext(entrypoint)[0].replace("\\", "/").replace("/", ".") - - -def validate(artifact_dir, config_path, capabilities): - """Inspect only failures hidden behind a successful image build.""" - report = Report(artifact_dir, capabilities) - - # The build owns config/YAML syntax and shape validation. We read only enough - # valid structure to locate code for the deeper checks below. - config, error = load_yaml(config_path) - if error is not None or not isinstance(config, dict): - report.unavailable( - "BUILD", - "runtime preflight skipped because the config cannot be read; " - "ventis build owns and reports this error.", - ) - return report - - source_dir = os.path.join(artifact_dir, SOURCE_DIR_NAME) - if not os.path.isdir(source_dir): - report.error( - "V032", - artifact_dir, - 0, - f"no `{SOURCE_DIR_NAME}/` beside `config/`", - "The artifact root holds the application source it deploys: " - f"`{SOURCE_DIR_NAME}/` is the copy that becomes /app, and every " - "entrypoint is relative to it. Without it the port has nothing to " - "build and nothing to keep it decoupled from the developer's tree.", - ) - return report - - agents_by_name = find_agent_declarations(os.path.dirname(config_path)) - - entries = config.get("agents") - if not isinstance(entries, list): - report.unavailable( - "BUILD", - "runtime preflight skipped because `agents:` is not a list; " - "ventis build owns and reports this error.", - ) - return report - - entrypoints = [] - for entry in entries: - if not isinstance(entry, dict) or entry.get("type", "agent") == "workflow": - continue - name = entry.get("name") - entrypoint = entry.get("entrypoint") - if isinstance(entrypoint, str): - entrypoints.append((name, entrypoint)) - if name in agents_by_name: - yaml_path, agent_block = agents_by_name[name] - check_adapter(report, yaml_path, agent_block, entry, source_dir) - entrypoint_path = os.path.join(source_dir, entrypoint or "") - if isinstance(entrypoint, str) and os.path.isfile(entrypoint_path): - check_entrypoint_module(report, source_dir, name, entrypoint) - check_requirements_coverage( - report, - source_dir, - entry, - entrypoint_path, - config_path, - BASE_AGENT_REQUIREMENTS, - ) - - # Where each agent's stub is written, and therefore the only import that - # reaches it over gRPC. - stub_modules = { - name: module_path(entrypoint) - for name, entrypoint in entrypoints - if name in agents_by_name - } - - for entry in entries: - if not isinstance(entry, dict) or entry.get("type", "agent") != "workflow": - continue - workflow_file = entry.get("workflow_file") - if not isinstance(workflow_file, str): - continue - workflow_path = os.path.join(source_dir, workflow_file) - if os.path.isfile(workflow_path): - check_workflow(report, workflow_path, stub_modules) - # The workflow image installs its own list. A module it imports for - # a helper drags that module's dependencies in even though the - # workflow makes no model call of its own. - check_requirements_coverage( - report, - source_dir, - entry, - workflow_path, - config_path, - BASE_WORKFLOW_REQUIREMENTS, - ) - - # These survive a green build and otherwise surface only in a container or - # on its first request. - check_flat_collisions(report, source_dir, entrypoints) - check_env_file(report, config, config_path, artifact_dir) - - entrypoint_paths = [ - os.path.join(source_dir, e) - for _, e in entrypoints - if os.path.isfile(os.path.join(source_dir, e)) - ] - check_import_root(report, source_dir, entrypoint_paths) - - port_paths = list(entrypoint_paths) - for entry in entries: - if isinstance(entry, dict) and isinstance(entry.get("workflow_file"), str): - candidate = os.path.join(source_dir, entry["workflow_file"]) - if os.path.isfile(candidate): - port_paths.append(candidate) - - # Secret detection remains because a green image build would permanently - # bake the credential into every image. - check_secrets(report, port_paths) - return report - - -# ------------------------------------------------------------------ # -# Output # -# ------------------------------------------------------------------ # - -LEVEL_ORDER = {ERROR: 0, WARN: 1, INFO: 2} - - -def _wrap(text, width, indent): - words = text.split() - lines = [] - current = "" - for word in words: - candidate = f"{current} {word}".strip() - if len(candidate) + len(indent) > width and current: - lines.append(indent + current) - current = word - else: - current = candidate - if current: - lines.append(indent + current) - return lines - - -def print_report(report, artifact_root): - caps = report.capabilities - if not caps.get("ventis"): - print("ventis is not importable here -- capability-gated rules are") - print("reported UNAVAILABLE rather than checked.\n") - else: - print("CanyonOS Core capabilities detected:") - for key, source in CAPABILITY_SOURCE.items(): - mark = "yes" if caps.get(key) else "no " - print(f" {mark} {key:<22} {source}") - print() - - findings = sorted( - report.findings, - key=lambda f: (LEVEL_ORDER[f["level"]], f["check"], f["path"], f["line"]), - ) - for finding in findings: - where = finding["path"] - if where and finding["line"]: - where = f"{where}:{finding['line']}" - header = f"{finding['check']} {finding['level']:<5}" - print(f"{header} {where}" if where else header) - for line in _wrap(finding["summary"], 78, " "): - print(line) - if finding["mechanism"]: - for line in _wrap(finding["mechanism"], 78, " "): - print(line) - print() - - errors, warnings = report.counts() - if not findings: - print(f"{artifact_root}: clean.") - return - print(f"{errors} error(s), {warnings} warning(s).") - - -def main(argv=None): - parser = argparse.ArgumentParser( - description="Check a CanyonOS Core port against the rules in SKILL.md." - ) - parser.add_argument( - "artifact_root", - nargs="?", - default=".", - help="the .car directory holding config/ and app/ (default: the cwd)", - ) - parser.add_argument( - "-c", - "--config", - default=DEFAULT_CONFIG_PATH, - help=f"config path relative to artifact_root (default: {DEFAULT_CONFIG_PATH})", - ) - parser.add_argument("--json", action="store_true", help="emit findings as JSON") - parser.add_argument( - "--strict", action="store_true", help="fail on warnings as well as errors" - ) - args = parser.parse_args(argv) - - artifact_root = os.path.abspath(args.artifact_root) - config_path = ( - args.config - if os.path.isabs(args.config) - else os.path.join(artifact_root, args.config) - ) - - capabilities = probe_capabilities() - report = validate(artifact_root, config_path, capabilities) - errors, warnings = report.counts() - - if args.json: - print( - json.dumps( - { - "artifact_root": artifact_root, - "capabilities": capabilities, - "errors": errors, - "warnings": warnings, - "findings": report.findings, - }, - indent=2, - ) - ) - else: - print_report(report, report.rel(artifact_root) or artifact_root) - - if errors or (args.strict and warnings): - return 1 - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/.claude/skills/porting-to-canyonos/SKILL.md b/.claude/skills/porting-to-canyonos/SKILL.md new file mode 100644 index 0000000..1dcfb8f --- /dev/null +++ b/.claude/skills/porting-to-canyonos/SKILL.md @@ -0,0 +1,258 @@ +--- +name: porting-to-canyonos +description: Ports existing LangChain, LangGraph, CrewAI, AutoGen, and hand-rolled Python agent projects onto CanyonOS Core, whose CLI is `canyonos` and whose artifacts live in a `.car` directory. Writes `.car/config`, copies source into `.car/app`, writes adapters and the workflow, and validates the port. Stops after validation and asks before running `canyonos deploy`, which performs both build and deployment. Use when converting, migrating, adapting, packaging, validating, or deploying an existing agent or multi-agent project onto CanyonOS Core, or when a `.car` port fails validation, build, load, or deployment. +--- + +# Port an agent project to CanyonOS Core + +Requires Python, Docker, and the `canyonos` CLI. `prepare.py` uses only the +Python standard library; `validate.py` needs Python 3 and `pyyaml`. + +CanyonOS Core is the product name and `canyonos` is its user-facing CLI. The +internal compatibility Python package, environment variables, and Docker +resources retain the `ventis`, `VENTIS_*`, and `ventis-*` names. These are +protocol identifiers, not CLI instructions or branding strings. Do not rename +them, and do not tell users to run the obsolete `ventis` CLI. + +## Port checklist + +Copy this into your response and check items off as you go. Every step below +maps to one line here. + +``` +Port progress: +- [ ] 1. Choose the import root; run prepare.py to create .car/config and .car/app +- [ ] 2. Survey the copy and choose service boundaries +- [ ] 3. Read adapter.md + manifest.md; write declarations, adapters, workflow; + use the `canyonos config` flow to review deployment choices, then write config +- [ ] 4. validate.py exits 0; report readiness and stop +``` + +Do not skip step 4. The porting workflow ends when validation exits 0: +report the files created, warnings and unresolved runtime blockers, then stop. +Never build or deploy as an implicit continuation of the port. + +## References + +Every reference is linked from here and read whole when its trigger fires. What +differs between the groups is the *kind* of trigger. + +**Before you write.** Triggered by the step, not by a symptom: a porter cannot +look up a rule whose violation builds green and fails in a container. Neither is +optional. + +- [references/adapter.md](references/adapter.md) -- choosing the entrypoint, + bridging async, session state. Read before writing into `.car/app`. +- [references/manifest.md](references/manifest.md) -- the agent yaml, the + complete manifest, and how to build a `requirements` list. Read before writing + into `.car/config`. + +**When the target has this shape.** Triggered by a fact about the source or the +deployment, all three knowable at step 1. + +- [references/packaging.md](references/packaging.md) -- read when a source + import does not resolve from `/app`, the source is nested, packaging metadata + is involved, or the source reads non-Python files at runtime. +- [references/source-survey.md](references/source-survey.md) -- read after + preparing the copy and before choosing service boundaries. +- [references/refresh.md](references/refresh.md) -- read when `.car/app` already + exists and the source has changed; preserve port edits while refreshing it. +- [references/llm-proxy.md](references/llm-proxy.md) -- read when the target + includes `llm_proxy`. +- [references/ec2.md](references/ec2.md) -- read when any config entry uses + `provider: EC2`. + +**After something failed.** Triggered by a symptom. + +- [references/troubleshooting.md](references/troubleshooting.md) -- read after an + explicitly approved deploy fails during build, startup, or a request; + symptom-to-cause tables. +- [references/runtime-contract.md](references/runtime-contract.md) -- read when + a validator finding needs explanation or the runtime mechanism is unclear. + +**For orientation.** + +- [references/example-port.md](references/example-port.md) -- one LangGraph port + end to end: the decisions, the files, and the evidence that closed it. + +## Goal: a self-contained `.car`, and a source tree that never learns about it + +The port lives entirely inside `.car/`, next to the application source and +never inside it: + +```text +.car/config/global_controller.yaml deployment manifest +.car/config/policy.yaml optional access restriction +.car/config/.yaml one callable surface per service +.car/app/ a copy of the application source +.car/app//.py the adapter, written where the code it wraps lives +.car/app//_workflow.py HTTP entry point; calls deploy() +.car/app/pyproject.toml conditional nested-import scaffolding +/ the developer's tree, untouched and unaware +``` + +`.car` has exactly two authored directories: `config/`, which holds every +declaration Canyon owns, and `app/`, the copy that becomes `/app` in every +container. The container keeps the directory structure the application already +had. Write adapters into that copy, in the module the code they wrap already +lives in -- not into new `agents/` and `workflow/` directories. `canyonos` +commands run from the application root and read `.car` below it. + +Nothing under `.car` points back out at the application source, and nothing in +the application source points at `.car`. Deleting `.car` returns the project to +exactly where it started; regenerating it touches no file the developer owns. + +The file count follows the deployment. A multi-agent port has one yaml/adapter +pair per service that is worth deploying separately. If a source class in the +copy already satisfies the runtime contract, point its config entry at that +file and do not write an adapter beside it. + +Everything the source already owns—prompts, tools, schemas, parsing, retries, +model clients, and node bodies—is imported from where the copy keeps it. The +port re-expresses only the CanyonOS Core boundary and framework-owned +orchestration. + +## 1. Prepare the artifact tree, then survey it + +**Choose the source's import root, which is not always its repository root.** +`/app` is the copy, and without the editable-install capability it is the only +entry on `sys.path`, so a source under `src/` that imports `from tools import +...` needs the *contents* of `src/` at `.car/app/`. Read the source's own +imports, not its directory names, to decide. Getting it wrong builds green and +answers `No agent loaded` on the first request; +[references/packaging.md](references/packaging.md) works the case through. + +Once the import root is known, use the skill's preparation script rather than +assembling `.car` with ad hoc copy commands: + +```bash +python3 /prepare.py .car +``` + +The script creates `.car/config/` and copies the import root's **contents** into +`.car/app/`, preserving its structure. It excludes version-control data, +`.car`, virtualenvs, caches, build outputs, bytecode, and credential-bearing +`.env*` files while retaining `.env.example`, `.env.sample`, and +`.env.template`. It rejects symbolic links because they either escape the +self-contained artifact or are skipped by the runtime source sweep. + +If `.car/app` already exists, read [references/refresh.md](references/refresh.md) +and use `--refresh`. It updates source-owned files while preserving port edits, +and stops atomically when both sides changed one path. Use `--force` only to +discard every edit in `.car/app`; it leaves `.car/config/` unchanged. + +Choosing the import root remains a porter decision; the script standardizes +only directory creation and copying. After it runs, every edit is inside +`.car`. The application source outside it is read-only for the rest of the port +-- `git status` at the end shows `.car/` and nothing else. + +Read [references/source-survey.md](references/source-survey.md), survey the +copy, and run the validator. The survey determines the source facts used in the +next two steps; do not infer them from framework conventions. + +## 2. Choose service boundaries + +Start with one service. Split only when it creates independent parallel work or +a distinct resource/replica profile. + +- Keep a ReAct loop together; every turn needs shared message history. +- Hoist supervisor task lists and `Send`-style fan-out into the workflow. +- Do not create a one-replica service with no distinct resource profile merely + to mirror every source graph node. + +Rewrite framework-owned edges as ordinary Python **only where they cross a +service boundary you chose**. A graph whose nodes all land in one agent has no +boundary to express: keep `graph.compile().invoke(...)` and wrap it. Rewriting +it anyway restates control flow the source already had working and buys no +deployment. Import the connected node functions unchanged wherever you do +rewrite. + +Construct runtime-injected service objects from source configuration; do not +invent models, dimensions, stores, or defaults silently. Report any choice the +source does not specify. A service object that holds state across requests -- a +vector store, a memory, a checkpointer built in `__init__` -- makes +`replicas: 1` a correctness requirement rather than a sizing choice, because +the controller picks a replica per call and the others cannot see that state. +Say so in the report; do not leave it implied. + +## 3. Write declarations and adapters + +Read [references/adapter.md](references/adapter.md) before writing an adapter, +and [references/manifest.md](references/manifest.md) before writing +`.car/config`. Neither is optional and neither is triggered by a symptom: every +rule in them builds green and fails inside a container. + +For each service, keep these names aligned: + +```text +config entry name == yaml agent.name == entrypoint class name +``` + +Write a no-argument, synchronous adapter class at the entrypoint selected by +`adapter.md`. Import source-owned behavior instead of duplicating it. Expose +`main(query: str)` in the workflow, import every service from its exact +entrypoint module, and call `deploy(main, port=...)` at module scope. + +For parallel remote calls, dispatch before resolving: + +```python +futures = [agent.work(item=item) for item in items] +results = [json.loads(future.value()) for future in futures] +``` + +Do not fuse dispatch and `.value()` in one comprehension. Do not add a main +guard; the workflow executes as `__main__` in production. + +Build declarations and per-image requirements from the copied import graph. +Then use the View/Change flow in `manifest.md` (and `canyonos config` when +interactive) to review developer-owned deployment choices. Write only the +reviewed candidate and rerun validation. + +## Source-integrity boundary + +The validator owns mechanical runtime rules; do not duplicate its check list in +the prompt. The porter owns the rules static analysis cannot prove: + +- Never edit outside `.car` or copy source-owned prompts, tools, schemas, model + calls, and node bodies into an adapter. +- Never swap the source provider, invent runtime configuration, or silently + move, drop, or reclassify a dependency. +- Rewrite framework control flow only where it crosses a service boundary; + preserve it inside a service. +- Never hardcode or bake a real credential into `.car`. + +When a source defect or unsupported runtime capability requires breaking one of +these boundaries, report the blocker and obtain approval for that specific +change. Do not broaden approval to unrelated source edits. + +## 4. Validate and stop + +Run static preflight from the application root: + +```bash +python3 /validate.py .car +``` + +Fix every ERROR and re-run until it exits 0. Warnings and capability +limitations are not permission to hide risk: list each one in the handoff and +say whether it blocks this source. Confirm that `git status` outside `.car` +shows no change to a file the developer owns. + +At that point, report that the `.car` port is validated and stop. Ask the user a +direct yes/no question before taking the next step: + +> Validation passed. Run `canyonos deploy` now? This will build images and start +> the deployment. + +Do not run a standalone build first. `canyonos deploy` owns both build and +deployment, and must run only after explicit user approval. Silence, an +unattended run, or the original request to "port" is not approval. + +If the user approves, run from the application root: + +```bash +canyonos deploy +``` + +Do not add build, probe, deployment-debugging, or cleanup work to this skill's +porting flow. diff --git a/.claude/skills/porting-to-canyonos/prepare.py b/.claude/skills/porting-to-canyonos/prepare.py new file mode 100755 index 0000000..963b0c5 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/prepare.py @@ -0,0 +1,363 @@ +#!/usr/bin/env python3 +"""Create a CanyonOS Core artifact tree from a chosen Python import root. + +The script owns the mechanical part of step 1: it creates ``.car/config`` and +copies the selected import root to ``.car/app`` with development artifacts and +credential files excluded. Choosing the correct import root still requires +reading the application's imports. +""" + +import argparse +import hashlib +import json +import os +import shutil +import sys +import uuid +from pathlib import Path, PurePosixPath + +EXCLUDED_DIRECTORIES = frozenset( + { + ".car", + ".git", + ".hg", + ".mypy_cache", + ".pytest_cache", + ".ruff_cache", + ".tox", + ".venv", + ".svn", + "__pycache__", + "build", + "dist", + "env", + "htmlcov", + "node_modules", + "venv", + } +) +EXCLUDED_FILE_SUFFIXES = (".pyc", ".pyo") +ENV_TEMPLATES = frozenset({".env.example", ".env.sample", ".env.template"}) +STATE_FILENAME = ".porting-state.json" + + +def _is_relative_to(path: Path, parent: Path) -> bool: + try: + path.relative_to(parent) + except ValueError: + return False + return True + + +def _reject_symlinks(import_root: Path) -> None: + """Keep the copied artifact independent of paths outside the copy.""" + for path in import_root.rglob("*"): + if path.is_symlink(): + relative = path.relative_to(import_root) + raise ValueError( + f"import root contains a symbolic link: {relative}; replace it " + "with the intended file or directory before preparing the port" + ) + + +def _ignore_factory(artifact_root: Path): + def ignore(directory: str, names: list[str]) -> set[str]: + directory_path = Path(directory) + ignored = set() + for name in names: + path = directory_path / name + if path.resolve() == artifact_root: + ignored.add(name) + elif path.is_dir() and name in EXCLUDED_DIRECTORIES: + ignored.add(name) + elif name.startswith(".env") and name not in ENV_TEMPLATES: + ignored.add(name) + elif name.endswith(EXCLUDED_FILE_SUFFIXES): + ignored.add(name) + return ignored + + return ignore + + +def _copy_source(import_root: Path, destination: Path, artifact_root: Path) -> None: + shutil.copytree( + import_root, + destination, + ignore=_ignore_factory(artifact_root), + copy_function=shutil.copy2, + symlinks=True, + ) + + +def _hash_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def _file_hashes(root: Path) -> dict[str, str]: + return { + path.relative_to(root).as_posix(): _hash_file(path) + for path in sorted(root.rglob("*")) + if path.is_file() and not path.is_symlink() + } + + +def _load_state(config_dir: Path) -> dict[str, str]: + state_path = config_dir / STATE_FILENAME + if not state_path.is_file(): + raise ValueError( + f"{state_path} is missing; this artifact predates refresh tracking. " + "Use --force only if discarding all edits in .car/app is intentional" + ) + try: + state = json.loads(state_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as error: + raise ValueError(f"cannot read refresh state {state_path}: {error}") from error + files = state.get("source_files") if isinstance(state, dict) else None + if ( + not isinstance(state, dict) + or state.get("version") != 1 + or not isinstance(files, dict) + or not all(_valid_state_entry(path, digest) for path, digest in files.items()) + ): + raise ValueError(f"invalid refresh state: {state_path}") + return files + + +def _valid_state_entry(path: object, digest: object) -> bool: + if not isinstance(path, str) or not isinstance(digest, str): + return False + relative = PurePosixPath(path) + return ( + path == relative.as_posix() + and not relative.is_absolute() + and path not in ("", ".") + and ".." not in relative.parts + and len(digest) == 64 + and all(character in "0123456789abcdef" for character in digest) + ) + + +def _write_state(path: Path, source_hashes: dict[str, str]) -> None: + path.write_text( + json.dumps( + {"version": 1, "source_files": source_hashes}, + indent=2, + sort_keys=True, + ) + + "\n", + encoding="utf-8", + ) + + +def _refresh_app( + app_dir: Path, + source_copy: Path, + staged_app: Path, + previous_source: dict[str, str], +) -> None: + current = _file_hashes(app_dir) + incoming = _file_hashes(source_copy) + conflicts = [] + + for relative in sorted(previous_source.keys() | incoming.keys() | current.keys()): + old = previous_source.get(relative) + new = incoming.get(relative) + edited = current.get(relative) + source_changed = new != old + app_changed = edited != old + if old is None and new is not None and edited not in (None, new): + conflicts.append(relative) + elif old is not None and source_changed and app_changed and edited != new: + conflicts.append(relative) + + # File/directory replacements need the same three-way protection. A new + # source file at `pkg` must not erase a port-only `pkg/adapter.py`, and a + # new source directory must not silently replace a port-authored file at + # `pkg`. + for incoming_path in incoming: + prefix = incoming_path + "/" + for current_path, edited in current.items(): + if not current_path.startswith(prefix): + continue + if edited != previous_source.get(current_path): + conflicts.append(current_path) + for current_path, edited in current.items(): + prefix = current_path + "/" + if any(incoming_path.startswith(prefix) for incoming_path in incoming): + if edited != previous_source.get(current_path): + conflicts.append(current_path) + + if conflicts: + conflicts = sorted(set(conflicts)) + shown = "\n ".join(conflicts[:20]) + suffix = "" if len(conflicts) <= 20 else f"\n ... and {len(conflicts) - 20} more" + raise ValueError( + "refresh found files changed in both the source and .car/app:\n " + f"{shown}{suffix}\nResolve them in .car/app, then update the source " + "or use --force only to discard all port edits" + ) + + shutil.copytree(app_dir, staged_app, copy_function=shutil.copy2, symlinks=True) + + # Apply safe source deletions first so file-to-directory changes have room. + for relative, old in previous_source.items(): + if relative in incoming or current.get(relative) != old: + continue + target = staged_app / relative + if target.is_file(): + target.unlink() + + for directory in sorted( + (path for path in staged_app.rglob("*") if path.is_dir()), + key=lambda path: len(path.parts), + reverse=True, + ): + try: + directory.rmdir() + except OSError: + pass + + for relative, new in incoming.items(): + old = previous_source.get(relative) + edited = current.get(relative) + if new == old or edited == new: + continue + if old is not None and edited != old: + continue + target = staged_app / relative + if target.is_dir(): + shutil.rmtree(target) + target.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(source_copy / relative, target) + + +def prepare( + import_root: Path, + artifact_root: Path, + force: bool = False, + refresh: bool = False, +) -> None: + import_root = import_root.expanduser().resolve() + artifact_root = artifact_root.expanduser().resolve() + app_dir = artifact_root / "app" + config_dir = artifact_root / "config" + + if not import_root.is_dir(): + raise ValueError(f"import root is not a directory: {import_root}") + if import_root == artifact_root: + raise ValueError("artifact root cannot also be the import root") + if _is_relative_to(import_root, artifact_root): + raise ValueError("import root cannot be inside the artifact root") + if app_dir.exists() and not app_dir.is_dir(): + raise ValueError(f"app path exists but is not a directory: {app_dir}") + if force and refresh: + raise ValueError("--force and --refresh are mutually exclusive") + if refresh and not app_dir.is_dir(): + raise ValueError(f"cannot refresh because {app_dir} does not exist") + if app_dir.exists() and not force and not refresh: + raise FileExistsError( + f"{app_dir} already exists; use --refresh to preserve port edits, or " + "--force to discard and replace the entire source copy" + ) + if config_dir.exists() and not config_dir.is_dir(): + raise ValueError(f"config path exists but is not a directory: {config_dir}") + + _reject_symlinks(import_root) + if refresh: + _reject_symlinks(app_dir) + + artifact_root.mkdir(parents=True, exist_ok=True) + config_dir.mkdir(exist_ok=True) + transaction_id = uuid.uuid4().hex + source_copy = artifact_root / f".source-{transaction_id}.tmp" + temporary_app = artifact_root / f".app-{transaction_id}.tmp" + previous_app = artifact_root / f".app-{uuid.uuid4().hex}.previous" + temporary_state = config_dir / f".{STATE_FILENAME}-{transaction_id}.tmp" + state_path = config_dir / STATE_FILENAME + + installed_new_app = False + try: + _copy_source(import_root, source_copy, artifact_root) + source_hashes = _file_hashes(source_copy) + if refresh: + previous_source = _load_state(config_dir) + _refresh_app(app_dir, source_copy, temporary_app, previous_source) + shutil.rmtree(source_copy) + else: + source_copy.rename(temporary_app) + _write_state(temporary_state, source_hashes) + if app_dir.exists(): + app_dir.rename(previous_app) + temporary_app.rename(app_dir) + installed_new_app = True + os.replace(temporary_state, state_path) + except Exception: + if source_copy.exists(): + shutil.rmtree(source_copy) + if temporary_app.exists(): + shutil.rmtree(temporary_app) + if temporary_state.exists(): + temporary_state.unlink() + if installed_new_app and app_dir.exists(): + shutil.rmtree(app_dir) + if previous_app.exists(): + previous_app.rename(app_dir) + raise + + if previous_app.exists(): + shutil.rmtree(previous_app) + + +def parse_args(argv=None): + parser = argparse.ArgumentParser( + description="Create .car/config and copy an import root into .car/app." + ) + parser.add_argument( + "import_root", + help="directory whose contents should become the contents of .car/app", + ) + parser.add_argument( + "artifact_root", + nargs="?", + default=".car", + help="artifact directory to create (default: .car)", + ) + parser.add_argument( + "--force", + action="store_true", + help="discard edits and replace an existing app/ copy; leave config/ unchanged", + ) + parser.add_argument( + "--refresh", + action="store_true", + help="update unchanged source files while preserving port-authored edits", + ) + return parser.parse_args(argv) + + +def main(argv=None) -> int: + args = parse_args(argv) + try: + prepare( + Path(args.import_root), + Path(args.artifact_root), + force=args.force, + refresh=args.refresh, + ) + except (OSError, ValueError) as error: + sys.stderr.write(f"prepare.py: {error}\n") + return 1 + + artifact_root = Path(args.artifact_root) + print(f"Created {artifact_root / 'config'}") + action = "Refreshed" if args.refresh else "Copied" + print(f"{action} {Path(args.import_root)} to {artifact_root / 'app'}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/.claude/skills/porting-to-canyonos-core/references/adapter.md b/.claude/skills/porting-to-canyonos/references/adapter.md similarity index 89% rename from .claude/skills/porting-to-canyonos-core/references/adapter.md rename to .claude/skills/porting-to-canyonos/references/adapter.md index 61d6c07..606e4ba 100644 --- a/.claude/skills/porting-to-canyonos-core/references/adapter.md +++ b/.claude/skills/porting-to-canyonos/references/adapter.md @@ -1,7 +1,7 @@ # Writing what goes into `.car/app` -Read this before writing any adapter. Every rule here builds green, passes -`ventis build`, and fails inside a container -- which is why the trigger is the +Read this before writing any adapter. Every rule here can pass static build +checks and fail only when a container loads -- which is why the trigger is the step, not a symptom. ## Contents @@ -43,10 +43,11 @@ another. fails at agent load. Make its top-level imports absolute; the modules it imports keep theirs. V035. 5. **Does module-level code perform a real run?** A script ending in - `result = crew.kickoff(...)` / `print(result)` fires that run on every - container start and every probe, before a request exists. Delete the - invocation and keep the construction: M18 protects prompts, tools, schemas - and model calls, not a script's own main body. + `result = crew.kickoff(...)` / `print(result)` fires that run whenever the + module loads, before a request exists. Delete the + invocation and keep the construction. SKILL.md's source-integrity boundary + protects prompts, tools, schemas, model calls and node bodies -- not a + script's own main body. ## Bridging async diff --git a/.claude/skills/porting-to-canyonos-core/references/ec2.md b/.claude/skills/porting-to-canyonos/references/ec2.md similarity index 51% rename from .claude/skills/porting-to-canyonos-core/references/ec2.md rename to .claude/skills/porting-to-canyonos/references/ec2.md index e06daa0..486049d 100644 --- a/.claude/skills/porting-to-canyonos-core/references/ec2.md +++ b/.claude/skills/porting-to-canyonos/references/ec2.md @@ -9,6 +9,12 @@ top-level `ec2` block supplies the runtime's required infrastructure and SSH settings. Read the target checkout's deploy preflight and EC2 runtime before writing the block; do not copy values from an example environment. +These identifiers come from the developer, in step 3's config round -- they are +the one part of the manifest with no safe default. If the round produces no +answer, leave the entry `provider: local` and report that EC2 was requested but +not configured. Never fill the block from an example, a previous port, or +another entry in the same manifest. + Typical required categories are: - AMI and instance type @@ -16,7 +22,7 @@ Typical required categories are: - security groups - SSH user and credentials accepted by the runtime -`ventis deploy` owns basic EC2 config validation. A preflight pass is not proof +`canyonos deploy` owns basic EC2 config validation. A preflight pass is not proof that provisioning, SSH, image transfer, or remote container startup works. ## Networking @@ -29,13 +35,14 @@ The environment file may be copied temporarily to a remote host by runtimes that expose the `env_file` capability. Confirm behavior from the capability probe and target runtime rather than assuming local Docker semantics. -## Probes and cleanup +## Deployment and cleanup -Run the same runtime and adapter probes against the exact image before remote -deployment. After deploy, verify the remote container logs; controller health -can be green even when agent loading failed. +After the user explicitly approves `canyonos deploy`, verify the remote +container logs; controller health can be green even when agent loading failed. +Do not start a separate build or deployment as part of validation. -Stop foreground deploy normally so the controller can terminate recorded EC2 -instances. If provisioning or startup fails before an instance is recorded, -inspect the cloud provider directly and remove exact leaked resources. Never use -a broad cleanup command against unrelated instances. +Ctrl+C stops CLI log monitoring, not necessarily the deployment. Ask before +running `canyonos stop` so the controller can terminate recorded EC2 instances. +If provisioning or startup fails before an instance is recorded, inspect the +cloud provider directly and remove exact leaked resources. Never use a broad +cleanup command against unrelated instances. diff --git a/.claude/skills/porting-to-canyonos-core/references/example-port.md b/.claude/skills/porting-to-canyonos/references/example-port.md similarity index 79% rename from .claude/skills/porting-to-canyonos-core/references/example-port.md rename to .claude/skills/porting-to-canyonos/references/example-port.md index 8308f18..4da2d00 100644 --- a/.claude/skills/porting-to-canyonos-core/references/example-port.md +++ b/.claude/skills/porting-to-canyonos/references/example-port.md @@ -1,7 +1,8 @@ # One port, end to end -A LangGraph email assistant, ported and deployed. Read this for the shape of the -decisions; the rules themselves are in SKILL.md. +A LangGraph email assistant, ported and validated, then deployed with explicit +approval. Read this for the shape of the decisions; the rules themselves are in +SKILL.md. ## Contents @@ -33,10 +34,12 @@ until the model calls `Done`. is `src/`, not the repository root: ```bash -rsync -a --exclude '.git' --exclude '.car' --exclude '__pycache__' \ - --exclude '.env' src/ .car/app/ +python3 /prepare.py src .car ``` +This creates `.car/config/` and copies the contents of `src/` into `.car/app/` +with the standard source and credential exclusions. + Copying the repository root instead puts those modules at `/app/src/tools` while `/app` is the only entry on `sys.path`. The build stays green, the replica reports healthy, and the first request answers `No agent loaded` with @@ -97,18 +100,27 @@ relative to `.car/app`. The workflow imports the agent from its entrypoint -- replaces with a stub. The platform sends `{query: string}` only, so the four email fields ride inside -`query` as JSON and the workflow unpacks them. `main` returns a dict, and -`GET /status/` hands it back under `result`. +`query` as JSON and the workflow unpacks them. The adapter returns a dict; the +runtime encodes it once for transport, so the workflow decodes the Future once +and returns an ordinary dict without another `json.dumps`: + +```python +def main(query: str) -> dict: + email = json.loads(query) + triage = json.loads(agent.triage(email_input=email).value()) + if triage["goto"] == "END": + return triage + return json.loads(agent.respond(messages=triage["messages"]).value()) +``` + +`GET /status/` hands that result back under `result`. ## The evidence ```text -validate.py .car 0 errors -ventis build ventis-emailagent, ventis-workflow -docker run ... import local_controller both images -docker run --env-file .env ... EmailAgent() loads -docker run ventis-workflow ... from email_assistant import EmailAgent stub imports -ventis deploy 2 replicas ready +validate.py .car 0 errors; porting workflow stopped +user approval yes, run deployment +canyonos deploy build complete; 2 replicas ready POST /main 202 {"request_id": ...} GET /status/ status: error, 401 from OpenAI ``` diff --git a/.claude/skills/porting-to-canyonos-core/references/llm-proxy.md b/.claude/skills/porting-to-canyonos/references/llm-proxy.md similarity index 90% rename from .claude/skills/porting-to-canyonos-core/references/llm-proxy.md rename to .claude/skills/porting-to-canyonos/references/llm-proxy.md index e43885c..d2047bb 100644 --- a/.claude/skills/porting-to-canyonos-core/references/llm-proxy.md +++ b/.claude/skills/porting-to-canyonos/references/llm-proxy.md @@ -22,8 +22,8 @@ source SDK, model ID, request body, and response parsing unchanged. Each SDK generation reads a different base-URL variable, and a wrapper library reads a different one from the SDK it wraps. Set only the name this reference used to give and the container reaches the real provider with a placeholder key: -a 401 that reads like a broken port, after validate, build and both probes have -passed. Set all of them for whichever providers the source uses: +a 401 that reads like a broken port, after validation and the deployment build +have passed. Set all of them for whichever providers the source uses: ```dotenv OPENAI_BASE_URL=http://host.docker.internal:8081/openai/v1 @@ -57,9 +57,9 @@ credentials in the separate proxy process, not in the port's `env_file`. Some sources build the HTTP call themselves -- `urllib.request` against a module constant like `API = "https://api.openai.com/v1/responses"` -- and read no -base-URL variable at all. Editing that constant is a source edit M18 and M21 -forbid, so the `env_file` is inert and the container can only ever reach the -real provider. Report this as a proxy blocker and stop. Do not hand the +base-URL variable at all. Editing that constant swaps the source provider's +endpoint, which SKILL.md's source-integrity boundary forbids, so the `env_file` +is inert and the container can only ever reach the real provider. Report this as a proxy blocker and stop. Do not hand the container a real upstream credential instead. Detect it before deploying: grep the source for the provider hostname. A literal @@ -79,7 +79,7 @@ curl http://127.0.0.1:8081/healthz Local CanyonOS Core containers resolve `host.docker.internal` through their Docker host mapping. On EC2 that name resolves to each EC2 Docker host, not the -machine running `ventis deploy`. Distributed deployments need a reachable proxy +machine running `canyonos deploy`. Distributed deployments need a reachable proxy address or one proxy on each host. ## Supported call shape diff --git a/.claude/skills/porting-to-canyonos-core/references/manifest.md b/.claude/skills/porting-to-canyonos/references/manifest.md similarity index 55% rename from .claude/skills/porting-to-canyonos-core/references/manifest.md rename to .claude/skills/porting-to-canyonos/references/manifest.md index e1ffb7d..e90188b 100644 --- a/.claude/skills/porting-to-canyonos-core/references/manifest.md +++ b/.claude/skills/porting-to-canyonos/references/manifest.md @@ -1,15 +1,77 @@ # Writing what goes into `.car/config` -Read this before writing the manifest or an agent declaration. `ventis build` -owns yaml syntax; nothing here is syntax. These are the values that build green -and then decide whether a container can import its own dependencies. +Read this before writing the manifest or an agent declaration. The validator +checks YAML structure and the public artifact contract before an approved +`canyonos deploy`; this reference explains how to derive the values inside it. ## Contents +- Who decides each key +- Review configuration through the CanyonOS CLI flow - Agent yaml - Requirements - The manifest, in full +## Who decides each key + +Two kinds of key share one file. A **derived** key has exactly one right answer +and the copy holds it; asking the developer can only make it worse. A +**developer** key is a deployment choice the source does not contain, and +deriving it means guessing and presenting the guess as a reading. + +The configuration review shows the whole manifest and asks about the second +column only, in one round, carrying these defaults. + +| Key | Decided by | Default when unanswered | +|---|---|---| +| `name`, `entrypoint`, `workflow_file`, `type` | derived — service boundaries, step 2 | — | +| `requirements` | derived — the entry's import graph | — | +| `database` | neither; omit it always (see below) | absent | +| `provider` | developer | `local` | +| `ec2:` block, `instance_type` | developer — no default is safe | entry stays `local` | +| `replicas` | developer, *unless* cross-request state forces `1` | `1` | +| `resources.cpu` / `resources.memory` | developer | `1` / `512` MiB | +| `api_port` | developer | `8080` | +| `redis_port`, `redis.host` / `.port` / `.db` | developer | `6379`, `localhost` / `6379` / `0` | +| `poll_interval` | developer | `5` | +| `env_file` | developer — the file's location and whether it exists | `.env` when the survey found credential reads, else absent | +| `policy.yaml` | developer | absent | + +Two entries in that table are not free choices, and saying so is part of showing +the config rather than asking about it: + +- **`replicas` stops being a choice once a service holds cross-request state.** + Where the step-2 survey found such state, SKILL.md already fixes `replicas: 1` + as a correctness requirement, so `1` is derived: report it as a constraint and + do not offer to raise it. +- **EC2 identifiers are wrong to invent.** ec2.md forbids copying them from an + example environment, and a wrong AMI, subnet, or security group fails at + deploy preflight or, worse, provisions something unreachable. Unanswered + means the entry stays `local`. + +## Review configuration through the CanyonOS CLI flow + +Use the interaction implemented by `canyonos config` before writing +`.car/config/global_controller.yaml`: + +1. Build the complete candidate manifest in memory from derived values and the + defaults above. +2. **View** prints the whole candidate, annotating defaults and source-imposed + constraints such as `replicas: 1` for in-memory state. +3. **Change** asks in one batch only for developer-owned values: provider and + EC2 fields, unconstrained replicas, resources, ports, secret-file location, + and access restrictions. Show each current/default value, apply answers, and + show the result. +4. Write the reviewed candidate and run the validator. + +Prefer running `canyonos config` when an interactive terminal is available; +otherwise reproduce View/Change in conversation. Do not ask for derived values +such as entrypoints or requirements. + +An unattended `canyonos integrate` run must not block on this interaction. Use +and report the displayed defaults. Never invent EC2 infrastructure identifiers: +without them, keep the entry `local`. + ## Agent yaml Declarations go in `.car/config/`, beside the manifest. The build reads every @@ -40,9 +102,9 @@ you wrote: In a peer image the entrypoint is a stub, but its package `__init__` and its siblings are real, so that image still installs what they import. 3. Omit distributions reachable only from source files no image imports, such as - a Gradio or Streamlit UI beside the agent. M22 forbids reclassifying a - declared dependency, not declining to ship an unreachable one; name what you - left out in the report. + a Gradio or Streamlit UI beside the agent. The source-integrity boundary + forbids reclassifying a declared dependency, not declining to ship an + unreachable one; name what you left out in the report. `validate.py` walks the same graph and reports what is missing as W006. @@ -59,8 +121,9 @@ today: - **The source predates a known SDK break**: pin contemporaneous with its last commit. A 2023 AutoGen script passing `request_timeout=` needs `pyautogen==0.1.14`, which depends on `openai<1`, not `autogen==0.7.5`, which - floors on `openai>=1.58` where that kwarg is `timeout`. M23 forbids rewriting - the source call, so the pin has to absorb the difference. Compare the source's + floors on `openai>=1.58` where that kwarg is `timeout`. The source-integrity + boundary forbids rewriting that call, so the pin has to absorb the + difference. Compare the source's commit date against the pin's release date whenever the source hardcodes SDK kwargs. diff --git a/.claude/skills/porting-to-canyonos-core/references/packaging.md b/.claude/skills/porting-to-canyonos/references/packaging.md similarity index 68% rename from .claude/skills/porting-to-canyonos-core/references/packaging.md rename to .claude/skills/porting-to-canyonos/references/packaging.md index cd8265a..e87c002 100644 --- a/.claude/skills/porting-to-canyonos-core/references/packaging.md +++ b/.claude/skills/porting-to-canyonos/references/packaging.md @@ -1,7 +1,8 @@ # Packaging and import roots Read this reference when an adapter imports nested source code, the source uses a -`src/` layout, or V031 reports an import-root problem. +`src/` layout, V031 reports an import-root problem, or the source reads +non-Python files at runtime. ## Contents @@ -10,6 +11,7 @@ Read this reference when an adapter imports nested source code, the source uses - Detect support, do not infer it from release history - Root metadata is the trigger - Dependencies in nested metadata +- Runtime data and configuration files - Validation boundary ## What `/app` can import @@ -50,7 +52,7 @@ Reach for the metadata below only when one copy root cannot serve every import Run: ```bash -python /validate.py .car +python3 /validate.py .car ``` Read the `editable_install` capability. If it is unavailable and the original @@ -105,8 +107,31 @@ config requirements. Report declared-but-unused toolchain dependencies and their image cost; let the owner decide whether source metadata should change. +## Runtime data and configuration files + +`prepare.py` copies non-Python files into `.car/app`, but that does not prove the +runtime's image sweep carries them into a container. Inventory every file opened +by the selected import graph: prompt templates, JSON schemas, PDFs, local +corpora, certificates, and framework configuration such as CrewAI +`agents.yaml` and `tasks.yaml`. + +Run `validate.py` and read its `sweeps_all_files` capability: + +- When available, retain each asset at the same path relative to the chosen + import root. Check any path derived from the original repository root or + process working directory; the container starts from `/app`. +- When unavailable, a required non-Python asset is a runtime blocker. Report it + and stop after validation. Do not conceal the gap by base64-encoding the file + into Python, changing a hardcoded path, or duplicating framework config into + adapter code; those changes restate source-owned data and behavior. + +Do not treat successful construction as evidence that configuration loaded. +Frameworks such as CrewAI may warn about a missing yaml and create an empty +configuration, then fail only when the first agent or task is accessed. Inspect +those decorators and file references statically during the survey. + ## Validation boundary -`ventis build` owns packaging syntax and installation errors. `validate.py` -checks only whether adapter imports appear to require a nested root that the -runtime will not expose. +The build phase of `canyonos deploy` owns packaging syntax and installation +errors. `validate.py` checks only whether adapter imports appear to require a +nested root that the runtime will not expose. diff --git a/.claude/skills/porting-to-canyonos/references/refresh.md b/.claude/skills/porting-to-canyonos/references/refresh.md new file mode 100644 index 0000000..eaab51b --- /dev/null +++ b/.claude/skills/porting-to-canyonos/references/refresh.md @@ -0,0 +1,38 @@ +# Refreshing an existing port + +Read this when `.car/app` already exists and the application source has changed. + +Run the same preparation command with `--refresh` and the same import root: + +```bash +python3 /prepare.py .car --refresh +``` + +The initial copy records source-file hashes in +`.car/config/.porting-state.json`. Refresh compares three states: + +```text +previous source hash → current source + ↘ current .car/app +``` + +- Only the source changed: update `.car/app`. +- Only `.car/app` changed: preserve the port edit. +- The source added a path unused by the port: add it. +- The source deleted an unmodified path: delete it from `.car/app`. +- Both sides changed the same path differently: make no changes and report all + conflicts. + +Resolve a conflict in `.car/app`, then either make the source match that result +or intentionally start over. The script does not guess a merge because an +adapter and its source often change for different reasons while sharing one +module. + +`--force` is not refresh. It discards the entire `.car/app` tree and replaces it +with a clean source copy while retaining `.car/config`. Use it only when every +adapter and workflow edit in `.car/app` is intentionally disposable. + +After refresh, survey changed imports, dependencies, runtime assets, and service +boundaries again, then run the full validator. A clean file merge is not proof +that the deployment contract still holds. + diff --git a/.claude/skills/porting-to-canyonos-core/references/runtime-contract.md b/.claude/skills/porting-to-canyonos/references/runtime-contract.md similarity index 83% rename from .claude/skills/porting-to-canyonos-core/references/runtime-contract.md rename to .claude/skills/porting-to-canyonos/references/runtime-contract.md index eb6ce0b..9d96aff 100644 --- a/.claude/skills/porting-to-canyonos-core/references/runtime-contract.md +++ b/.claude/skills/porting-to-canyonos/references/runtime-contract.md @@ -1,8 +1,8 @@ # CanyonOS Core runtime contract -The product is CanyonOS Core. Compatibility identifiers remain `ventis` for the -CLI and Python package, `VENTIS_*` for runtime variables, and `ventis-*` for -Docker resources. +The product is CanyonOS Core and its user-facing CLI is `canyonos`. Compatibility +identifiers remain `ventis` for the internal Python package, `VENTIS_*` for +runtime variables, and `ventis-*` for Docker resources. Read this reference when implementing an adapter or explaining a validator finding. Runtime-dependent behavior is expressed as capabilities; run @@ -23,10 +23,10 @@ release history. ## Artifact root and discovery -`ventis build` runs from the application root and reads `.car` below it. That -artifact root holds `config/` beside `app/`, the copy of the application source -that becomes `/app` inside every image. Paths in the config are relative to -`app/` and may not escape it. +The build phase of `canyonos deploy` runs from the application root and reads +`.car` below it. That artifact root holds `config/` beside `app/`, the copy of +the application source that becomes `/app` inside every image. Paths in the +config are relative to `app/` and may not escape it. | Input | Discovery | |---|---| @@ -123,8 +123,8 @@ Consequences: Agent import and construction exceptions are caught by the controller. A failed agent may still advertise healthy because health is written independently of -successful agent loading. That is why image probes import both the runtime and -the entrypoint explicitly. +successful agent loading. After an explicitly approved deployment, inspect +container logs rather than treating health as proof that the entrypoint loaded. ## Workflow execution @@ -145,7 +145,7 @@ and resolving inside one comprehension serializes work without raising an error; dispatch all calls first, then resolve them. The workflow container also starts runtime controller code and has its own -package resolution. Probe it independently from agent images. +package resolution. A failure there can differ from failures in agent images. ## Build context and collisions @@ -191,9 +191,9 @@ produce a green image build that dies on: import local_controller ``` -Always run that probe before probing the entrypoint. Treat a generated-code / -runtime-version mismatch as a CanyonOS Core runtime issue, not a reason to alter -source dependencies silently. +Treat a generated-code/runtime-version mismatch during an explicitly approved +`canyonos deploy` as a CanyonOS Core runtime issue, not a reason to alter source +dependencies silently. ## Credentials capability @@ -208,8 +208,8 @@ source needs credentials, report the capability blocker rather than hardcoding or vendoring a secret. A source that constructs its client at import time works only when credentials -are already in the container environment. Image entrypoint probes therefore use -the same env file as deployment. +are already in the container environment. After an explicitly approved deploy, +check loading failures against the same env file configured for deployment. For proxy-specific credential separation, read [llm-proxy.md](llm-proxy.md). @@ -224,10 +224,12 @@ and remote networking are covered in [ec2.md](ec2.md). ## Cleanup boundary -Stopping foreground deploy normally invokes controller cleanup for recorded -containers and Redis. Hard kills and failures before resource registration may -leave resources behind. +`canyonos deploy` follows controller logs after starting the deployment. Ctrl+C +stops that log stream; use `canyonos stop` to request controller teardown when +the user asks to stop the deployment. Hard kills and failures before resource +registration may leave resources behind. -`ventis clean` removes generated `stubs/`, `grpc_stubs/`, and -`docker_container/` under `.car`. It does not remove containers or images. Remove exact leftovers explicitly and preserve `app/`, -`config/`, and requested evidence. +`canyonos clean` removes generated `stubs/`, `grpc_stubs/`, and +`docker_container/`; it does not remove containers or images. Remove exact +leftovers explicitly and preserve `.car/app`, `.car/config`, and requested +evidence. diff --git a/.claude/skills/porting-to-canyonos/references/source-survey.md b/.claude/skills/porting-to-canyonos/references/source-survey.md new file mode 100644 index 0000000..b44538a --- /dev/null +++ b/.claude/skills/porting-to-canyonos/references/source-survey.md @@ -0,0 +1,29 @@ +# Surveying the copied source + +Read this after `prepare.py` and before choosing service boundaries. Survey +`.car/app`, not a guessed abstraction of the original repository. + +Identify all of the following: + +1. The production entry point and callable input/output. If several + implementations look plausible, trace imports from the documented route, + CLI, or launch path instead of choosing by filename. +2. Framework-owned control flow: graphs, crews, chats, routing, fan-out, + commands, and interrupts. +3. Runtime-injected stores, context, memory, sessions, and callback managers. +4. Sync/async boundaries and objects tied to an event loop. +5. The transitive import graph and the source's pinned runtime distributions. +6. Model provider, credential names, streaming, and optional `llm_proxy` use. +7. Independent work that benefits from separate resource or replica profiles. +8. Whether imports resolve with `.car/app` as `/app`; read `packaging.md` when + they do not. +9. Non-Python runtime files such as prompts, CrewAI YAML, PDFs, templates, + schemas, and corpora. If the runtime cannot sweep all files, report this as a + blocker; do not embed files or rewrite paths to hide it. +10. Whether every Python file on the selected import graph parses. Existing + syntax errors are source defects; report them and obtain approval before + changing even the copied version. + +Run `python3 /validate.py .car` after the survey and after every +change. Missing or malformed required inputs fail closed. If a required runtime +capability is reported unavailable, stop instead of assuming it exists. diff --git a/.claude/skills/porting-to-canyonos-core/references/troubleshooting.md b/.claude/skills/porting-to-canyonos/references/troubleshooting.md similarity index 88% rename from .claude/skills/porting-to-canyonos-core/references/troubleshooting.md rename to .claude/skills/porting-to-canyonos/references/troubleshooting.md index 3744ec2..213163c 100644 --- a/.claude/skills/porting-to-canyonos-core/references/troubleshooting.md +++ b/.claude/skills/porting-to-canyonos/references/troubleshooting.md @@ -1,7 +1,8 @@ # Troubleshooting -Read this after a failed build, image probe, deploy, or request. For mechanisms, -read [runtime-contract.md](runtime-contract.md). For proxy or remote-host failures, +Read this after an explicitly approved `canyonos deploy` fails during build, +startup, or a request. For mechanisms, read +[runtime-contract.md](runtime-contract.md). For proxy or remote-host failures, read [llm-proxy.md](llm-proxy.md) or [ec2.md](ec2.md). ## Build or deploy stops early @@ -69,6 +70,7 @@ read [llm-proxy.md](llm-proxy.md) or [ec2.md](ec2.md). | Symptom | Likely cause | |---|---| -| `ventis clean` succeeds but containers remain | The command removes generated directories only | -| `ventis clean` succeeds but images remain | Image deletion is separate and requires exact tags | -| Next deployment collides with old resources | Foreground deploy was killed or crashed before controller cleanup | +| Ctrl+C leaves the deployment running | `canyonos deploy` follows logs; Ctrl+C stops monitoring, not the deployment. Ask before running `canyonos stop` | +| `canyonos clean` succeeds but containers remain | The command removes generated directories only | +| `canyonos clean` succeeds but images remain | Image deletion is separate and requires exact tags | +| Next deployment collides with old resources | The deployment was killed or crashed before controller cleanup | diff --git a/.claude/skills/porting-to-canyonos/validate.py b/.claude/skills/porting-to-canyonos/validate.py new file mode 100755 index 0000000..61788e8 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validate.py @@ -0,0 +1,311 @@ +#!/usr/bin/env python3 +"""Preflight a CanyonOS port before an approved deployment. + +This checks the public `.car` artifact contract first, then parses Python without +importing it to catch failures that would otherwise stay hidden until a +container loads an agent, starts a workflow, or serves its first request. It +fails closed when the required inputs cannot be checked. A replica is not +evidence: the controller writes `healthy` to Redis before `_load_agent` runs. + + python3 validate.py [artifact_root] [-c config/global_controller.yaml] + [--json] [--strict] + +`artifact_root` is the `.car` directory: `config/` beside `app/`, the copy of +the application source that becomes /app inside every container. + +Exit 1 if any ERROR was reported, 0 otherwise. --strict also fails on warnings. + +Runtime capabilities vary across CanyonOS Core installations. This script probes +the importable `ventis` package directly. A capability-gated check reports +UNAVAILABLE when its behavior cannot be proven. +""" + +import argparse +import json +import os +import sys + +SKILL_DIR = os.path.dirname(os.path.abspath(__file__)) +if SKILL_DIR not in sys.path: + sys.path.insert(0, SKILL_DIR) + +from validation.adapter import check_adapter +from validation.core import ERROR, INFO, WARN, Report, load_yaml +from validation.dependencies import ( + check_requirements_coverage, + check_secrets, +) +from validation.entrypoint import ( + check_entrypoint_module, + check_flat_collisions, +) +from validation.manifest import ( + check_declaration_bindings, + check_manifest_structure, + check_policy, + check_self_contained_tree, + discover_agent_declarations, +) +from validation.packaging import check_env_file, check_import_root +from validation.python_source import module_path +from validation.runtime import ( + BASE_AGENT_REQUIREMENTS, + BASE_WORKFLOW_REQUIREMENTS, + CAPABILITY_SOURCE, + probe_capabilities, +) +from validation.workflow import check_workflow + +DEFAULT_CONFIG_PATH = "config/global_controller.yaml" +# ventis/cli.py SOURCE_DIR_NAME -- the duplicated application source. +SOURCE_DIR_NAME = "app" + + +# Path existence and readability are deploy-preflight checks. Do not +# duplicate them here. + + +# ------------------------------------------------------------------ # +# Driver # +# ------------------------------------------------------------------ # + + +def validate(artifact_dir, config_path, capabilities): + """Check the public artifact contract and deeper runtime failure modes.""" + report = Report(artifact_dir, capabilities) + + config, error = load_yaml(config_path) + if error is not None or not isinstance(config, dict): + report.error( + "V001", + config_path, + 0, + f"the global manifest cannot be read: {error or 'expected a YAML mapping'}", + "Validation finishes before an approved `canyonos deploy`, so an " + "unreadable manifest cannot be deferred to deploy.", + ) + return report + + source_dir = os.path.join(artifact_dir, SOURCE_DIR_NAME) + if not os.path.isdir(source_dir): + report.error( + "V032", + artifact_dir, + 0, + f"no `{SOURCE_DIR_NAME}/` beside `config/`", + "The artifact root holds the application source it deploys: " + f"`{SOURCE_DIR_NAME}/` is the copy that becomes /app, and every " + "entrypoint is relative to it. Without it the port has nothing to " + "build and nothing to keep it decoupled from the developer's tree.", + ) + return report + + check_self_contained_tree(report, artifact_dir) + + config_dir = os.path.dirname(config_path) + entries = check_manifest_structure(report, config, config_path, source_dir) + if entries is None: + return report + + agents_by_name = discover_agent_declarations(report, config_dir, config_path) + check_declaration_bindings(report, entries, agents_by_name, config_path) + check_policy(report, config_dir) + + entrypoints = [] + for entry in entries: + if not isinstance(entry, dict) or entry.get("type", "agent") == "workflow": + continue + name = entry.get("name") + entrypoint = entry.get("entrypoint") + if isinstance(entrypoint, str): + entrypoints.append((name, entrypoint)) + if name in agents_by_name: + yaml_path, agent_block = agents_by_name[name] + check_adapter(report, yaml_path, agent_block, entry, source_dir) + entrypoint_path = os.path.join(source_dir, entrypoint or "") + if isinstance(entrypoint, str) and os.path.isfile(entrypoint_path): + check_entrypoint_module(report, source_dir, name, entrypoint) + check_requirements_coverage( + report, + source_dir, + entry, + entrypoint_path, + config_path, + BASE_AGENT_REQUIREMENTS, + ) + + # Where each agent's stub is written, and therefore the only import that + # reaches it over gRPC. + stub_modules = { + name: module_path(entrypoint) + for name, entrypoint in entrypoints + if name in agents_by_name + } + stubbed_entrypoint_paths = [ + os.path.join(source_dir, entrypoint) + for name, entrypoint in entrypoints + if name in agents_by_name + ] + + for entry in entries: + if not isinstance(entry, dict) or entry.get("type", "agent") != "workflow": + continue + workflow_file = entry.get("workflow_file") + if not isinstance(workflow_file, str): + continue + workflow_path = os.path.join(source_dir, workflow_file) + if os.path.isfile(workflow_path): + check_workflow(report, workflow_path, stub_modules) + # The workflow image installs its own list. A module it imports for + # a helper drags that module's dependencies in even though the + # workflow makes no model call of its own. + check_requirements_coverage( + report, + source_dir, + entry, + workflow_path, + config_path, + BASE_WORKFLOW_REQUIREMENTS, + shadowed_paths=stubbed_entrypoint_paths, + ) + + # These survive a green build and otherwise surface only in a container or + # on its first request. + check_flat_collisions(report, source_dir, entrypoints) + check_env_file(report, config, config_path, artifact_dir) + + entrypoint_paths = [ + os.path.join(source_dir, e) + for _, e in entrypoints + if os.path.isfile(os.path.join(source_dir, e)) + ] + check_import_root(report, source_dir, entrypoint_paths) + + port_paths = list(entrypoint_paths) + for entry in entries: + if isinstance(entry, dict) and isinstance(entry.get("workflow_file"), str): + candidate = os.path.join(source_dir, entry["workflow_file"]) + if os.path.isfile(candidate): + port_paths.append(candidate) + + # Secret detection remains because a green image build would permanently + # bake the credential into every image. + check_secrets(report, port_paths) + return report + + +# ------------------------------------------------------------------ # +# Output # +# ------------------------------------------------------------------ # + +LEVEL_ORDER = {ERROR: 0, WARN: 1, INFO: 2} + + +def _wrap(text, width, indent): + words = text.split() + lines = [] + current = "" + for word in words: + candidate = f"{current} {word}".strip() + if len(candidate) + len(indent) > width and current: + lines.append(indent + current) + current = word + else: + current = candidate + if current: + lines.append(indent + current) + return lines + + +def print_report(report, artifact_root): + caps = report.capabilities + if not caps.get("ventis"): + print("ventis is not importable here -- capability-gated rules are") + print("reported UNAVAILABLE rather than checked.\n") + else: + print("CanyonOS Core capabilities detected:") + for key, source in CAPABILITY_SOURCE.items(): + mark = "yes" if caps.get(key) else "no " + print(f" {mark} {key:<22} {source}") + print() + + findings = sorted( + report.findings, + key=lambda f: (LEVEL_ORDER[f["level"]], f["check"], f["path"], f["line"]), + ) + for finding in findings: + where = finding["path"] + if where and finding["line"]: + where = f"{where}:{finding['line']}" + header = f"{finding['check']} {finding['level']:<5}" + print(f"{header} {where}" if where else header) + for line in _wrap(finding["summary"], 78, " "): + print(line) + if finding["mechanism"]: + for line in _wrap(finding["mechanism"], 78, " "): + print(line) + print() + + errors, warnings = report.counts() + if not findings: + print(f"{artifact_root}: clean.") + return + print(f"{errors} error(s), {warnings} warning(s).") + + +def main(argv=None): + parser = argparse.ArgumentParser( + description="Check a CanyonOS Core port against the rules in SKILL.md." + ) + parser.add_argument( + "artifact_root", + nargs="?", + default=".", + help="the .car directory holding config/ and app/ (default: the cwd)", + ) + parser.add_argument( + "-c", + "--config", + default=DEFAULT_CONFIG_PATH, + help=f"config path relative to artifact_root (default: {DEFAULT_CONFIG_PATH})", + ) + parser.add_argument("--json", action="store_true", help="emit findings as JSON") + parser.add_argument( + "--strict", action="store_true", help="fail on warnings as well as errors" + ) + args = parser.parse_args(argv) + + artifact_root = os.path.abspath(args.artifact_root) + config_path = ( + args.config + if os.path.isabs(args.config) + else os.path.join(artifact_root, args.config) + ) + + capabilities = probe_capabilities() + report = validate(artifact_root, config_path, capabilities) + errors, warnings = report.counts() + + if args.json: + print( + json.dumps( + { + "artifact_root": artifact_root, + "capabilities": capabilities, + "errors": errors, + "warnings": warnings, + "findings": report.findings, + }, + indent=2, + ) + ) + else: + print_report(report, report.rel(artifact_root) or artifact_root) + + if errors or (args.strict and warnings): + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/.claude/skills/porting-to-canyonos/validation/__init__.py b/.claude/skills/porting-to-canyonos/validation/__init__.py new file mode 100644 index 0000000..aa7a1a3 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/__init__.py @@ -0,0 +1 @@ +"""Composable validation checks for the CanyonOS porting skill.""" diff --git a/.claude/skills/porting-to-canyonos/validation/adapter.py b/.claude/skills/porting-to-canyonos/validation/adapter.py new file mode 100644 index 0000000..445ff10 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/adapter.py @@ -0,0 +1,175 @@ +"""V006-V010 -- adapter faults the controller swallows inside _load_agent.""" + +import ast +import builtins +import os + +from validation.core import line_of +from validation.python_source import ( + class_methods, + find_class, + parameter_names, + parse_python, + required_parameters, +) + +BUILTIN_TYPE_NAMES = frozenset( + name + for name in dir(builtins) + if isinstance(getattr(builtins, name), type) and not name[0].isupper() +) + + +def check_adapter(report, agent_yaml_path, agent_block, entry, project_dir): + """V006 V007 V008 V009 V010.""" + name = agent_block["name"] + functions = agent_block.get("functions") or [] + check_argument_types(report, agent_yaml_path, functions) + + entrypoint = entry.get("entrypoint") + if not entrypoint: + return + entrypoint_path = os.path.join(project_dir, entrypoint) + if not os.path.isfile(entrypoint_path): + return + + tree, error = parse_python(entrypoint_path) + if tree is None: + report.error( + "V006", + entrypoint_path, + 0, + f"the entrypoint does not parse: {error}", + "_load_agent exec_module's it and swallows the exception; the first " + "request answers 'No agent loaded'.", + ) + return + + class_node = find_class(tree, name) + if class_node is None: + classes = [n.name for n in tree.body if isinstance(n, ast.ClassDef)] + found = ", ".join(classes) if classes else "no classes at all" + report.error( + "V006", + entrypoint_path, + 1, + f"no class named `{name}` at module level (found: {found})", + "_load_agent does getattr(module, VENTIS_AGENT_NAME) and swallows " + "the AttributeError. The class name must equal agent.name exactly.", + ) + return + + methods = class_methods(class_node) + check_constructor(report, entrypoint_path, name, methods) + + for func in functions: + if not isinstance(func, dict) or not isinstance(func.get("name"), str): + continue + check_method(report, entrypoint_path, agent_yaml_path, name, func, methods) + + +def check_argument_types(report, agent_yaml_path, functions): + """V010 -- the type string is pasted into an ast.Name, never checked.""" + for func in functions: + if not isinstance(func, dict): + continue + for arg in func.get("arguments") or []: + if not isinstance(arg, dict) or "type" not in arg: + continue + declared = arg.get("type") + if not isinstance(declared, str): + continue # stub generation reports malformed type values + if declared in BUILTIN_TYPE_NAMES: + continue + report.error( + "V010", + agent_yaml_path, + line_of(arg, "type"), + f"`type: {declared}` is not a builtin", + "stub_generator pastes it verbatim into the generated " + "annotation, and the stub module imports only Future and " + "inspect. Anything else raises NameError when the stub is " + "imported -- after a green build. Use str int float bool dict " + "list.", + ) + + +def check_constructor(report, entrypoint_path, name, methods): + """V007 -- _load_agent calls agent_class() with no arguments.""" + init = methods.get("__init__") + if init is None: + return + required = required_parameters(init) + if required: + report.error( + "V007", + entrypoint_path, + init.lineno, + f"`{name}.__init__` requires {', '.join(required)}", + "_load_agent calls agent_class() with no arguments; the TypeError is " + "swallowed and the first request answers 'No agent loaded'. Read " + "configuration from the environment inside __init__ instead.", + ) + + +def check_method(report, entrypoint_path, agent_yaml_path, class_name, func, methods): + """V008 V009.""" + func_name = func["name"] + method = methods.get(func_name) + if method is None: + report.error( + "V008", + entrypoint_path, + 0, + f"`{class_name}` has no method `{func_name}`", + "The yaml declares it, so callers get a stub for it; the controller " + f"then answers \"Agent {class_name} has no method '{func_name}'\".", + ) + return + + # V009 -- nothing on the execution path awaits. + if isinstance(method, ast.AsyncFunctionDef): + report.error( + "V009", + entrypoint_path, + method.lineno, + f"`{class_name}.{func_name}` is `async def`", + "The executor calls method(**args) with no await, so Redis receives " + "''. Keep the signature synchronous and call " + "asyncio.run(...) inside the body.", + ) + + # V008 -- the controller calls method(**args) with the yaml's names. + declared = [ + arg["name"] + for arg in func.get("arguments") or [] + if isinstance(arg, dict) and isinstance(arg.get("name"), str) + ] + actual = parameter_names(method) + required = required_parameters(method) + + missing = [arg for arg in declared if arg not in actual] + if missing: + report.error( + "V008", + entrypoint_path, + method.lineno, + f"`{class_name}.{func_name}` has no parameter " + f"{', '.join(repr(m) for m in missing)}, but the yaml declares it", + "LocalController does method(**args) with the yaml's argument names. " + "A mismatch is TypeError: unexpected keyword argument, at request " + f"time. See {os.path.basename(agent_yaml_path)}.", + ) + + unfilled = [arg for arg in required if arg not in declared] + if unfilled: + report.error( + "V008", + entrypoint_path, + method.lineno, + f"`{class_name}.{func_name}` requires {', '.join(unfilled)}, which " + "the yaml does not declare", + "Only declared arguments are ever sent, and the generated stub gives " + "none of them a default. Declare them in the yaml or default them " + "in the signature.", + ) diff --git a/.claude/skills/porting-to-canyonos/validation/core.py b/.claude/skills/porting-to-canyonos/validation/core.py new file mode 100644 index 0000000..490b949 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/core.py @@ -0,0 +1,98 @@ +"""Shared result and YAML primitives for validation checks.""" + +import os +from typing import ClassVar + +try: + import yaml +except ImportError: # pragma: no cover - pyyaml is a CanyonOS dependency + raise RuntimeError("validate.py needs pyyaml: pip install pyyaml") from None + + +ERROR = "ERROR" +WARN = "WARN" +INFO = "INFO" + + +class LineDict(dict): + """A mapping that remembers where it and each of its keys were written.""" + + line = 0 + key_lines: ClassVar[dict] = {} + + +class LineLoader(yaml.SafeLoader): + pass + + +def _construct_mapping(loader, node): + data = LineDict() + yield data + data.update(loader.construct_mapping(node, deep=False)) + data.line = node.start_mark.line + 1 + data.key_lines = { + key.value: key.start_mark.line + 1 + for key, _ in node.value + if isinstance(key, yaml.ScalarNode) + } + + +LineLoader.add_constructor( + yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, _construct_mapping +) + + +def line_of(mapping, key=None): + if not isinstance(mapping, LineDict): + return 0 + if key is not None: + return mapping.key_lines.get(key, mapping.line) + return mapping.line + + +def load_yaml(path): + """Parse path and return ``(data, error)`` without raising.""" + try: + with open(path, "r", encoding="utf-8") as handle: + return yaml.load(handle, Loader=LineLoader), None + except Exception as exc: # noqa: BLE001 - every parse failure is a finding + return None, str(exc) + + +class Report: + def __init__(self, project_dir, capabilities): + self.project_dir = project_dir + self.capabilities = capabilities + self.findings = [] + + def add(self, check, level, path, line, summary, mechanism): + self.findings.append( + { + "check": check, + "level": level, + "path": self.rel(path) if path else "", + "line": line or 0, + "summary": summary, + "mechanism": mechanism, + } + ) + + def error(self, check, path, line, summary, mechanism): + self.add(check, ERROR, path, line, summary, mechanism) + + def warn(self, check, path, line, summary, mechanism): + self.add(check, WARN, path, line, summary, mechanism) + + def unavailable(self, check, summary): + self.add(check, INFO, "", 0, summary, "") + + def rel(self, path): + try: + return os.path.relpath(path, self.project_dir) + except ValueError: + return path + + def counts(self): + errors = sum(1 for finding in self.findings if finding["level"] == ERROR) + warnings = sum(1 for finding in self.findings if finding["level"] == WARN) + return errors, warnings diff --git a/.claude/skills/porting-to-canyonos/validation/dependencies.py b/.claude/skills/porting-to-canyonos/validation/dependencies.py new file mode 100644 index 0000000..4060319 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/dependencies.py @@ -0,0 +1,207 @@ +"""W003, W006 -- credentials and imports a successful build does not reject.""" + +import ast +import os +import re + +from validation.python_source import ( + parse_python, + reachable_imports, + resolves_flat, + resolves_nested, +) +from validation.runtime import ( + IMPORT_TO_DISTRIBUTION, + NAMESPACE_DISTRIBUTIONS, + RUNTIME_FLAT_NAMES, + STDLIB_MODULE_NAMES, +) + +SECRET_PATTERNS = [ + (re.compile(r"sk-[A-Za-z0-9_-]{20,}"), "an OpenAI-style secret key"), + (re.compile(r"AKIA[0-9A-Z]{16}"), "an AWS access key id"), + (re.compile(r"gh[pousr]_[A-Za-z0-9]{20,}"), "a GitHub token"), + (re.compile(r"AIza[0-9A-Za-z_-]{30,}"), "a Google API key"), +] + + +SECRET_NAME = re.compile(r"(API_KEY|SECRET|TOKEN|PASSWORD|CREDENTIAL)", re.IGNORECASE) + + +def check_secrets(report, port_paths): + """W003 -- env_file is the way in; nothing else is.""" + for path in port_paths: + try: + with open(path, "r", encoding="utf-8") as handle: + lines = handle.read().splitlines() + except OSError: + continue + for number, line in enumerate(lines, start=1): + for pattern, description in SECRET_PATTERNS: + if pattern.search(line): + report.error( + "W003", + path, + number, + f"this line looks like {description}", + "Never put a secret in the source tree or the build " + "context. The build sweeps the project into every " + "image.", + ) + break + + tree, _ = parse_python(path) + if tree is None: + continue + for node in ast.walk(tree): + if not isinstance(node, ast.Assign): + continue + if not ( + isinstance(node.value, ast.Constant) + and isinstance(node.value.value, str) + and node.value.value.strip() + ): + continue + for target in node.targets: + if isinstance(target, ast.Name) and SECRET_NAME.search(target.id): + report.warn( + "W003", + path, + node.lineno, + f"`{target.id}` is assigned a literal string", + "Read it from the environment instead; the build sweeps " + "this file into every image.", + ) + + +def _pyproject_dependencies(project_dir): + """What `-e .` installs alongside `requirements:`, or None if unreadable. + + None and the empty set mean different things here: empty means the project + declares no dependencies, None means we could not find out -- a setup.py, or + a tomllib this interpreter does not have. The caller must not treat the + second as the first, or it warns about imports the install would satisfy. + """ + path = os.path.join(project_dir, "pyproject.toml") + if not os.path.isfile(path): + return None + try: + import tomllib + except ImportError: # < 3.11 + return None + try: + with open(path, "rb") as handle: + data = tomllib.load(handle) + except Exception: # noqa: BLE001 - malformed metadata is uv's error to give + return None + deps = (data.get("project") or {}).get("dependencies") + if not isinstance(deps, list): + return None + return {_normalize_distribution(d) for d in deps if isinstance(d, str)} + + +def check_requirements_coverage( + report, + project_dir, + entry, + root_path, + config_path, + base_requirements, + shadowed_paths=(), +): + """W006 -- an import the container cannot satisfy. + + Walks the whole import graph the image executes from `root_path`, not just + that one file: a distribution reached through a local module or a package + __init__ is exactly as missing, and exactly as invisible until the container + starts. + """ + declared = { + _normalize_distribution(item) + for item in (entry.get("requirements") or []) + if isinstance(item, str) + } + + # Where the editable install exists, `-e .` resolves the project's own + # [project.dependencies] in the same pass as `requirements:`. Warning about + # those is a false positive, and a false warning about a dependency is worse + # than none: it teaches the reader to dismiss this check. + editable = report.capabilities.get("editable_install") + metadata = any( + os.path.isfile(os.path.join(project_dir, name)) + for name in ("pyproject.toml", "setup.py", "setup.cfg") + ) + unreadable_metadata = False + if editable and metadata: + project_deps = _pyproject_dependencies(project_dir) + if project_deps is None: + unreadable_metadata = True + else: + declared |= project_deps + base = {_normalize_distribution(item) for item in base_requirements} + satisfied = base | declared + + external = reachable_imports(project_dir, root_path, shadowed_paths) + for dotted, (where, lineno) in sorted(external.items()): + name = dotted.split(".")[0] + if name in STDLIB_MODULE_NAMES or name == "ventis": + continue + # Provided by the image itself: the shared runtime is copied flat over + # the swept tree. A stub is not listed here -- it replaces a module the + # source copy already carries, so the tree checks below cover it. + if f"{name}.py" in RUNTIME_FLAT_NAMES: + continue + if resolves_flat(project_dir, name) or resolves_nested(project_dir, name): + continue + prefix = NAMESPACE_DISTRIBUTIONS.get(name) + if prefix and any( + item == prefix or item.startswith(prefix + "-") for item in declared + ): + continue + if _candidate_distributions(name) & satisfied: + continue + if unreadable_metadata: + mechanism = ( + "The container installs the base list, `requirements:`, and -- " + "since this project declares packaging metadata -- whatever " + "`-e .` resolves from it. That metadata could not be read here, " + f"so if it already requires `{name}` this line is noise; " + "otherwise it is a ModuleNotFoundError inside _load_agent and " + "'No agent loaded' on the first request." + ) + else: + mechanism = ( + "The container installs the base list plus `requirements:` and " + "nothing else, so this is a ModuleNotFoundError inside " + "_load_agent and 'No agent loaded' on the first request. If the " + f"distribution is named something other than `{name}`, declare " + f"that name in {report.rel(config_path)}." + ) + if os.path.realpath(where) != os.path.realpath(root_path): + mechanism += ( + f" This image never names `{name}` in {report.rel(root_path)}; " + f"it runs {report.rel(where)} on the way there, and that module " + "needs it." + ) + report.error( + "W006", + where, + lineno, + f"`import {dotted}` is in neither the runtime's base list nor " + f"{entry.get('name') or 'this entry'}'s `requirements:`", + mechanism, + ) + + +def _candidate_distributions(name): + """Every distribution name that would satisfy `import `.""" + return { + _normalize_distribution(item) + for item in IMPORT_TO_DISTRIBUTION.get(name, (name,)) + } + + +def _normalize_distribution(name): + return re.split(r"[<>=!\[;\s]", name.strip().lower(), maxsplit=1)[0].replace( + "_", "-" + ) diff --git a/.claude/skills/porting-to-canyonos/validation/entrypoint.py b/.claude/skills/porting-to-canyonos/validation/entrypoint.py new file mode 100644 index 0000000..902c3f8 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/entrypoint.py @@ -0,0 +1,141 @@ +"""V019, V020, V033-V035 -- traps set by which module the entrypoint names.""" + +import ast +import os + +from validation.python_source import module_path, parse_python +from validation.runtime import RUNTIME_FLAT_NAMES + + +def check_flat_collisions(report, source_dir, entrypoints): + """V019 V020 -- later copies land on earlier ones at the context root.""" + for entry in sorted(os.listdir(source_dir)): + path = os.path.join(source_dir, entry) + if not os.path.isfile(path) or not entry.endswith(".py"): + continue + + # V019 -- the shared runtime is copied flat, after the project sweep. + if entry in RUNTIME_FLAT_NAMES: + report.error( + "V019", + path, + 1, + f"a project module named `{entry}` sits at the root of the source copy", + "The shared CanyonOS Core runtime is copied flat into the image after " + "the project sweep, so this file is overwritten by CanyonOS Core's own " + f"{entry}. Rename it or move it into a package directory.", + ) + + # V020 -- one module cannot be the entrypoint of two agents. + owners = {} + for name, entrypoint in entrypoints: + owners.setdefault(entrypoint, []).append(name) + for entrypoint, names in sorted(owners.items()): + if len(names) < 2: + continue + report.error( + "V020", + os.path.join(source_dir, entrypoint), + 1, + f"{' and '.join(sorted(names))} both declare `{entrypoint}` as their " + "entrypoint", + "Each agent's stub is written over its own entrypoint, so the two " + "land on one path and the last one built wins. Every caller then " + "reaches whichever agent that was. Give each agent its own module.", + ) + + +def check_entrypoint_module(report, source_dir, name, entrypoint): + """V033 V034 V035. + + Two runtime facts collide here. The build writes this agent's stub over + `entrypoint` in every image except this agent's own, and the controller + loads the real file by path rather than by import. Each breaks a module + layout that is correct everywhere else in Python. + """ + path = os.path.join(source_dir, entrypoint) + if not os.path.isfile(path): + return + + segments = os.path.splitext(entrypoint)[0].replace("\\", "/").split("/") + invalid = [part for part in segments if not part.isidentifier()] + if invalid: + report.error( + "V034", + path, + 0, + f"`{invalid[0]}` in the entrypoint path is not a Python identifier", + "The controller loads the entrypoint by file path, so this file runs " + "-- but the workflow has to import the class from " + f"`{module_path(entrypoint)}` (V023), and that is a SyntaxError, not " + "an ImportError. Rename the file inside the copy, or point " + "`entrypoint` at a normally-named sibling that loads this file by " + "path and re-exposes the class.", + ) + + tree, _ = parse_python(path) + if tree is not None: + for node in ast.walk(tree): + if isinstance(node, ast.ImportFrom) and node.level: + spelling = "." * node.level + (node.module or "") + report.error( + "V035", + path, + node.lineno, + f"the entrypoint's own `from {spelling} import ...` is relative", + "_load_agent loads this file with spec_from_file_location(" + "VENTIS_AGENT_FILE.replace('.py', ''), path). That name keeps " + "the entrypoint's directory separator, so it has no parent " + "package and __package__ is empty: every relative import in " + "this file raises 'attempted relative import with no known " + "parent package' at agent load, behind 'No agent loaded'. " + "Make this file's own top-level imports absolute; modules it " + "imports may keep theirs.", + ) + break + + directory = os.path.dirname(entrypoint) + if not directory: + return + init_path = os.path.join(source_dir, directory, "__init__.py") + if not os.path.isfile(init_path): + return + module = os.path.splitext(os.path.basename(entrypoint))[0] + package = directory.replace("\\", "/").replace("/", ".") + init_tree, _ = parse_python(init_path) + if init_tree is None: + return + for node in ast.walk(init_tree): + if not isinstance(node, ast.ImportFrom): + continue + target = node.module or "" + hit = ( + target == module + if node.level + else target + in ( + module, + f"{package}.{module}", + ) + ) + if not hit and node.level and not node.module: + hit = any(alias.name == module for alias in node.names) + if not hit: + continue + report.error( + "V033", + init_path, + node.lineno, + f"`{package}/__init__.py` re-exports from `{module}`, the entrypoint " + f"for {name}", + "Python runs a package's __init__.py before any of its submodules, " + "and in every image except this agent's own the module at the " + "entrypoint is the generated stub, which defines the agent class and " + f"nothing else. Any peer image that imports anything from `{package}` " + "-- the workflow importing the agent class included -- re-runs this " + "re-export against the stub and dies at container startup with " + f"ImportError. Point `entrypoint` at a module `{package}/__init__.py` " + "does not re-export from; add one that imports the real module if " + "every existing module is re-exported.", + ) + break diff --git a/.claude/skills/porting-to-canyonos/validation/manifest.py b/.claude/skills/porting-to-canyonos/validation/manifest.py new file mode 100644 index 0000000..a69b7d4 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/manifest.py @@ -0,0 +1,294 @@ +"""Fail-closed checks for the public CanyonOS artifact contract.""" + +import glob +import os + +from validation.core import line_of, load_yaml + + +def _safe_relative_python_path(value): + if not isinstance(value, str) or not value.strip(): + return False + normalized = value.replace("\\", "/") + return ( + not normalized.startswith("/") + and ".." not in normalized.split("/") + and normalized.endswith(".py") + ) + + +def check_manifest_structure(report, config, config_path, source_dir): + """Return entries only when deeper checks can traverse them safely.""" + entries = config.get("agents") + if not isinstance(entries, list): + report.error( + "V001", + config_path, + line_of(config, "agents"), + "`agents:` must be a list", + "The CanyonOS manifest cannot be traversed or built without an agents list.", + ) + return None + + valid = True + agent_count = sum( + 1 + for entry in entries + if isinstance(entry, dict) and entry.get("type", "agent") == "agent" + ) + workflow_count = sum( + 1 + for entry in entries + if isinstance(entry, dict) and entry.get("type", "agent") == "workflow" + ) + if agent_count < 1: + report.error( + "V002", + config_path, + line_of(config, "agents"), + "the manifest must contain at least one agent service", + "A CanyonOS port needs a callable service behind its workflow.", + ) + valid = False + if workflow_count != 1: + report.error( + "V002", + config_path, + line_of(config, "agents"), + f"the manifest must contain exactly one workflow service; found {workflow_count}", + "The deployment exposes one `/main` workflow and builds one workflow image.", + ) + valid = False + for index, entry in enumerate(entries): + if not isinstance(entry, dict): + report.error( + "V002", + config_path, + 0, + f"agents[{index}] must be a mapping", + "CanyonOS reads each agents item as a service declaration.", + ) + valid = False + continue + + name = entry.get("name") + if not isinstance(name, str) or not name.strip(): + report.error( + "V002", + config_path, + line_of(entry, "name"), + f"agents[{index}] has no non-empty string `name`", + "Names bind manifest entries, declarations, generated stubs, and images.", + ) + valid = False + else: + earlier = [ + item.get("name") + for item in entries[:index] + if isinstance(item, dict) and isinstance(item.get("name"), str) + ] + collision = next( + (other for other in earlier if other.lower() == name.lower()), None + ) + if collision is not None: + report.error( + "V002", + config_path, + line_of(entry, "name"), + f"`{name}` collides with `{collision}` after lowercase normalization", + "CanyonOS uses lowercase image and target names, so one " + "service overwrites the other.", + ) + valid = False + + service_type = entry.get("type", "agent") + if service_type not in ("agent", "workflow"): + report.error( + "V002", + config_path, + line_of(entry, "type"), + f"`type: {service_type}` is neither `agent` nor `workflow`", + "Only those two service shapes have a CanyonOS build contract.", + ) + valid = False + + provider = entry.get("provider", "local") + if provider not in ("local", "EC2"): + report.error( + "V002", + config_path, + line_of(entry, "provider"), + f"unsupported provider spelling `{provider}`", + "Use lowercase `local` or uppercase `EC2`; runtime provider " + "handling is case-sensitive.", + ) + valid = False + + replicas = entry.get("replicas", 1) + if isinstance(replicas, bool) or not isinstance(replicas, int) or replicas < 1: + report.error( + "V002", + config_path, + line_of(entry, "replicas"), + "`replicas` must be an integer greater than zero", + "CanyonOS creates one placement per replica and cannot deploy " + "an empty or fractional set.", + ) + valid = False + + requirements = entry.get("requirements", []) + if not isinstance(requirements, list) or not all( + isinstance(item, str) and item.strip() for item in requirements + ): + report.error( + "V002", + config_path, + line_of(entry, "requirements"), + "`requirements` must be a list of non-empty strings", + "CanyonOS writes this list into the image requirements file.", + ) + valid = False + + path_key = "workflow_file" if service_type == "workflow" else "entrypoint" + relative = entry.get(path_key) + if not _safe_relative_python_path(relative): + report.error( + "V002", + config_path, + line_of(entry, path_key), + f"`{path_key}` must be a relative .py path contained by `.car/app`", + "Absolute and parent-relative paths escape the self-contained CanyonOS artifact.", + ) + valid = False + elif not os.path.isfile(os.path.join(source_dir, relative)): + report.error( + "V002", + config_path, + line_of(entry, path_key), + f"`{path_key}: {relative}` does not exist in `.car/app`", + "The deploy build cannot create this service without its Python entry file.", + ) + valid = False + + return entries if valid else None + + +def check_self_contained_tree(report, artifact_dir): + """Reject symlinks because `.car` must not depend on outside state.""" + for root, directories, files in os.walk(artifact_dir, followlinks=False): + for name in [*directories, *files]: + path = os.path.join(root, name) + if not os.path.islink(path): + continue + report.error( + "V036", + path, + 0, + "`.car` contains a symbolic link", + "A symlink can escape the artifact or be skipped by the " + "Python-file sweep. Copy the intended file or directory into " + "the artifact explicitly.", + ) + + +def discover_agent_declarations(report, config_dir, config_path): + """Load declarations without silently discarding malformed or duplicate YAML.""" + declarations = {} + for path in sorted(glob.glob(os.path.join(config_dir, "*.yaml"))): + data, error = load_yaml(path) + if error is not None: + report.error( + "V003", + path, + 0, + f"YAML cannot be parsed: {error}", + "CanyonOS reads every YAML file in the config directory during deploy.", + ) + continue + if not isinstance(data, dict): + if os.path.realpath(path) == os.path.realpath(config_path): + report.error( + "V003", + path, + 0, + "the global manifest must be a mapping", + "A scalar or empty manifest has no CanyonOS configuration contract.", + ) + continue + agent = data.get("agent") + if agent is None: + continue + if ( + not isinstance(agent, dict) + or not isinstance(agent.get("name"), str) + or not agent["name"] + ): + report.error( + "V003", + path, + line_of(data, "agent"), + "`agent` must contain a non-empty string `name`", + "The declaration name is the binding used to generate its stub.", + ) + continue + name = agent["name"] + if name in declarations: + report.error( + "V003", + path, + line_of(agent, "name"), + f"duplicate declaration for `{name}`", + "Filename ordering would otherwise choose one declaration silently.", + ) + continue + declarations[name] = (path, agent) + return declarations + + +def check_declaration_bindings(report, entries, declarations, config_path): + """Require a one-to-one binding for every agent service.""" + configured = { + entry["name"] for entry in entries if entry.get("type", "agent") != "workflow" + } + for name in sorted(configured - declarations.keys()): + report.error( + "V004", + config_path, + 0, + f"agent `{name}` has no matching declaration in `.car/config`", + f"Without `agent.name: {name}`, CanyonOS cannot generate the service stub.", + ) + for name in sorted(declarations.keys() - configured): + path, _ = declarations[name] + report.warn( + "V004", + path, + 0, + f"declaration `{name}` has no agent service in the manifest", + "It is stale or unused and will not produce a deployable service.", + ) + + +def check_policy(report, config_dir): + path = os.path.join(config_dir, "policy.yaml") + if not os.path.exists(path): + return + data, error = load_yaml(path) + if error is not None or not isinstance(data, dict): + report.error( + "V005", + path, + 0, + "policy.yaml must be a YAML mapping", + error or "CanyonOS reads `rules` from this mapping during deploy.", + ) + return + rules = data.get("rules") + if not isinstance(rules, list) or not rules: + report.error( + "V005", + path, + line_of(data, "rules"), + "policy.yaml must contain a non-empty `rules` list", + "Remove the file for unrestricted access; an empty policy is not a valid restriction.", + ) diff --git a/.claude/skills/porting-to-canyonos/validation/packaging.py b/.claude/skills/porting-to-canyonos/validation/packaging.py new file mode 100644 index 0000000..d0682fc --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/packaging.py @@ -0,0 +1,108 @@ +"""V030-V031 -- capability-gated rules about credentials and import roots.""" + +import os + +from validation.core import line_of +from validation.python_source import ( + parse_python, + resolves_flat, + resolves_nested, + toplevel_import_names, +) +from validation.runtime import RUNTIME_FLAT_NAMES + + +def check_env_file(report, config, config_path, artifact_dir): + """V030 -- gated on detected env-file injection support.""" + declared = config.get("env_file") + supported = report.capabilities.get("env_file") + + if not supported: + if declared: + report.error( + "V030", + config_path, + line_of(config, "env_file"), + f"`env_file: {declared}` is set, but this CanyonOS Core never reads it", + "No resolve_env_file in the importable ventis package, so the " + "key is silently dropped and the container answers a provider " + "credential error on the first request. This port requires the " + "`env_file` runtime capability.", + ) + else: + report.unavailable( + "V030", + "env_file is not supported by the importable `ventis` runtime. " + "Credentials have no declared path into a container on this tree.", + ) + return + + if not declared: + report.warn( + "V030", + config_path, + line_of(config), + "no `env_file:` in the config", + "Only runtime-managed VENTIS_* variables are guaranteed without it. " + "If the source reads credentials from the environment, the first " + "request fails on a provider error.", + ) + return + + +def check_import_root(report, source_dir, entrypoint_paths): + """V031 -- gated on detected editable-install support.""" + supported = report.capabilities.get("editable_install") + has_metadata = any( + os.path.isfile(os.path.join(source_dir, name)) + for name in ("pyproject.toml", "setup.py", "setup.cfg") + ) + + non_flat = [] + for path in entrypoint_paths: + tree, _ = parse_python(path) + if tree is None: + continue + for name, lineno in toplevel_import_names(tree).items(): + if f"{name}.py" in RUNTIME_FLAT_NAMES: + continue + if resolves_flat(source_dir, name): + continue + location = resolves_nested(source_dir, name) + if location: + non_flat.append((path, lineno, name, location)) + + if not supported: + report.unavailable( + "V031", + "the editable install (`-e .`) is not supported by the importable " + "`ventis` runtime. Only names rooted at /app import inside a container.", + ) + for path, lineno, name, location in non_flat: + report.error( + "V031", + path, + lineno, + f"`import {name}` resolves to {location}, which is not at the " + "root of the source copy", + "sys.path[0] is /app and this CanyonOS Core runs no editable install, " + "so only modules swept to the root import. The adapter raises " + "ModuleNotFoundError inside _load_agent and the first request " + "answers 'No agent loaded'.", + ) + return + + if non_flat and not has_metadata: + for path, lineno, name, location in non_flat: + report.error( + "V031", + path, + lineno, + f"`import {name}` resolves to {location}, and the source copy's " + "root has no packaging metadata", + "A pyproject.toml, setup.py or setup.cfg at the root of the " + "source copy is what adds `-e .`; metadata nested deeper in the " + "tree is ignored. Add minimal root metadata pointing at the " + "existing package directory. Without it the install is skipped " + "silently.", + ) diff --git a/.claude/skills/porting-to-canyonos/validation/python_source.py b/.claude/skills/porting-to-canyonos/validation/python_source.py new file mode 100644 index 0000000..50df1e6 --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/python_source.py @@ -0,0 +1,179 @@ +"""Static Python-source discovery used by adapter and packaging checks.""" + +import ast +import os + + +def parse_python(path): + """Return ``(AST, None)`` or ``(None, error)`` without importing the file.""" + try: + with open(path, "r", encoding="utf-8") as handle: + source = handle.read() + except OSError as exc: + return None, str(exc) + try: + return ast.parse(source, filename=path), None + except SyntaxError as exc: + return None, f"{exc.msg} (line {exc.lineno})" + + +def find_class(tree, name): + for node in tree.body: + if isinstance(node, ast.ClassDef) and node.name == name: + return node + return None + + +def class_methods(class_node): + return { + node.name: node + for node in class_node.body + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) + } + + +def parameter_names(func_node): + """Return every keyword-callable parameter, excluding ``self``/``cls``.""" + args = func_node.args + positional = [arg.arg for arg in args.posonlyargs + args.args] + if positional and positional[0] in ("self", "cls"): + positional = positional[1:] + return positional + [arg.arg for arg in args.kwonlyargs] + + +def required_parameters(func_node): + """Return parameters without defaults, excluding ``self``/``cls``.""" + args = func_node.args + positional = [arg.arg for arg in args.posonlyargs + args.args] + if positional and positional[0] in ("self", "cls"): + positional = positional[1:] + if args.defaults: + positional = positional[: len(positional) - len(args.defaults)] + kwonly = [ + arg.arg + for arg, default in zip(args.kwonlyargs, args.kw_defaults) + if default is None + ] + return positional + kwonly + + +def toplevel_import_names(tree): + """Return top-level import names and their first line numbers.""" + names = {} + for node in ast.walk(tree): + if isinstance(node, ast.Import): + for alias in node.names: + names.setdefault(alias.name.split(".")[0], node.lineno) + elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module: + names.setdefault(node.module.split(".")[0], node.lineno) + return names + + +def dotted_import_names(tree): + """Return absolute dotted imports and their first line numbers.""" + names = {} + for node in ast.walk(tree): + if isinstance(node, ast.Import): + for alias in node.names: + names.setdefault(alias.name, node.lineno) + elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module: + names.setdefault(node.module, node.lineno) + return names + + +def _local_module_files(project_dir, dotted): + """Return local files executed by importing ``dotted``, outermost first.""" + parts = dotted.split(".") + found = [] + prefix = project_dir + for depth, part in enumerate(parts): + if os.path.isdir(os.path.join(prefix, part)): + init_path = os.path.join(prefix, part, "__init__.py") + if os.path.isfile(init_path): + found.append(init_path) + prefix = os.path.join(prefix, part) + continue + leaf = os.path.join(prefix, part + ".py") + if depth == len(parts) - 1 and os.path.isfile(leaf): + found.append(leaf) + return found + return found + + +def _relative_import_files(project_dir, path, tree): + """Return local files executed by a module's relative imports.""" + root = os.path.realpath(project_dir) + found = [] + for node in ast.walk(tree): + if not isinstance(node, ast.ImportFrom) or not node.level: + continue + base = os.path.dirname(path) + for _ in range(node.level - 1): + base = os.path.dirname(base) + resolved = os.path.realpath(base) + if resolved != root and not resolved.startswith(root + os.sep): + continue + target = os.path.join(base, *(node.module.split(".") if node.module else [])) + candidates = [target + ".py", os.path.join(target, "__init__.py")] + candidates += [os.path.join(target, alias.name + ".py") for alias in node.names] + candidates += [ + os.path.join(target, alias.name, "__init__.py") for alias in node.names + ] + found += [candidate for candidate in candidates if os.path.isfile(candidate)] + return found + + +def reachable_imports(project_dir, root_path, shadowed_paths=()): + """Return third-party imports reachable from ``root_path`` transitively.""" + external = {} + seen = set() + shadowed = {os.path.realpath(path) for path in shadowed_paths} + queue = [os.path.realpath(root_path)] + while queue: + path = queue.pop() + if path in shadowed or path in seen or not os.path.isfile(path): + continue + seen.add(path) + tree, _ = parse_python(path) + if tree is None: + continue + for dotted, lineno in dotted_import_names(tree).items(): + local = _local_module_files(project_dir, dotted) + if local: + queue += [os.path.realpath(item) for item in local] + else: + external.setdefault(dotted, (path, lineno)) + queue += [ + os.path.realpath(item) + for item in _relative_import_files(project_dir, path, tree) + ] + return external + + +def module_path(entrypoint): + """Dotted module name an entrypoint has inside the container.""" + return os.path.splitext(entrypoint)[0].replace("\\", "/").replace("/", ".") + + +def resolves_flat(project_dir, name): + """Whether Python can resolve `name` with /app as its import root. + + A directory does not need __init__.py: PEP 420 namespace packages resolve + from sys.path just like regular packages. + """ + return os.path.isfile(os.path.join(project_dir, f"{name}.py")) or os.path.isdir( + os.path.join(project_dir, name) + ) + + +def resolves_nested(project_dir, name): + """Where below /app `name` lives but cannot resolve as a top-level name.""" + for root, dirs, files in os.walk(project_dir): + dirs[:] = [d for d in dirs if not d.startswith(".") and d != "__pycache__"] + if root == project_dir: + continue + if f"{name}.py" in files: + return os.path.relpath(os.path.join(root, f"{name}.py"), project_dir) + if name in dirs: + return os.path.relpath(os.path.join(root, name), project_dir) + return None diff --git a/.claude/skills/porting-to-canyonos/validation/runtime.py b/.claude/skills/porting-to-canyonos/validation/runtime.py new file mode 100644 index 0000000..4ec438a --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/runtime.py @@ -0,0 +1,120 @@ +"""Runtime capabilities and dependency facts used by validation checks.""" + +import importlib +import os +import sys + + +RUNTIME_FLAT_NAMES = frozenset( + { + "future.py", + "ventis_context.py", + "local_controller.py", + "local_controller_frontend.py", + "redis_client.py", + "grpc_options.py", + "gpu_metrics.py", + "bedrock.py", + "deploy.py", + "session_logging.py", + "workflow_launcher.py", + } +) + +IMPORT_TO_DISTRIBUTION = { + "attr": ("attrs",), + "autogen": ("pyautogen", "ag2", "autogen", "autogen-agentchat"), + "bs4": ("beautifulsoup4",), + "cv2": ("opencv-python",), + "dateutil": ("python-dateutil",), + "dotenv": ("python-dotenv",), + "grpc": ("grpcio",), + "grpc_tools": ("grpcio-tools",), + "jwt": ("pyjwt",), + "PIL": ("pillow",), + "psycopg": ("psycopg",), + "psycopg2": ("psycopg2-binary",), + "pydantic_settings": ("pydantic-settings",), + "sklearn": ("scikit-learn",), + "typing_extensions": ("typing-extensions",), + "yaml": ("pyyaml",), +} + +NAMESPACE_DISTRIBUTIONS = {"llama_index": "llama-index"} + +CAPABILITY_SOURCE = { + "env_file": "runtime env-file injection", + "editable_install": "editable project installation", + "sweeps_all_files": "full project-file sweep", +} + + +def _base_requirements(): + agent = [ + "grpcio", + "grpcio-tools", + "redis", + "pyyaml", + "psutil", + "ipdb", + "ipython", + "boto3", + ] + workflow = [*agent, "flask", "sqlalchemy", "psycopg[binary]"] + try: + from ventis import stub_generator + except Exception: # noqa: BLE001 - a broken install must not crash validation + return agent, workflow + return ( + list(getattr(stub_generator, "BASE_AGENT_REQUIREMENTS", agent)), + list(getattr(stub_generator, "BASE_WORKFLOW_REQUIREMENTS", workflow)), + ) + + +def _stdlib_names(): + names = getattr(sys, "stdlib_module_names", None) + if names: + return frozenset(names) + found = set(sys.builtin_module_names) + library = os.path.dirname(os.__file__) + try: + entries = os.listdir(library) + except OSError: + return frozenset(found) + for entry in entries: + if entry.endswith(".py"): + found.add(entry[:-3]) + elif "." not in entry and "-" not in entry: + found.add(entry) + return frozenset(found) + + +BASE_AGENT_REQUIREMENTS, BASE_WORKFLOW_REQUIREMENTS = _base_requirements() +STDLIB_MODULE_NAMES = _stdlib_names() + + +def probe_capabilities(): + """Probe the installed compatibility runtime behind the CanyonOS CLI.""" + capabilities = dict.fromkeys(CAPABILITY_SOURCE, False) + capabilities["ventis"] = False + try: + from ventis import stub_generator + except Exception: # noqa: BLE001 - unavailable runtime is reported, not fatal + return capabilities + + capabilities["ventis"] = True + capabilities["editable_install"] = hasattr(stub_generator, "_install_step") + capabilities["sweeps_all_files"] = hasattr(stub_generator, "_sweep_project_files") + + for module_name in ( + "ventis.controller.utils.env_file", + "ventis.utils.env_file", + ): + try: + module = importlib.import_module(module_name) + except Exception: # noqa: BLE001 - try the other supported location + continue + if hasattr(module, "resolve_env_file"): + capabilities["env_file"] = True + break + return capabilities diff --git a/.claude/skills/porting-to-canyonos/validation/workflow.py b/.claude/skills/porting-to-canyonos/validation/workflow.py new file mode 100644 index 0000000..613c4ce --- /dev/null +++ b/.claude/skills/porting-to-canyonos/validation/workflow.py @@ -0,0 +1,213 @@ +"""V016-V018, V023 -- the workflow module and how it reaches an agent.""" + +import ast + +from validation.python_source import parameter_names, parse_python, required_parameters + + +def check_stub_imports(report, workflow_path, tree, stub_modules): + """V023 -- the workflow must import each agent from its own entrypoint module. + + The build writes a stub over exactly one path: the agent's `entrypoint` + inside the source copy. An import that reaches the class any other way -- + flat, through a package re-export, or from a second copy of the module -- + resolves to the real class instead, and the workflow runs the agent + in-process with none of the deployment behind it. The class name is another + trap: the deploy build prints one with a `Stub` suffix that it never writes. + """ + for node in ast.walk(tree): + if not isinstance(node, ast.ImportFrom) or not node.module: + continue + for alias in node.names: + name = alias.name + base = name.removesuffix("Stub") + expected = stub_modules.get(base) + if expected is None: + continue + if name.endswith("Stub"): + report.error( + "V023", + workflow_path, + node.lineno, + f"`{name}` is the name the build prints, not the class it writes", + "generate_stub sets class_name = agent_config['name'] and " + "then recomputes it with a 'Stub' suffix for the log line " + "only. The message names a class that does not exist; the " + f"class is `{base}`.", + ) + elif node.module != expected: + report.error( + "V023", + workflow_path, + node.lineno, + f"`from {node.module} import {name}` -- the stub for {name} " + f"is written to {expected.replace('.', '/')}.py", + "The build replaces the module at the agent's entrypoint " + "and nothing else, so this import reaches the real class " + "and runs the agent in this process instead of over gRPC. " + f"Import it from `{expected}`, where the source already " + "keeps it.", + ) + + +def check_workflow(report, workflow_path, stub_modules=None): + """V016 V017 V018 V023.""" + tree, error = parse_python(workflow_path) + if tree is None: + report.error("V016", workflow_path, 0, f"does not parse: {error}", "") + return + + main = None + for node in tree.body: + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and ( + node.name == "main" + ): + main = node + break + + if main is None: + defined = [ + n.name + for n in tree.body + if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef)) + ] + found = ", ".join(defined) if defined else "no top-level functions" + report.error( + "V016", + workflow_path, + 1, + f"no top-level function named `main` (found: {found})", + "CanyonOS Core serves POST /, but the deployment platform's " + "test endpoint posts to a hardcoded /main. A differently named " + "workflow builds, deploys and stays unreachable -- 404, container " + "healthy.", + ) + else: + check_main_signature(report, workflow_path, main) + + if stub_modules: + check_stub_imports(report, workflow_path, tree, stub_modules) + + # V016 -- deploy() is what starts Flask. + if not any( + isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "deploy" + for node in ast.walk(tree) + ): + report.error( + "V016", + workflow_path, + 1, + "the workflow never calls `deploy(...)`", + "workflow_launcher.py exec's this file and nothing else starts the " + "HTTP server; the container comes up serving nothing.", + ) + + check_main_guard(report, workflow_path, tree) + check_fused_fanout(report, workflow_path, tree) + + +def check_main_signature(report, workflow_path, main): + """V016 -- the platform sends exactly {"query": ...}.""" + if isinstance(main, ast.AsyncFunctionDef): + report.error( + "V016", + workflow_path, + main.lineno, + "`main` is `async def`", + "deploy() calls workflow_fn(**kwargs) on a Flask worker thread with " + "no await; the response body would be a coroutine repr.", + ) + params = parameter_names(main) + if not params: + report.error( + "V016", + workflow_path, + main.lineno, + "`main` takes no arguments", + 'The platform posts {"query": "..."} and deploy() splats the ' + "body in as kwargs -- TypeError on every request.", + ) + return + if params[0] != "query": + report.error( + "V016", + workflow_path, + main.lineno, + f"`main`'s first parameter is `{params[0]}`, not `query`", + "The platform's body schema is strictly validated as " + "{query: string}; any other key is rejected with 400 in the control " + "plane, before the request reaches the host.", + ) + extra = [p for p in required_parameters(main) if p != "query"] + if extra: + report.error( + "V016", + workflow_path, + main.lineno, + f"`main` requires {', '.join(extra)} beyond `query`", + "Only `query` is ever sent, so every other parameter needs a " + "default or the call raises on every request. Pack richer input " + "into `query`.", + ) + + +def check_main_guard(report, workflow_path, tree): + """V017 -- the workflow is exec'd, so __name__ == "__main__".""" + for node in tree.body: + if not isinstance(node, ast.If): + continue + test = node.test + if ( + isinstance(test, ast.Compare) + and isinstance(test.left, ast.Name) + and test.left.id == "__name__" + and any( + isinstance(c, ast.Constant) and c.value == "__main__" + for c in test.comparators + ) + ): + report.error( + "V017", + workflow_path, + node.lineno, + '`if __name__ == "__main__":` block in the workflow', + "workflow_launcher.py runs exec(open().read()), so " + "__name__ IS '__main__' here and this block executes in " + "production, at container start.", + ) + + +def check_fused_fanout(report, workflow_path, tree): + """V018 -- .value() blocks, so dispatching and resolving in one + comprehension runs the fan-out one call at a time.""" + comprehensions = (ast.ListComp, ast.SetComp, ast.GeneratorExp) + for node in ast.walk(tree): + if not isinstance(node, comprehensions + (ast.DictComp,)): + continue + elements = ( + [node.key, node.value] if isinstance(node, ast.DictComp) else [node.elt] + ) + for element in elements: + for inner in ast.walk(element): + if ( + isinstance(inner, ast.Call) + and isinstance(inner.func, ast.Attribute) + and inner.func.attr == "value" + and isinstance(inner.func.value, ast.Call) + ): + report.error( + "V018", + workflow_path, + node.lineno, + "one comprehension both dispatches a call and resolves " + "it with .value()", + ".value() blocks, so each call completes before the " + "next is dispatched. It does not error -- the fan-out " + "is just silently serial, and with it the reason to be " + "on CanyonOS Core. Dispatch every call first, then resolve: " + "futures = [a.work(i) for i in items] then " + "[f.value() for f in futures].", + ) + return