Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,15 @@ GITHUB_BASE_URL=https://github.com
# Leave empty to use the Incus default storage pool.
INCUS_STORAGE_POOL=

# Build-runner safety floor. The full image profile reserves 200 GiB.
MIN_BUILD_FREE_GIB=220
# Optional build disk floor. Zero uses the variant-specific manifest requirement.
MIN_BUILD_FREE_GIB=0

# The bootstrap installs the newest Incus candidate from repositories that the
# operator has already configured. It never adds or removes package sources.
MIN_INCUS_VERSION=6.0.5

# Reuse verified APT/npm/pip/Playwright downloads between clean build VMs.
VDM_BUILD_CACHE=true

# LLM gateway and local OpenAI-compatible inference endpoint.
VDM_LLM_GATEWAY_URL=
Expand Down
2 changes: 1 addition & 1 deletion .gitea/workflows/build-images.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ jobs:
run: ./tests/validate-repository.sh
- name: Verify Incus build runner
run: |
./scripts/ci/check-incus-runner.sh
./scripts/ci/check-incus-runner.sh "${{ matrix.variant }}"
./scripts/ci/check-architecture.sh "${{ matrix.architecture }}"
- name: Build image
run: ./scripts/build-image.sh "${{ matrix.variant }}"
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/build-images.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ jobs:
run: ./tests/validate-repository.sh
- name: Verify Incus build runner
run: |
./scripts/ci/check-incus-runner.sh
./scripts/ci/check-incus-runner.sh "${{ matrix.variant }}"
./scripts/ci/check-architecture.sh "${{ matrix.architecture }}"
- name: Build image
run: ./scripts/build-image.sh "${{ matrix.variant }}"
Expand Down
20 changes: 20 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,25 @@
# Changelog

## 0.3.0-rc.3 - 2026-07-29

- Separated adaptive image-build resources from launch-time runtime sizes,
reserving host memory/CPU and failing before Incus changes when current
capacity is insufficient.
- Added resumable stopped build checkpoints, non-secret failure diagnostics and
a detached managed cache for APT, npm, pip and Playwright downloads.
- Removed unnecessary build and first-launch snapshots; verified images now
publish directly from stopped instances.
- Moved bridge/ACL ownership to the Incus default project, added collision and
legacy-migration guards, made ACL state explicit and retained restricted
project isolation.
- Hardened host bootstrap around dpkg/APT health, active Incus upgrades and
latest configured package candidates without changing repositories,
coreutils providers, Docker or firewall policy.
- Fixed TSX version normalisation, replaced an undeclared Python `uv`
dependency with the installed `pipx`, and corrected the Joomla live-test
executable typo with an executed guest-dispatch regression test.
- Added complete host, hardware, caching, recovery and network documentation.

## 0.3.0-rc.2 - 2026-07-27

- Added `joomla-mcp` as a manifest-owned composable capability for PHP and full images.
Expand Down
17 changes: 17 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ cp .env.example .env
./tests/validate-repository.sh
./scripts/bootstrap-host.sh
./scripts/apply-incus.sh
./scripts/ci/check-incus-runner.sh php

export LOCAL_BUILD_VARIANTS=php
./scripts/local-image-release.sh
Expand All @@ -25,6 +26,21 @@ export LOCAL_BUILD_VARIANTS=php

An existing VM created from an older image does not gain newly installed packages. Launch or replace it from the new versioned PHP image.

`bootstrap-host.sh` first verifies APT/dpkg and the host's essential `cp`, `mv`
and `rm` providers. It installs the newest Incus candidate from repositories
already configured by the operator, but never adds/removes a package source,
changes the coreutils provider, or alters Docker firewall rules. An Incus
upgrade is refused while instances are running unless the operator explicitly
schedules and acknowledges that downtime.

Image builds use manifest-owned build limits, not the larger launch-time VM
sizes. The preflight reserves at least 20% of host memory, ignores swap as a
capacity source, leaves a host CPU available, and creates nothing when current
capacity is insufficient. Failed builds are stopped and retained as resumable
checkpoints with diagnostics under `build/diagnostics/`. Verified APT, npm,
pip and Playwright downloads are reused from a managed cache that is detached
before an image is published.

## Configure Joomla MCP

Configure one fixed public HTTPS Joomla origin. The default profile is read-only:
Expand Down Expand Up @@ -90,6 +106,7 @@ Start with:
- `docs/joomla-mcp-study.md`
- `docs/credentials-and-brokers.md`
- `docs/local-image-builds.md`
- `docs/host-and-network-safety.md`
- `docs/versioning-and-promotion.md`

## Licence and ownership
Expand Down
2 changes: 1 addition & 1 deletion VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
0.3.0-rc.2
0.3.0-rc.3
6 changes: 6 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,3 +50,9 @@ must be explicitly retained or pushed to CI artifact storage.

Use Incus projects for organisational separation, profiles for resource classes and versioned images for rollout.
For large central installations, add an Incus cluster and a dedicated image-builder project.

The restricted workload project shares two host-managed Incus bridge networks
from the `default` project (`features.networks=false`). This matches Incus
project isolation: the project may use only the two names listed by
`restricted.networks.access`, while network lifecycle remains with the host
administrator. No OVN or Open vSwitch service is required.
2 changes: 2 additions & 0 deletions docs/credentials-and-brokers.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ The host file is created by:
It lives under `/run/user/$UID/vdm-opencode`, must be a regular single-link file
owned by the caller with mode `0600`, and is never passed as a string of secret
values to `incus exec`.
`VDM_RUNTIME_BASE` may select another caller-owned mode-`0700` tmpfs mount;
non-tmpfs paths are rejected so credentials cannot be persisted accidentally.

This reduces persistence but does not hide a token from the process that must
use it. Therefore all injected tokens must be:
Expand Down
114 changes: 114 additions & 0 deletions docs/host-and-network-safety.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
# Host and network safety

This platform uses native Incus virtual machines and managed Linux bridge
networks. Host preparation is intentionally fail-closed: a repository command
must never repair the package manager by removing packages, replace a core
utility provider, rewrite an existing network, or inject firewall rules behind
the operator's back.

## Incus installation and upgrades

`scripts/bootstrap-host.sh` supports Ubuntu and Debian hosts. It runs
`apt-get update`, installs the newest `incus` candidate from repositories that
are already configured, and verifies the installed package and server version.
It does not add the Zabbly repository or any other package source.

Incus recommends an LTS branch for production. Ubuntu and Debian provide native
packages; Zabbly also provides supported current packages for listed
distributions. Selecting or changing that source is an operator-controlled host
policy, not something this repository automates.

Before APT changes anything, bootstrap verifies:

```bash
dpkg --audit
sudo apt-get check
test -x /usr/bin/cp
test -x /usr/bin/mv
test -x /usr/bin/rm
```

If the configured candidate would upgrade an active Incus daemon, the script
lists running instances and exits. Stop them during scheduled downtime, or set
`VDM_ALLOW_INCUS_UPGRADE_WITH_RUNNING_INSTANCES=true` for that one explicitly
approved run.

The platform never installs or removes `gnu-coreutils`,
`coreutils-from-uutils`, or an alternative `dd`; it never creates a systemd
`PATH` override for Incus.

## Network ownership

The workload project has:

```yaml
features.networks: "false"
restricted.networks.access: vdm-buildbr0,vdm-agentbr0
```

The bridges and their ACLs live in the Incus `default` project and carry:

```yaml
user.vdm.managed: "true"
user.vdm.platform: vdm-opencode-platform
```

`apply-incus.sh` refuses to overwrite a same-named network or ACL without those
markers. Before creating a bridge, it also rejects overlapping host routes for
`10.248.17.0/24` or `10.248.18.0/24`.

Older repository revisions created the bridges inside `vdm-agents`. The apply
script can migrate that layout only when the project contains no instances and
owns no unrelated networks or ACLs. Otherwise it stops and lists the blocker;
it never deletes an instance to force a migration.

All ACL rules explicitly use `state: enabled`, avoiding version-dependent empty
state parsing. Ingress stays empty and the NIC default ingress/egress actions
stay `reject`.

## Bridge networking, not OVN

Both networks are native Incus `bridge` networks with NAT, DHCP and DNS. This
layout does not install or require Open vSwitch, `ovn-central`, `ovn-host`, or
the OVN database sockets.

Verify the applied state:

```bash
incus project get vdm-agents features.networks
incus --project default network show vdm-buildbr0
incus --project default network show vdm-agentbr0
incus --project default network acl show vdm-build
```

The first command must print `false`. Each network and ACL must show
`user.vdm.platform: vdm-opencode-platform`.

## Docker coexistence

The repository does not modify Docker, nftables, iptables, UFW, or the
`DOCKER-USER` chain. Docker can set the global forwarding policy to `drop`,
which may block Incus bridge egress. Diagnose this separately:

```bash
sysctl net.ipv4.conf.all.forwarding
sudo iptables -S FORWARD
sudo iptables -S DOCKER-USER 2>/dev/null || true
```

Choose one of the Incus-documented host policies deliberately: configure
Docker's `ip-forward-no-drop`, enable forwarding before Docker starts, or add
persistent bridge-specific egress rules. Do not let a project bootstrap choose
between those system-wide policies automatically. See the
[Incus firewall documentation](https://linuxcontainers.org/incus/docs/main/howto/network_bridge_firewalld/#prevent-connectivity-issues-with-incus-and-docker).

## Publication and failure handling

The builder publishes the verified stopped instance directly. It does not make
a `sanitized` snapshot, and the launcher does not make a `factory` snapshot.
This avoids an unnecessary full raw-disk copy on the `dir` storage driver.

On failure, the exact managed build VM is stopped and retained. The next
identical build resumes it; `--clean` deletes only a checkpoint whose platform,
variant and fingerprint ownership metadata all match. No broad project or host
cleanup is performed.
5 changes: 3 additions & 2 deletions docs/image-variants.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,9 @@ instead of weakening the common ACL.
## Resource sizes

Tiny, small, medium, standard, large, xlarge and builder profiles are selected
at VM launch. They are independent of image composition, avoiding a
combinatorial image catalogue.
at VM launch. They are independent of image composition and of the separate
manifest-owned build plans. A `standard` 16 GiB runtime profile therefore does
not cause a temporary image builder to allocate 16 GiB.

## Adding support

Expand Down
3 changes: 3 additions & 0 deletions docs/known-limitations.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@

- Static and mocked validation is available, but the six real Incus images have
not yet been built in this delivery environment.
- Build-resource selection, resumable checkpoints and cache detachment are
covered by repository tests, but release promotion still requires a real
clean `--no-cache` build on two independent hardware-virtualisation hosts.
- JoomEngine MCP for Joomla `0.7.0` is now pinned, installed and fail-closed in
PHP/full images, but upstream still retains live-production evidence gates for
action families. This repository's safe helper proves only the configured
Expand Down
78 changes: 74 additions & 4 deletions docs/local-image-builds.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,18 +4,85 @@ This path builds, packages, verifies and optionally publishes Incus images entir

## One-time builder setup

Use a trusted Ubuntu host with hardware virtualisation, read/write access to `/dev/kvm`, Incus, and at least the configured `MIN_BUILD_FREE_GIB`.
Use a trusted Ubuntu or Debian host with hardware virtualisation, read/write
access to `/dev/kvm`, and a supported Incus package. The bootstrap installs the
newest candidate from the host's already configured repositories. Incus LTS is
recommended for production builders; changing from the distribution package to
another supported package source is a separate operator decision.

```bash
git clone https://github.com/vast-development-method/opencode-platform.git
cd opencode-platform
cp .env.example .env
./scripts/bootstrap-host.sh
./scripts/ci/check-incus-runner.sh
./scripts/ci/check-incus-runner.sh typescript
```

The builder does not need a GitHub Actions runner or a Gitea Actions runner.

`bootstrap-host.sh` is deliberately conservative:

- APT and dpkg must already be internally consistent.
- `/usr/bin/cp`, `/usr/bin/mv`, and `/usr/bin/rm` must exist and be executable.
- No package repository or coreutils provider is added, removed or replaced.
- If a newer Incus package would restart a daemon with running instances, the
command stops and lists those instances. Set
`VDM_ALLOW_INCUS_UPGRADE_WITH_RUNNING_INSTANCES=true` only during scheduled
downtime.

## Build capacity

Build resources are distinct from launch-time runtime sizes. The manifest
defines a minimum and preferred build plan for every variant. Before it creates
or starts a VM, the builder:

- keeps the greater of 20% of total RAM or 2 GiB available to the host;
- reserves an additional 512 MiB for QEMU overhead;
- leaves at least one online CPU to the host;
- checks current `MemAvailable`, not merely installed RAM;
- does not count swap;
- checks variant disk, cache and host reserve requirements.

| Variant | Minimum build | Preferred build | Virtual disk |
|---|---:|---:|---:|
| `base` | 2 CPUs / 3 GiB | 2 CPUs / 4 GiB | 40 GiB |
| `php` | 2 CPUs / 6 GiB | 4 CPUs / 8 GiB | 100 GiB |
| `python` | 2 CPUs / 4 GiB | 4 CPUs / 6 GiB | 60 GiB |
| `cpp` | 2 CPUs / 4 GiB | 4 CPUs / 6 GiB | 100 GiB |
| `typescript` | 2 CPUs / 6 GiB | 4 CPUs / 8 GiB | 80 GiB |
| `full` | 4 CPUs / 8 GiB | 6 CPUs / 12 GiB | 160 GiB |

A 14 GiB, 4-core/8-thread laptop is sufficient for TypeScript when other
workloads leave roughly 9.5 GiB available. If only 3.4 GiB is available, the
builder reports the exact shortfall and exits before applying Incus resources.
Closing other workloads is sufficient; adding swap is not a repository
requirement.

## Caching, resume and clean builds

The default managed `vdm-build-cache` volume retains only package/browser
downloads. Package managers still perform their normal signature and integrity
checks. The volume is detached before verification is published, so it is not
part of the image.

If a build fails, the VM is stopped rather than destroyed. Completed stages
resume when the same source and variant are run again, and a non-secret
diagnostic bundle is written under `build/diagnostics/`.

```bash
# Resume the exact retained checkpoint.
./scripts/build-image.sh typescript

# Discard that checkpoint but retain verified download caches.
./scripts/build-image.sh --clean typescript

# Independent clean-room evidence: no checkpoint and no shared cache.
./scripts/build-image.sh --clean --no-cache typescript
```

Successful builds delete their temporary VM automatically. The managed cache
remains bounded by the manifest.

## Build locally without publishing

Build all six variants:
Expand Down Expand Up @@ -69,7 +136,7 @@ For a reproducible production release, build the exact protected tag and require

```bash
git fetch --tags --prune origin
git switch --detach v0.3.0-rc.2
git switch --detach v0.3.0-rc.3

export LOCAL_BUILD_VARIANTS=all
export LOCAL_REQUIRE_CLEAN=true
Expand All @@ -87,7 +154,7 @@ export GITEA_PACKAGE_TOKEN
unset GITEA_PACKAGE_TOKEN
```

Replace `v0.3.0-rc.2` with the version being released. Tags must not be recreated or force-moved after packages are published.
Replace `v0.3.0-rc.3` with the version being released. Tags must not be recreated or force-moved after packages are published.

## Supported environment variables

Expand All @@ -101,6 +168,9 @@ Replace `v0.3.0-rc.2` with the version being released. Tags must not be recreate
| `GITEA_PACKAGE_OWNER` | unset | Gitea package owner. |
| `GITEA_PACKAGE_USER` | unset | Dedicated publishing user. |
| `GITEA_PACKAGE_TOKEN` | unset | Runtime-only package-write token. |
| `MIN_BUILD_FREE_GIB` | `0` | Optional disk floor; zero uses the variant-specific manifest requirement. |
| `VDM_BUILD_CACHE` | `true` | Reuse the managed package/browser download cache. |
| `MIN_INCUS_VERSION` | `6.0.5` | Oldest supported Incus server; bootstrap still installs the newest configured candidate. |

## Import and use

Expand Down
Loading