diff --git a/docs/docs/designs/lab-v2-core-network.md b/docs/docs/designs/lab-v2-core-network.md index 53c2651..5093d2b 100644 --- a/docs/docs/designs/lab-v2-core-network.md +++ b/docs/docs/designs/lab-v2-core-network.md @@ -113,10 +113,10 @@ bound to permanent hardware MAC addresses. Unnamed temporary clients use the documented dynamic pools. IncusOS seeds request DHCP on each node's 10GbE RJ45 management interface; AMT independently requests DHCP on the 2.5GbE interface. -Clients use their `gw01` VLAN gateway as the DNS resolver. `gw01` forwards -`glab.lol` to its local mirror of the private Route 53 zone and sends other -queries to configured recursive resolvers. The gateway does not serve the -legacy `lab.gilman.io` zone. +Clients use their `gw01` VLAN gateway as the DNS resolver. CoreDNS on `gw01` +answers `glab.lol` from its local mirror of the private Route 53 zone and sends +other queries to configured recursive resolvers. The gateway does not serve +the legacy `lab.gilman.io` zone. This cold-start model assumes `gw01` is installed and its version-controlled network configuration is applied before managed machines boot. @@ -155,18 +155,30 @@ port, direction, and owner in the version-controlled gateway policy. ## Configuration and Deployment `gw01`, `sw-core01`, and `sw-mgmt01` each have one version-controlled -configuration source. A deployment: - -1. Renders the effective configuration. -2. Validates syntax and policy. -3. Shows the effective change for operator review. -4. Applies the candidate without saving it as startup configuration. -5. Verifies required connectivity and denied flows. -6. Saves only after verification succeeds. -7. Restores the previous configuration when verification fails. - -Drift detection compares each running configuration with its repository source. -Legacy configuration is migration input, not a second source of truth. +configuration source. The authoritative `gw01` source is the tracked +configuration template and CoreDNS assets in `GilmanLab/networking`; encrypted +inputs remain in `GilmanLab/secrets` and are rendered in memory. + +The `networking_vyos` package owns validation, rendering, locking, asset +staging, and verification. Its `pyinfra-vyos` 0.1.0 boundary is the `Version`, +redacted `ConfigurationCommands`, and `PendingSave` facts plus `config_load()`, +`config()`, and `config_save()`. Pull-request CI validates the tracked template +and assets without decrypting secrets, contacting the router, or deploying +configuration. + +A sync validates local inputs, then acquires a fail-fast local lock that covers +all remote stages. The lock serializes operators sharing one checkout; +operators on separate controllers must coordinate. A read-only preflight +requires no pending save before the sync stages the non-secret CoreDNS assets +and loads a candidate containing no password hash. A separate generic +configuration operation applies the hash before a fresh process verifies the +running configuration. Only successful verification can call `config_save()`; +a second fresh process must then report `PendingSave=False`. + +The [VyOS gateway deployment runbook](../runbooks/vyos-gateway-deployment.md) +defines the operator commands, secret and SSH inputs, lock semantics, +verification, and console-recovery procedure. A failed candidate verification +never changes the boot configuration. ## Management and Recovery @@ -179,7 +191,8 @@ The PiKVM and TESmart chain provides remote console access to the connected hosts. A local monitor, keyboard, and mouse attached to `pikvm01` remain the break-glass path when `gw01` or routed access is unavailable. -Recovery credentials do not reside in committed device configuration. +Plaintext recovery credentials and SSH private keys do not reside in committed +device configuration. ## Failure Boundaries @@ -195,14 +208,17 @@ Recovery credentials do not reside in committed device configuration. ## Delivery -Migration from the current VyOS configuration removes VLAN 20, Tinkerbell -firewall rules, PowerDNS, IncusOS artifact serving, `bootstrap-k0s`, BGP, and -legacy UM760 bridge behavior. It renames the router to `gw01`, establishes the -management-switch trunk, and applies the canonical DHCP reservations and -firewall boundaries. +The accepted `gw01` configuration is delivered from `GilmanLab/networking` +through the `networking_vyos` operator commands. The tracked template removes +VLAN 20, Tinkerbell firewall rules, PowerDNS, IncusOS artifact serving, +`bootstrap-k0s`, BGP, and legacy UM760 bridge behavior. It names the router +`gw01`, establishes the management-switch trunk, and applies the canonical DHCP +reservations and firewall boundaries. The network may migrate one VLAN at a time. Preserve the routed transit and -current OOB access until replacement paths pass verification. +current OOB access until replacement paths pass verification. Legacy deployment +material is migration input only; it is not an active Ansible deployment path +or a second source of truth. ## Verification @@ -222,8 +238,10 @@ A deployment is valid when: - each MS-02 retains management when its AMT link is disconnected and retains AMT when its management link is disconnected; - no BGP peers or retired gateway services remain; -- a failed candidate deployment leaves or restores the previous startup - configuration. +- a successful sync completes a fresh post-save check with + `PendingSave=False`; and +- a failed candidate verification does not save the candidate, so a reboot + restores the previous startup configuration. ## Alternatives Considered diff --git a/docs/docs/index.md b/docs/docs/index.md index dd339b7..42df0c7 100644 --- a/docs/docs/index.md +++ b/docs/docs/index.md @@ -46,3 +46,6 @@ and runbook lives here. - [Change the Tailscale policy](runbooks/tailscale-policy-change.md) covers changing, verifying, reverting, and emergency-editing the tailnet policy. +- [Deploy the VyOS gateway configuration](runbooks/vyos-gateway-deployment.md) + covers validation, inspection, guarded deployment, verification, and + console recovery for `gw01`. diff --git a/docs/docs/reference/networking/address-plan.md b/docs/docs/reference/networking/address-plan.md index 9c5fcbf..9d53392 100644 --- a/docs/docs/reference/networking/address-plan.md +++ b/docs/docs/reference/networking/address-plan.md @@ -10,6 +10,11 @@ interface mapping, DHCP allocations, and logical switch port roles. The [physical connection map](physical-connections.md) remains authoritative for installed cables. +Deploy changes to the gateway with the +[VyOS gateway deployment runbook](../../runbooks/vyos-gateway-deployment.md). +The runbook identifies the repository source and operator commands; this page +remains the source for network values. + ## Routed prefixes | Network | Prefix | Gateway | Purpose | @@ -119,10 +124,9 @@ not required for initial IncusOS management boot. `gw01` is available before managed hosts boot and owns cold-start DHCP and DNS forwarding. Clients use their VLAN gateway address as the DNS resolver. -`gw01` forwards `glab.lol` queries to the local mirror at `10.10.10.54`. The -mirror copies the private Route 53 zone from the AWS subnet-router service. -Other queries use the configured recursive resolvers. `gw01` does not run -PowerDNS and does not serve `lab.gilman.io`. +The CoreDNS service on `gw01` answers `glab.lol` from a local mirror of the +private Route 53 zone and forwards other queries to the configured recursive +resolvers. `gw01` does not run PowerDNS and does not serve `lab.gilman.io`. ## Routing protocols diff --git a/docs/docs/runbooks/vyos-gateway-deployment.md b/docs/docs/runbooks/vyos-gateway-deployment.md new file mode 100644 index 0000000..5977a65 --- /dev/null +++ b/docs/docs/runbooks/vyos-gateway-deployment.md @@ -0,0 +1,212 @@ +--- +title: Deploy the VyOS gateway configuration +description: Validate, inspect, deploy, verify, and recover the gw01 configuration. +--- + +# Deploy the VyOS gateway configuration + +Use this runbook to deploy the version-controlled `gw01` configuration from +[`GilmanLab/networking`](https://github.com/GilmanLab/networking). The +[Lab v2 core network design](../designs/lab-v2-core-network.md) defines the +accepted behavior. The +[network address and VLAN plan](../reference/networking/address-plan.md) is the +canonical source for addresses, interfaces, and VLANs. + +## Preconditions + +- Use a reviewed checkout of `GilmanLab/networking`. Run every command below + from that checkout's root. +- Install the repository's Moon toolchain and dependencies. +- Have SSH access to `gw01` and a verified `known_hosts` entry for the target. + The deployment always uses strict host-key checking. +- Have a separate checkout of the private `GilmanLab/secrets` repository and + SOPS decryption access for the `network-vyos` scope. Routine access uses the + KMS path described in + [ADR-0003](../decisions/0003-use-kms-with-pgp-recovery-for-secrets.md). +- Confirm that no other operator or controller is deploying `gw01`. The + fail-fast lock serializes syncs only within the same networking checkout. +- Have PiKVM or local-console access before changing routing, firewall, SSH, or + transit configuration. + +Pull requests run static validation only. Merging a change does not deploy it +to `gw01`; an operator must run `moon run network:vyos-sync`. + +## Source and secret inputs + +The authoritative gateway source is: + +- [`vyos/gw01/config.boot.tmpl`](https://github.com/GilmanLab/networking/blob/master/vyos/gw01/config.boot.tmpl); +- [`vyos/gw01/assets/coredns/Corefile`](https://github.com/GilmanLab/networking/blob/master/vyos/gw01/assets/coredns/Corefile); and +- [`vyos/gw01/assets/scripts/dns-mirror-fetch-glab-lol.sh`](https://github.com/GilmanLab/networking/blob/master/vyos/gw01/assets/scripts/dns-mirror-fetch-glab-lol.sh). + +The [`networking_vyos` package](https://github.com/GilmanLab/networking/tree/master/src/networking_vyos) +owns validation, secret handling, locking, staging, and verification. Its +`pyinfra-vyos` 0.1.0 boundary consists of the `Version`, redacted +`ConfigurationCommands`, and `PendingSave` facts plus `config_load()`, +`config()`, and `config_save()`. No decrypted configuration file is written. + +If the secrets repository is not already present, clone it outside the public +networking repository. Set `GLAB_SECRETS_DIR` to its absolute path: + +```bash +mkdir -p "$HOME/code/glab" +test -d "$HOME/code/glab/secrets/.git" || \ + git clone git@github.com:GilmanLab/secrets.git "$HOME/code/glab/secrets" +export GLAB_SECRETS_DIR="$HOME/code/glab/secrets" +``` + +`GLAB_SECRETS_DIR` has no default. The checkout must contain this encrypted +input: + +| Path relative to `GLAB_SECRETS_DIR` | Required keys | +| --- | --- | +| `network/vyos/ssh.sops.yaml` | `public_key`, `password_hash` | + +Do not decrypt the file to disk or print its values. The full configuration +load substitutes only `@@VYOS_PUBLIC_KEY@@`. A separate generic `config()` +operation applies `password_hash` after the load, before verification, so a +full-load failure cannot print the hash with the candidate configuration. + +Connection flags take precedence over environment variables, which take +precedence over defaults: + +| Flag | Environment variable | Default | +| --- | --- | --- | +| `--host` | `VYOS_HOST` | `10.0.0.2` | +| `--ssh-user` | `VYOS_SSH_USER` | `vyos` | +| `--ssh-key` | `VYOS_SSH_KEY` | `~/.ssh/vyos-gateway` | +| `--known-hosts` | `VYOS_KNOWN_HOSTS` | `~/.ssh/known_hosts` | + +Pass flags to the underlying module after Moon's `--`. + +`VYOS_SSH_KEY` points to the controller-local private-key file used for the SSH +connection. It is separate from the encrypted `private_key`, which is retained +for key distribution and recovery. Provision the local file with owner-only +permissions before running the commands; the deployment does not write the +decrypted private key to that path. + +For a non-default connection, set all changed values explicitly: + +```bash +export VYOS_HOST=10.0.0.2 +export VYOS_SSH_USER=vyos +export VYOS_SSH_KEY="$HOME/.ssh/vyos-gateway" +export VYOS_KNOWN_HOSTS="$HOME/.ssh/known_hosts" +``` + +Confirm that the selected host-key file contains the independently verified +`gw01` key: + +```bash +ssh-keygen -F "$VYOS_HOST" -f "$VYOS_KNOWN_HOSTS" +``` + +Do not disable host-key checking to work around a missing or changed entry. +Verify a changed fingerprint through PiKVM or the local console before updating +`known_hosts`. + +## Validate and inspect + +1. Validate the tracked template, sentinels, and non-secret assets locally: + + ```bash + moon run network:vyos-validate + ``` + + This static check does not read `GLAB_SECRETS_DIR` or contact `gw01`. Stop if + validation fails. + +2. Read the current router state: + + ```bash + moon run network:vyos-facts + ``` + + The facts command is read-only and requests redacted VyOS configuration + output. It must report `pending_save: False`, derived from the + `PendingSave` fact, before a sync. Stop if the router has an existing unsaved + change. + + Use JSON when another tool will consume the result: + + ```bash + moon run network:vyos-facts -- --json + ``` + +## Deploy + +1. Start the interactive sync: + + ```bash + moon run network:vyos-sync + ``` + + Review the prompt before approving the change. For an already reviewed, + noninteractive invocation, use: + + ```bash + moon run network:vyos-sync -- --yes + ``` + +2. Wait for the command to finish. The sync performs this guarded sequence: + + 1. repeats local validation and decrypts the expected SSH values; + 2. acquires a fail-fast local lock at `.moon/cache/gw01.lock` for all remote + stages; + 3. runs a read-only preflight and requires the `PendingSave` fact to be + `False`; + 4. stages the remaining non-secret CoreDNS assets; + 5. creates the empty nftables compatibility chains required by the installed + VyOS rolling image, then loads and commits the secret-free candidate with + `config_load(..., save=False)`; + 6. applies the console password hash with `config(..., save=False)`; + 7. verifies the running candidate in a fresh process; + 8. calls `config_save()` only after verification succeeds; and + 9. performs a fresh final check that requires the `PendingSave` fact to be + `False`. + + The lock is local to this checkout and controller. It does not exclude a + sync started from another controller, so operators must coordinate before + running the command. A failed verification never saves the candidate as the + boot configuration. + +## Verify + +Run a new facts process after a successful sync: + +```bash +moon run network:vyos-facts -- --json +``` + +Confirm all of the following: + +- the JSON field `pending_save` is `false`; +- the reported configuration remains redacted; and +- the changed routing, DHCP, DNS, Tailscale, CoreDNS, or firewall behavior meets + the applicable criteria in the + [core network design](../designs/lab-v2-core-network.md#verification). + +## Failure and recovery + +If local validation, secret loading, lock acquisition, or the read-only +preflight fails, fix the reported condition and run the procedure again. These +failures occur before the candidate is loaded. + +If candidate verification fails after the load, do not save from the VyOS CLI. +The rejected candidate can remain active in the running configuration, but the +previous boot configuration is unchanged. If SSH or routed access is lost, use +PiKVM or the local console to reboot `gw01`. The reboot restores the previous +boot configuration. + +If `config_save()` fails or the final check does not report the `PendingSave` +fact as `False`, do not assume which configuration will survive a reboot. +Keep console access, inspect the running and boot configurations locally, and +resolve the save state before another deployment. + +Escalate instead of retrying blindly when: + +- preflight reports an unsaved change you do not own; +- the host key changes without an explained router reinstall or key rotation; +- verification fails but the previous boot configuration does not recover + access; or +- the post-save JSON check reports `"pending_save": true`. diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 61bb6ac..c59f590 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -47,6 +47,7 @@ nav: - Tailscale policy: reference/networking/tailscale-policy.md - Runbooks: - Change the Tailscale policy: runbooks/tailscale-policy-change.md + - Deploy the VyOS gateway configuration: runbooks/vyos-gateway-deployment.md plugins: - search