docs(networking): document gateway deployment

docs/docs/designs/lab-v2-core-network.md

@@ -113,10 +113,10 @@ bound to permanent hardware MAC addresses. Unnamed temporary clients use the
 documented dynamic pools. IncusOS seeds request DHCP on each node's 10GbE RJ45
 management interface; AMT independently requests DHCP on the 2.5GbE interface.
 
-Clients use their `gw01` VLAN gateway as the DNS resolver. `gw01` forwards
-`glab.lol` to its local mirror of the private Route 53 zone and sends other
-queries to configured recursive resolvers. The gateway does not serve the
-legacy `lab.gilman.io` zone.
+Clients use their `gw01` VLAN gateway as the DNS resolver. CoreDNS on `gw01`
+answers `glab.lol` from its local mirror of the private Route 53 zone and sends
+other queries to configured recursive resolvers. The gateway does not serve
+the legacy `lab.gilman.io` zone.
 
 This cold-start model assumes `gw01` is installed and its version-controlled
 network configuration is applied before managed machines boot.
@@ -155,18 +155,30 @@ port, direction, and owner in the version-controlled gateway policy.
 ## Configuration and Deployment
 
 `gw01`, `sw-core01`, and `sw-mgmt01` each have one version-controlled
-configuration source. A deployment:
-
-1. Renders the effective configuration.
-2. Validates syntax and policy.
-3. Shows the effective change for operator review.
-4. Applies the candidate without saving it as startup configuration.
-5. Verifies required connectivity and denied flows.
-6. Saves only after verification succeeds.
-7. Restores the previous configuration when verification fails.
-
-Drift detection compares each running configuration with its repository source.
-Legacy configuration is migration input, not a second source of truth.
+configuration source. The authoritative `gw01` source is the tracked
+configuration template and CoreDNS assets in `GilmanLab/networking`; encrypted
+inputs remain in `GilmanLab/secrets` and are rendered in memory.
+
+The `networking_vyos` package owns validation, rendering, locking, asset
+staging, and verification. Its `pyinfra-vyos` 0.1.0 boundary is the `Version`,
+redacted `ConfigurationCommands`, and `PendingSave` facts plus `config_load()`,
+`config()`, and `config_save()`. Pull-request CI validates the tracked template
+and assets without decrypting secrets, contacting the router, or deploying
+configuration.
+
+A sync validates local inputs, then acquires a fail-fast local lock that covers
+all remote stages. The lock serializes operators sharing one checkout;
+operators on separate controllers must coordinate. A read-only preflight
+requires no pending save before the sync stages the non-secret CoreDNS assets
+and loads a candidate containing no password hash. A separate generic
+configuration operation applies the hash before a fresh process verifies the
+running configuration. Only successful verification can call `config_save()`;
+a second fresh process must then report `PendingSave=False`.
+
+The [VyOS gateway deployment runbook](../runbooks/vyos-gateway-deployment.md)
+defines the operator commands, secret and SSH inputs, lock semantics,
+verification, and console-recovery procedure. A failed candidate verification
+never changes the boot configuration.
 
 ## Management and Recovery
 
@@ -179,7 +191,8 @@ The PiKVM and TESmart chain provides remote console access to the connected
 hosts. A local monitor, keyboard, and mouse attached to `pikvm01` remain the
 break-glass path when `gw01` or routed access is unavailable.
 
-Recovery credentials do not reside in committed device configuration.
+Plaintext recovery credentials and SSH private keys do not reside in committed
+device configuration.
 
 ## Failure Boundaries
 
@@ -195,14 +208,17 @@ Recovery credentials do not reside in committed device configuration.
 
 ## Delivery
 
-Migration from the current VyOS configuration removes VLAN 20, Tinkerbell
-firewall rules, PowerDNS, IncusOS artifact serving, `bootstrap-k0s`, BGP, and
-legacy UM760 bridge behavior. It renames the router to `gw01`, establishes the
-management-switch trunk, and applies the canonical DHCP reservations and
-firewall boundaries.
+The accepted `gw01` configuration is delivered from `GilmanLab/networking`
+through the `networking_vyos` operator commands. The tracked template removes
+VLAN 20, Tinkerbell firewall rules, PowerDNS, IncusOS artifact serving,
+`bootstrap-k0s`, BGP, and legacy UM760 bridge behavior. It names the router
+`gw01`, establishes the management-switch trunk, and applies the canonical DHCP
+reservations and firewall boundaries.
 
 The network may migrate one VLAN at a time. Preserve the routed transit and
-current OOB access until replacement paths pass verification.
+current OOB access until replacement paths pass verification. Legacy deployment
+material is migration input only; it is not an active Ansible deployment path
+or a second source of truth.
 
 ## Verification
 
@@ -222,8 +238,10 @@ A deployment is valid when:
 - each MS-02 retains management when its AMT link is disconnected and retains
   AMT when its management link is disconnected;
 - no BGP peers or retired gateway services remain;
-- a failed candidate deployment leaves or restores the previous startup
-  configuration.
+- a successful sync completes a fresh post-save check with
+  `PendingSave=False`; and
+- a failed candidate verification does not save the candidate, so a reboot
+  restores the previous startup configuration.
 
 ## Alternatives Considered
 

docs/docs/index.md

@@ -46,3 +46,6 @@ and runbook lives here.
 
 - [Change the Tailscale policy](runbooks/tailscale-policy-change.md) covers
   changing, verifying, reverting, and emergency-editing the tailnet policy.
+- [Deploy the VyOS gateway configuration](runbooks/vyos-gateway-deployment.md)
+  covers validation, inspection, guarded deployment, verification, and
+  console recovery for `gw01`.

docs/docs/reference/networking/address-plan.md

@@ -10,6 +10,11 @@ interface mapping, DHCP allocations, and logical switch port roles. The
 [physical connection map](physical-connections.md) remains authoritative for
 installed cables.
 
+Deploy changes to the gateway with the
+[VyOS gateway deployment runbook](../../runbooks/vyos-gateway-deployment.md).
+The runbook identifies the repository source and operator commands; this page
+remains the source for network values.
+
 ## Routed prefixes
 
 | Network | Prefix | Gateway | Purpose |
@@ -119,10 +124,9 @@ not required for initial IncusOS management boot.
 `gw01` is available before managed hosts boot and owns cold-start DHCP and DNS
 forwarding. Clients use their VLAN gateway address as the DNS resolver.
 
-`gw01` forwards `glab.lol` queries to the local mirror at `10.10.10.54`. The
-mirror copies the private Route 53 zone from the AWS subnet-router service.
-Other queries use the configured recursive resolvers. `gw01` does not run
-PowerDNS and does not serve `lab.gilman.io`.
+The CoreDNS service on `gw01` answers `glab.lol` from a local mirror of the
+private Route 53 zone and forwards other queries to the configured recursive
+resolvers. `gw01` does not run PowerDNS and does not serve `lab.gilman.io`.
 
 ## Routing protocols
 

docs/docs/runbooks/vyos-gateway-deployment.md

@@ -0,0 +1,212 @@
+---
+title: Deploy the VyOS gateway configuration
+description: Validate, inspect, deploy, verify, and recover the gw01 configuration.
+---
+
+# Deploy the VyOS gateway configuration
+
+Use this runbook to deploy the version-controlled `gw01` configuration from
+[`GilmanLab/networking`](https://github.com/GilmanLab/networking). The
+[Lab v2 core network design](../designs/lab-v2-core-network.md) defines the
+accepted behavior. The
+[network address and VLAN plan](../reference/networking/address-plan.md) is the
+canonical source for addresses, interfaces, and VLANs.
+
+## Preconditions
+
+- Use a reviewed checkout of `GilmanLab/networking`. Run every command below
+  from that checkout's root.
+- Install the repository's Moon toolchain and dependencies.
+- Have SSH access to `gw01` and a verified `known_hosts` entry for the target.
+  The deployment always uses strict host-key checking.
+- Have a separate checkout of the private `GilmanLab/secrets` repository and
+  SOPS decryption access for the `network-vyos` scope. Routine access uses the
+  KMS path described in
+  [ADR-0003](../decisions/0003-use-kms-with-pgp-recovery-for-secrets.md).
+- Confirm that no other operator or controller is deploying `gw01`. The
+  fail-fast lock serializes syncs only within the same networking checkout.
+- Have PiKVM or local-console access before changing routing, firewall, SSH, or
+  transit configuration.
+
+Pull requests run static validation only. Merging a change does not deploy it
+to `gw01`; an operator must run `moon run network:vyos-sync`.
+
+## Source and secret inputs
+
+The authoritative gateway source is:
+
+- [`vyos/gw01/config.boot.tmpl`](https://github.com/GilmanLab/networking/blob/master/vyos/gw01/config.boot.tmpl);
+- [`vyos/gw01/assets/coredns/Corefile`](https://github.com/GilmanLab/networking/blob/master/vyos/gw01/assets/coredns/Corefile); and
+- [`vyos/gw01/assets/scripts/dns-mirror-fetch-glab-lol.sh`](https://github.com/GilmanLab/networking/blob/master/vyos/gw01/assets/scripts/dns-mirror-fetch-glab-lol.sh).
+
+The [`networking_vyos` package](https://github.com/GilmanLab/networking/tree/master/src/networking_vyos)
+owns validation, secret handling, locking, staging, and verification. Its
+`pyinfra-vyos` 0.1.0 boundary consists of the `Version`, redacted
+`ConfigurationCommands`, and `PendingSave` facts plus `config_load()`,
+`config()`, and `config_save()`. No decrypted configuration file is written.
+
+If the secrets repository is not already present, clone it outside the public
+networking repository. Set `GLAB_SECRETS_DIR` to its absolute path:
+
+```bash
+mkdir -p "$HOME/code/glab"
+test -d "$HOME/code/glab/secrets/.git" || \
+  git clone git@github.com:GilmanLab/secrets.git "$HOME/code/glab/secrets"
+export GLAB_SECRETS_DIR="$HOME/code/glab/secrets"
+```
+
+`GLAB_SECRETS_DIR` has no default. The checkout must contain this encrypted
+input:
+
+| Path relative to `GLAB_SECRETS_DIR` | Required keys |
+| --- | --- |
+| `network/vyos/ssh.sops.yaml` | `public_key`, `password_hash` |
+
+Do not decrypt the file to disk or print its values. The full configuration
+load substitutes only `@@VYOS_PUBLIC_KEY@@`. A separate generic `config()`
+operation applies `password_hash` after the load, before verification, so a
+full-load failure cannot print the hash with the candidate configuration.
+
+Connection flags take precedence over environment variables, which take
+precedence over defaults:
+
+| Flag | Environment variable | Default |
+| --- | --- | --- |
+| `--host` | `VYOS_HOST` | `10.0.0.2` |
+| `--ssh-user` | `VYOS_SSH_USER` | `vyos` |
+| `--ssh-key` | `VYOS_SSH_KEY` | `~/.ssh/vyos-gateway` |
+| `--known-hosts` | `VYOS_KNOWN_HOSTS` | `~/.ssh/known_hosts` |
+
+Pass flags to the underlying module after Moon's `--`.
+
+`VYOS_SSH_KEY` points to the controller-local private-key file used for the SSH
+connection. It is separate from the encrypted `private_key`, which is retained
+for key distribution and recovery. Provision the local file with owner-only
+permissions before running the commands; the deployment does not write the
+decrypted private key to that path.
+
+For a non-default connection, set all changed values explicitly:
+
+```bash
+export VYOS_HOST=10.0.0.2
+export VYOS_SSH_USER=vyos
+export VYOS_SSH_KEY="$HOME/.ssh/vyos-gateway"
+export VYOS_KNOWN_HOSTS="$HOME/.ssh/known_hosts"
+```
+
+Confirm that the selected host-key file contains the independently verified
+`gw01` key:
+
+```bash
+ssh-keygen -F "$VYOS_HOST" -f "$VYOS_KNOWN_HOSTS"
+```
+
+Do not disable host-key checking to work around a missing or changed entry.
+Verify a changed fingerprint through PiKVM or the local console before updating
+`known_hosts`.
+
+## Validate and inspect
+
+1. Validate the tracked template, sentinels, and non-secret assets locally:
+
+   ```bash
+   moon run network:vyos-validate
+   ```
+
+   This static check does not read `GLAB_SECRETS_DIR` or contact `gw01`. Stop if
+   validation fails.
+
+2. Read the current router state:
+
+   ```bash
+   moon run network:vyos-facts
+   ```
+
+   The facts command is read-only and requests redacted VyOS configuration
+   output. It must report `pending_save: False`, derived from the
+   `PendingSave` fact, before a sync. Stop if the router has an existing unsaved
+   change.
+
+   Use JSON when another tool will consume the result:
+
+   ```bash
+   moon run network:vyos-facts -- --json
+   ```
+
+## Deploy
+
+1. Start the interactive sync:
+
+   ```bash
+   moon run network:vyos-sync
+   ```
+
+   Review the prompt before approving the change. For an already reviewed,
+   noninteractive invocation, use:
+
+   ```bash
+   moon run network:vyos-sync -- --yes
+   ```
+
+2. Wait for the command to finish. The sync performs this guarded sequence:
+
+   1. repeats local validation and decrypts the expected SSH values;
+   2. acquires a fail-fast local lock at `.moon/cache/gw01.lock` for all remote
+      stages;
+   3. runs a read-only preflight and requires the `PendingSave` fact to be
+      `False`;
+   4. stages the remaining non-secret CoreDNS assets;
+   5. creates the empty nftables compatibility chains required by the installed
+      VyOS rolling image, then loads and commits the secret-free candidate with
+      `config_load(..., save=False)`;
+   6. applies the console password hash with `config(..., save=False)`;
+   7. verifies the running candidate in a fresh process;
+   8. calls `config_save()` only after verification succeeds; and
+   9. performs a fresh final check that requires the `PendingSave` fact to be
+      `False`.
+
+   The lock is local to this checkout and controller. It does not exclude a
+   sync started from another controller, so operators must coordinate before
+   running the command. A failed verification never saves the candidate as the
+   boot configuration.
+
+## Verify
+
+Run a new facts process after a successful sync:
+
+```bash
+moon run network:vyos-facts -- --json
+```
+
+Confirm all of the following:
+
+- the JSON field `pending_save` is `false`;
+- the reported configuration remains redacted; and
+- the changed routing, DHCP, DNS, Tailscale, CoreDNS, or firewall behavior meets
+  the applicable criteria in the
+  [core network design](../designs/lab-v2-core-network.md#verification).
+
+## Failure and recovery
+
+If local validation, secret loading, lock acquisition, or the read-only
+preflight fails, fix the reported condition and run the procedure again. These
+failures occur before the candidate is loaded.
+
+If candidate verification fails after the load, do not save from the VyOS CLI.
+The rejected candidate can remain active in the running configuration, but the
+previous boot configuration is unchanged. If SSH or routed access is lost, use
+PiKVM or the local console to reboot `gw01`. The reboot restores the previous
+boot configuration.
+
+If `config_save()` fails or the final check does not report the `PendingSave`
+fact as `False`, do not assume which configuration will survive a reboot.
+Keep console access, inspect the running and boot configurations locally, and
+resolve the save state before another deployment.
+
+Escalate instead of retrying blindly when:
+
+- preflight reports an unsaved change you do not own;
+- the host key changes without an explained router reinstall or key rotation;
+- verification fails but the previous boot configuration does not recover
+  access; or
+- the post-save JSON check reports `"pending_save": true`.

docs/mkdocs.yml

@@ -47,6 +47,7 @@ nav:
           - Tailscale policy: reference/networking/tailscale-policy.md
   - Runbooks:
       - Change the Tailscale policy: runbooks/tailscale-policy-change.md
+      - Deploy the VyOS gateway configuration: runbooks/vyos-gateway-deployment.md
 
 plugins:
   - search