Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 113 additions & 0 deletions docs/docs/decisions/0004-manage-routeros-devices-with-opentofu.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
---
status: accepted
date: 2026-08-20
---

# ADR-0004: Manage RouterOS Devices with OpenTofu

## Context and Problem Statement

RouterOS configuration consists largely of addressable resources with stable
identities, but manual changes do not provide a reviewed desired state or a
repeatable drift check. Related GilmanLab repositories already establish
OpenTofu and remote-state conventions, while the networking repository's
`networking_vyos` package implements a deployment model specific to VyOS. How
should GilmanLab manage RouterOS devices?

## Decision Drivers

- Represent the RouterOS flat resource model as declarative resources.
- Use the established and actively maintained `terraform-routeros/routeros`
provider instead of implementing RouterOS discovery and reconciliation.
- Reuse the lab's OpenTofu, S3 state, locking, and operator-command
conventions.
- Isolate each device's credentials, state, and apply blast radius.
- Preserve a recovery path when RouterOS cannot automatically roll back an API
change that disconnects management.

## Considered Options

- Manage each RouterOS device in a separate OpenTofu root with the
`terraform-routeros/routeros` provider.
- Extend the `networking_vyos` pyinfra pattern with a RouterOS sibling package.
- Continue applying RouterOS configuration manually.

## Decision Outcome

Manage RouterOS devices declaratively with OpenTofu and the
`terraform-routeros/routeros` provider. `sw-core01` is the first managed device
and has the root `routeros/sw-core01/` in `GilmanLab/networking`. Manage `rtr01`
and the CCR2004 later as sibling device roots when they enter scope.

Each device root has an independent state key. For `sw-core01`, the key is
`networking/routeros/sw-core01.tfstate`. Shared modules may be introduced only
after multiple roots have demonstrated the same configuration need.

### Consequences

- Good, because RouterOS resources have a reviewable desired state and an
operator can use a plan to detect drift before a change.
- Good, because the provider supplies RouterOS resource discovery and
reconciliation rather than a new device-specific implementation.
- Good, because existing OpenTofu backend and locking conventions apply without
coupling multiple devices to one state file.
- Bad, because RouterOS REST changes have no commit-confirmed rollback. The
management-path resources must be adopted without changes, an operator must
download a sensitive export before every apply, and cutovers require
downloaded backups.
- Bad, because GitHub-hosted runners cannot reach the devices. CI can run only
offline format and validation checks; operators are responsible for live
plans and drift detection.
- Bad, because each additional RouterOS device adds a root, state key,
credential, and certificate lifecycle to operate.

### Confirmation

The implementation conforms to this decision when:

- each managed RouterOS device has one OpenTofu root in
`GilmanLab/networking/routeros/` and one isolated state key;
- the roots use `terraform-routeros/routeros` and do not use a RouterOS sibling
of `networking_vyos`;
- the `sw-core01` management bridge, trunk port, VLAN 10 row, management VLAN
interface, address, and default route are imported without replacement;
- every apply has a downloaded pre-apply export, and each cutover record
includes a downloaded backup;
- pull-request CI performs offline checks without device credentials or device
access; and
- an operator runs a live plan before every change and after each RouterOS
upgrade.

## Pros and Cons of the Options

### OpenTofu with the RouterOS Provider

- Good, because the configuration maps directly to provider resources and
produces a plan before mutation.
- Good, because per-device roots isolate state and applies.
- Good, because it reuses established S3 backend and locking conventions.
- Bad, because a provider operation that breaks the management path has no
automatic device-side rollback.
- Bad, because useful plans require live device access.

### Extend the pyinfra Pattern

- Good, because it would resemble the existing `networking_vyos` operator
workflow.
- Bad, because the VyOS package boundary and candidate-save behavior are
specific to VyOS.
- Bad, because GilmanLab would have to implement and maintain RouterOS resource
discovery, diffing, and idempotent mutation already supplied by the provider.

### Manual RouterOS Configuration

- Good, because it requires no state backend or provider.
- Bad, because the intended configuration, review history, and drift are not
represented as one declarative source.
- Bad, because repeated changes depend on operator command history.

## More Information

See the [Lab v2 core network design](../designs/lab-v2-core-network.md), the
[sw-core01 configuration runbook](../runbooks/sw-core01-configuration.md), and
the [network address and VLAN plan](../reference/networking/address-plan.md).
27 changes: 20 additions & 7 deletions docs/docs/designs/lab-v2-core-network.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,10 @@ status: accepted
authors:
- GilmanLab
created: 2026-08-14
updated: 2026-08-18
updated: 2026-08-20
related-decisions:
- ADR-0001
- ADR-0004
---

# Lab v2 core network
Expand Down Expand Up @@ -40,7 +41,7 @@ test and spike host outside the IncusOS cluster.

## Goals

- Give every bare-metal management and OOB endpoint a deterministic DHCP
- Give every bare-metal host management and OOB endpoint a deterministic DHCP
reservation.
- Keep management, OOB, and sandbox/workload traffic in separate VLANs.
- Keep routing, DHCP, DNS forwarding, traffic policy, and NAT on `gw01`.
Expand Down Expand Up @@ -108,10 +109,11 @@ locally when needed.
The [network address and VLAN plan](../reference/networking/address-plan.md)
is the single source for address and port values.

`gw01` provides DHCP on every client VLAN. Named endpoints use reservations
bound to permanent hardware MAC addresses. Unnamed temporary clients use the
documented dynamic pools. IncusOS seeds request DHCP on each node's 10GbE RJ45
management interface; AMT independently requests DHCP on the 2.5GbE interface.
`gw01` provides DHCP on every client VLAN. Named host endpoints use
reservations bound to permanent hardware MAC addresses. Network-device
management addresses are static interface addresses. IncusOS seeds request
DHCP on each node's 10GbE RJ45 management interface; AMT independently requests
DHCP on the 2.5GbE interface.

Clients use their `gw01` VLAN gateway as the DNS resolver. CoreDNS on `gw01`
answers `glab.lol` from its local mirror of the private Route 53 zone and sends
Expand Down Expand Up @@ -159,6 +161,17 @@ configuration source. The authoritative `gw01` source is the tracked
configuration template and CoreDNS assets in `GilmanLab/networking`; encrypted
inputs remain in `GilmanLab/secrets` and are rendered in memory.

The authoritative `sw-core01` source is the
[`routeros/sw-core01/`](https://github.com/GilmanLab/networking/tree/master/routeros/sw-core01)
OpenTofu root in `GilmanLab/networking`. An operator runs `just plan` and
`just apply` from that root on a workstation and downloads a configuration
snapshot before every apply. Pull-request CI runs only offline `fmt` and
`validate` checks because GitHub-hosted runners cannot reach the device.

The [sw-core01 configuration runbook](../runbooks/sw-core01-configuration.md)
defines the one-time bootstrap, snapshot discipline, adoption cutover, routine
operator flow, verification, and recovery procedure.

The `networking_vyos` package owns validation, rendering, locking, asset
staging, and verification. Its `pyinfra-vyos` 0.1.0 boundary is the `Version`,
redacted `ConfigurationCommands`, and `PendingSave` facts plus `config_load()`,
Expand Down Expand Up @@ -227,7 +240,7 @@ A deployment is valid when:
- every connected interface reports the expected link state and negotiated
speed;
- each VLAN appears only on its assigned access ports and trunks;
- each named endpoint receives its reserved address;
- each DHCP-managed named endpoint receives its reserved address;
- each client receives `gw01` as its default gateway and DNS resolver;
- `glab.lol` resolves through the local mirror while public DNS still resolves;
- the `rtr01` and `gw01` route tables contain the transit and lab routes;
Expand Down
5 changes: 5 additions & 0 deletions docs/docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ and runbook lives here.
- [ADR-0003: Use AWS KMS with PGP Recovery for Secrets](decisions/0003-use-kms-with-pgp-recovery-for-secrets.md)
uses scoped KMS access for routine decryption and a YubiKey-backed PGP key
for recovery outside AWS.
- [ADR-0004: Manage RouterOS Devices with OpenTofu](decisions/0004-manage-routeros-devices-with-opentofu.md)
uses an isolated OpenTofu root and state for each RouterOS device.

## Designs

Expand Down Expand Up @@ -49,6 +51,9 @@ and runbook lives here.
- [Rebuild nas01](runbooks/rebuild-nas01.md) reinstalls IncusOS on `nas01`
from git-defined install media and restores it as the cluster bootstrap
node.
- [Manage the sw-core01 configuration](runbooks/sw-core01-configuration.md)
covers bootstrap, adoption, routine OpenTofu changes, verification, and
recovery for the core switch.
- [Deploy the VyOS gateway configuration](runbooks/vyos-gateway-deployment.md)
covers validation, inspection, guarded deployment, verification, and
console recovery for `gw01`.
10 changes: 5 additions & 5 deletions docs/docs/reference/networking/address-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,10 +51,10 @@ required administration flows explicitly and permits established replies.
| Endpoint | Address | Allocation |
| --- | --- | --- |
| `gw01` management gateway | `10.10.10.1` | Interface address |
| `sw-core01` management | `10.10.10.2` | DHCP reservation |
| `sw-core01` management | `10.10.10.2` | Interface address |
| `gw01` sandbox/workload gateway | `10.10.40.1` | Interface address |
| `gw01` OOB gateway | `10.10.70.1` | Interface address |
| `sw-mgmt01` management | `10.10.70.2` | DHCP reservation |
| `sw-mgmt01` management | `10.10.70.2` | Interface address |
| `gw01` `glab.lol` mirror | `10.10.10.54` | Local service address |

### Hosts
Expand All @@ -69,9 +69,9 @@ required administration flows explicitly and permits established replies.
| `pikvm01` | — | `10.10.70.20` | Direct untagged attachment to `gw01` |
| `kvm01` | — | `10.10.70.21` | Direct untagged attachment to `gw01` |

`gw01` supplies DHCP on every client VLAN. Infrastructure and named hosts use
DHCP reservations; the gateway interface and local DNS mirror addresses are
static. Dynamic clients use `.200` through `.250` within each client VLAN.
`gw01` supplies DHCP on every client VLAN. Named hosts use DHCP reservations.
Gateway and managed-switch interface addresses and the local DNS mirror address
are static. Dynamic clients use `.200` through `.250` within each client VLAN.
Reservations use each endpoint's permanent hardware MAC address as recorded in
the version-controlled gateway configuration.

Expand Down
Loading