From 3ab99bb47f252aba11fb610f2b18f9d477bb0937 Mon Sep 17 00:00:00 2001 From: Joshua Gilman Date: Fri, 14 Aug 2026 17:09:31 -0700 Subject: [PATCH] docs: document switch topology --- ...os-for-layer-3-and-mikrotik-for-layer-2.md | 80 ---------------- ...os-for-layer-3-and-switches-for-layer-2.md | 93 +++++++++++++++++++ .../designs/drafts/lab-v2-core-network.md | 53 +++++++---- docs/docs/index.md | 9 +- docs/docs/reference/hardware.md | 14 ++- docs/mkdocs.yml | 2 +- 6 files changed, 144 insertions(+), 107 deletions(-) delete mode 100644 docs/docs/decisions/0001-use-vyos-for-layer-3-and-mikrotik-for-layer-2.md create mode 100644 docs/docs/decisions/0001-use-vyos-for-layer-3-and-switches-for-layer-2.md diff --git a/docs/docs/decisions/0001-use-vyos-for-layer-3-and-mikrotik-for-layer-2.md b/docs/docs/decisions/0001-use-vyos-for-layer-3-and-mikrotik-for-layer-2.md deleted file mode 100644 index a5a005b..0000000 --- a/docs/docs/decisions/0001-use-vyos-for-layer-3-and-mikrotik-for-layer-2.md +++ /dev/null @@ -1,80 +0,0 @@ ---- -status: accepted -date: 2026-08-14 ---- - -# ADR-0001: Use VyOS for Layer 3 and MikroTik for Layer 2 - -## Context and Problem Statement - -The core network needs explicit ownership for switching, routing, and traffic -policy. The network uses a VyOS gateway and a MikroTik switch. Which device owns -each network function? - -## Decision Drivers - -- Keep routed gateways and firewall policy on one device. -- Keep the switch configuration focused on VLAN transport and physical links. -- Make the enforcement point for traffic between routed lab segments explicit. -- Use the selected VyOS and MikroTik hardware. - -## Considered Options - -- Use VyOS for Layer 3 and MikroTik for Layer 2. -- Use MikroTik for Layer 2 and Layer 3, with VyOS at the external edge. -- Use one flat Layer 2 lab network, with VyOS as its external gateway. - -## Decision Outcome - -Use VyOS for routed lab gateways, route selection, firewall policy, and NAT. -Use MikroTik for Layer 2 switching, VLAN transport, and physical link -aggregation. - -DHCP, DNS, and time-service ownership are outside the scope of this decision. - -### Consequences - -- Good, because routed traffic has one policy-enforcement point. -- Good, because each device has a distinct configuration boundary. -- Bad, because traffic between routed lab segments depends on VyOS. -- Bad, because routed segments carried through the switch depend on the trunk - between VyOS and MikroTik. - -### Confirmation - -The implementation conforms to this decision when: - -- VyOS owns the gateway address for each routed lab segment. -- VyOS contains the firewall and NAT policy for routed lab traffic. -- MikroTik does not route traffic between lab segments. -- MikroTik configuration defines VLAN membership, trunks, access ports, and - physical link aggregation. - -A management address on the MikroTik switch does not violate this decision. - -## Pros and Cons of the Options - -### VyOS Layer 3 and MikroTik Layer 2 - -- Good, because routing and firewall policy use the same configuration - boundary. -- Good, because the switch remains independent of higher-level traffic policy. -- Bad, because VyOS is on the forwarding path for all routed lab traffic. - -### MikroTik Layer 2 and Layer 3 - -- Good, because the switch can route traffic without sending it through the - VyOS trunk. -- Bad, because firewall and routing ownership would be split between devices. -- Bad, because the network would need policy coordination between MikroTik and - VyOS. - -### Flat Layer 2 lab network - -- Good, because it requires fewer routed interfaces and policies. -- Bad, because it cannot enforce boundaries between lab network functions. -- Bad, because broadcasts and Layer 2 failures share one domain. - -## More Information - -See the [Lab v2 core network design](../designs/drafts/lab-v2-core-network.md). diff --git a/docs/docs/decisions/0001-use-vyos-for-layer-3-and-switches-for-layer-2.md b/docs/docs/decisions/0001-use-vyos-for-layer-3-and-switches-for-layer-2.md new file mode 100644 index 0000000..46a1a81 --- /dev/null +++ b/docs/docs/decisions/0001-use-vyos-for-layer-3-and-switches-for-layer-2.md @@ -0,0 +1,93 @@ +--- +status: accepted +date: 2026-08-14 +--- + +# ADR-0001: Use VyOS for Layer 3 and Dedicated Switches for Layer 2 + +## Context and Problem Statement + +The core network needs explicit ownership for switching, routing, and traffic +policy. The network uses a VyOS gateway, a MikroTik CRS309-1G-8S+IN core switch, +and a TRENDnet TEG-3102WS management/OOB switch. Which devices own each network +function? + +## Decision Drivers + +- Keep routed gateways and firewall policy on one device. +- Keep switch configuration focused on Layer 2 transport and physical links. +- Make the enforcement point for traffic between routed lab segments explicit. +- Carry MS-02 management/OOB traffic on its dedicated copper switch. +- Use the selected VyOS, MikroTik, and TRENDnet hardware. + +## Considered Options + +- Use VyOS for Layer 3 and dedicated switches for Layer 2. +- Use MikroTik for Layer 2 and Layer 3, with VyOS at the external edge. +- Use one flat Layer 2 lab network, with VyOS as its external gateway. + +## Decision Outcome + +Use VyOS for routed lab gateways, route selection, firewall policy, and NAT. Use +the MikroTik CRS309-1G-8S+IN for core Layer 2 switching and VLAN transport. Use +the TRENDnet TEG-3102WS for Layer 2 management/OOB connectivity from both +non-SFP NICs on each MS-02. The TEG-3102WS uplinks directly to the VP6630, which +provides the management/OOB gateway and firewall policy. + +DHCP, DNS, and time-service ownership are outside the scope of this decision. + +### Consequences + +- Good, because routed traffic has one policy-enforcement point. +- Good, because each device has a distinct configuration boundary. +- Good, because the MS-02 management/OOB links use a dedicated physical switch. +- Bad, because traffic between routed lab segments depends on VyOS. +- Bad, because routed segments carried through the CRS309-1G-8S+IN depend on + its trunk to VyOS. +- Bad, because MS-02 management/OOB access depends on the TEG-3102WS and its + uplink to the VP6630. + +### Confirmation + +The implementation conforms to this decision when: + +- VyOS owns the gateway address for each routed lab segment, including the + management/OOB segment. +- VyOS contains the firewall and NAT policy for routed lab traffic. +- Neither switch routes traffic between lab segments. +- CRS309-1G-8S+IN configuration defines core VLAN membership, trunks, access + ports, and physical link aggregation. +- TEG-3102WS configuration and cabling connect both non-SFP NICs from each MS-02 + to the VP6630 management/OOB gateway. + +Management addresses on the switches do not violate this decision. + +## Pros and Cons of the Options + +### VyOS Layer 3 and Dedicated Layer 2 Switches + +- Good, because routing and firewall policy use the same configuration + boundary. +- Good, because the switches remain independent of higher-level traffic policy. +- Good, because core and management/OOB traffic use separate physical switches. +- Bad, because VyOS is on the forwarding path for all routed lab traffic. +- Bad, because each switch is a failure boundary for its connected links. + +### MikroTik Layer 2 and Layer 3 + +- Good, because the core switch can route traffic without sending it through + the VyOS trunk. +- Bad, because firewall and routing ownership would be split between devices. +- Bad, because the network would need policy coordination between MikroTik and + VyOS. + +### Flat Layer 2 Lab Network + +- Good, because it requires fewer routed interfaces and policies. +- Bad, because it cannot enforce boundaries between lab network functions. +- Bad, because broadcasts and Layer 2 failures share one domain. + +## More Information + +See the [Lab v2 core network design](../designs/drafts/lab-v2-core-network.md) +and [hardware reference](../reference/hardware.md). diff --git a/docs/docs/designs/drafts/lab-v2-core-network.md b/docs/docs/designs/drafts/lab-v2-core-network.md index 11ba0ad..0d1fac0 100644 --- a/docs/docs/designs/drafts/lab-v2-core-network.md +++ b/docs/docs/designs/drafts/lab-v2-core-network.md @@ -14,9 +14,10 @@ related-decisions: ## Summary The core network uses a Minisforum VP6630 running VyOS for Layer 3 routing, -firewall policy, and NAT. A MikroTik switch handles Layer 2 switching and VLAN -transport. A MikroTik CCR2004 connects the lab to the home network and the -internet. +firewall policy, and NAT. A MikroTik CRS309-1G-8S+IN handles core Layer 2 +switching and VLAN transport. A TRENDnet TEG-3102WS connects both non-SFP NICs +from each MS-02 to the VP6630 for management/OOB traffic. A MikroTik CCR2004 +connects the lab to the home network and the internet. This design defines device responsibilities, logical topology, configuration requirements, failure boundaries, and verification criteria. Address @@ -26,7 +27,8 @@ ownership are outside this document. ## Goals - Keep routing and traffic policy on VyOS. -- Keep VLAN transport and physical switching on MikroTik. +- Keep core VLAN transport and physical switching on the CRS309-1G-8S+IN. +- Carry MS-02 management/OOB traffic through the TEG-3102WS. - Route home-to-lab traffic without source NAT. - Apply source NAT to lab-to-internet traffic on VyOS. - Store network-device configuration in version control. @@ -51,24 +53,28 @@ ownership are outside this document. flowchart LR HOME[Home network] --> CCR[CCR2004] CCR -->|Routed transit| VYOS[VP6630 running VyOS] - VYOS -->|802.1Q trunk| SWITCH[MikroTik switch] - SWITCH --> SEGMENTS[Lab network segments] + VYOS -->|802.1Q trunk| CRS[CRS309-1G-8S+IN] + CRS --> SEGMENTS[Lab network segments] + VYOS -->|Management/OOB uplink| TEG[TEG-3102WS] + TEG -->|Two non-SFP NICs per node| MS02[MS-02 nodes] ``` The CCR2004 routes traffic between the home network and the VyOS transit interface. VyOS routes lab prefixes, applies firewall policy, and performs -source NAT for internet egress. The MikroTik switch carries VLANs between VyOS -and connected lab devices. +source NAT for internet egress. The CRS309-1G-8S+IN carries lab VLANs between +VyOS and connected lab devices. The TEG-3102WS connects directly to the VP6630 +and carries management/OOB traffic for both non-SFP NICs on each MS-02. ## Device Responsibilities | Device | Responsibilities | | --- | --- | | MikroTik CCR2004 | Home-network routing, internet access, and the upstream side of the routed lab transit | -| Minisforum VP6630 running VyOS | Lab gateways, route selection, firewall policy, source NAT, and the downstream side of the routed transit | -| MikroTik switch | VLAN transport, access ports, trunks, and physical link aggregation | +| Minisforum VP6630 running VyOS | Lab gateways, route selection, firewall policy, source NAT, the downstream side of the routed transit, and the management/OOB gateway | +| MikroTik CRS309-1G-8S+IN | Core VLAN transport, access ports, trunks, and physical link aggregation | +| TRENDnet TEG-3102WS | Layer 2 management/OOB connectivity for both non-SFP NICs on each MS-02 and a direct uplink to the VP6630 | -[ADR-0001](../../decisions/0001-use-vyos-for-layer-3-and-mikrotik-for-layer-2.md) +[ADR-0001](../../decisions/0001-use-vyos-for-layer-3-and-switches-for-layer-2.md) defines the Layer 2 and Layer 3 boundary. ## Routing and NAT @@ -78,7 +84,7 @@ The routing design has these invariants: - The CCR2004 has routes for lab prefixes through the VyOS transit address. - VyOS uses the CCR2004 transit address as its default route. - VyOS owns the gateway address for every routed lab segment. -- MikroTik does not route between lab segments. +- The CRS309-1G-8S+IN and TEG-3102WS do not route between lab segments. - Home-to-lab traffic retains its original source address. - VyOS applies source NAT to lab-to-internet traffic. - Firewall rules distinguish new connections from established reply traffic. @@ -92,6 +98,7 @@ VyOS enforces policy for: - Lab segments to the internet - Traffic between routed lab segments - Traffic addressed to VyOS +- Management/OOB traffic through the TEG-3102WS - Management traffic addressed to network devices Each firewall rule identifies the source, destination, protocol, destination @@ -101,8 +108,8 @@ the reverse direction. ## Configuration Requirements -VyOS and MikroTik each have one version-controlled configuration source. The -deployment process: +VyOS and both switches each have one version-controlled configuration source. +The deployment process: 1. Renders the effective configuration. 2. Validates syntax and policy before deployment. @@ -117,20 +124,23 @@ Drift detection compares each running configuration with its repository source. ## Management and Recovery Firewall policy limits routine management access to approved source networks. +The TEG-3102WS carries management/OOB traffic from both non-SFP NICs on each +MS-02 directly to the VP6630. -Each device has a recovery path that remains available when its production -configuration or primary network link fails. Recovery credentials do not reside -in device configuration committed to the repository. +Each network device has a recovery path that remains available when its +production configuration or primary network link fails. Recovery credentials do +not reside in device configuration committed to the repository. ## Failure Boundaries | Failure | Effect | | --- | --- | | CCR2004 failure | The lab loses home-network and internet connectivity. Internal lab switching and routing remain available. | -| VP6630 or VyOS failure | Routed lab segments lose their gateways, inter-segment routing, policy enforcement, and internet egress. | -| MikroTik switch failure | Devices connected through the switch lose Layer 2 connectivity. | +| VP6630 or VyOS failure | Routed lab segments lose their gateways, inter-segment routing, policy enforcement, management/OOB gateway, and internet egress. | +| CRS309-1G-8S+IN failure | Devices connected through the core switch lose Layer 2 connectivity. | +| TEG-3102WS or its VP6630 uplink failure | Both non-SFP NICs on each MS-02 lose management/OOB connectivity. | | Routed transit failure | Home-to-lab and lab-to-internet traffic stop. Internal lab traffic remains available within its unaffected Layer 2 and Layer 3 paths. | -| VyOS-to-MikroTik trunk failure | VLANs carried by the trunk lose their VyOS gateways. | +| VyOS-to-CRS309 trunk failure | VLANs carried by the trunk lose their VyOS gateways. | | Invalid configuration | Deployment verification fails and the previous configuration is restored. | ## Verification @@ -146,5 +156,8 @@ A deployment is valid when the observed behavior matches these checks: - Each permitted firewall flow succeeds. - Each denied firewall flow fails. - Established reply traffic succeeds without enabling a new reverse flow. +- Both non-SFP NICs on each MS-02 connect through the TEG-3102WS. +- MS-02 management/OOB traffic reaches the VP6630 through the TEG-3102WS + uplink. - Management access succeeds only from approved source networks. - A failed deployment restores the previous configuration. diff --git a/docs/docs/index.md b/docs/docs/index.md index fc0befc..cb590f1 100644 --- a/docs/docs/index.md +++ b/docs/docs/index.md @@ -7,17 +7,18 @@ description: Architecture and operating documentation for the GilmanLab core net # GilmanLab Networking This repository defines the GilmanLab core network. VyOS handles routing, -firewall policy, and NAT. MikroTik handles switching and VLAN transport. +firewall policy, and NAT. A MikroTik CRS309-1G-8S+IN handles core switching and +VLAN transport. A TRENDnet TEG-3102WS handles MS-02 management/OOB switching. ## Documents - [Lab v2 core network design](designs/drafts/lab-v2-core-network.md) defines the core topology, device boundaries, configuration requirements, and verification criteria. -- [ADR-0001: Use VyOS for Layer 3 and MikroTik for Layer 2](decisions/0001-use-vyos-for-layer-3-and-mikrotik-for-layer-2.md) +- [ADR-0001: Use VyOS for Layer 3 and Dedicated Switches for Layer 2](decisions/0001-use-vyos-for-layer-3-and-switches-for-layer-2.md) assigns routing and switching responsibilities. -- [Hardware](reference/hardware.md) identifies the core network devices and - their roles. +- [Hardware](reference/hardware.md) identifies the core network devices, + connections, and roles. Compute platforms, workload networking, and application delivery are outside the core network documentation. diff --git a/docs/docs/reference/hardware.md b/docs/docs/reference/hardware.md index a7dd8d8..f2b6dab 100644 --- a/docs/docs/reference/hardware.md +++ b/docs/docs/reference/hardware.md @@ -1,6 +1,6 @@ --- title: Hardware -description: Core network devices and their assigned roles. +description: Core network devices, connections, and assigned roles. --- # Hardware @@ -10,7 +10,8 @@ description: Core network devices and their assigned roles. | Component | Platform | Role | | --- | --- | --- | | Lab gateway | Minisforum VP6630 running VyOS | Routes lab networks, enforces firewall policy, and performs source NAT | -| Lab switch | MikroTik switch | Carries VLANs and connects lab devices at Layer 2 | +| Core switch | MikroTik `CRS309-1G-8S+IN` | Carries lab VLANs and connects lab devices at Layer 2 | +| Management/OOB switch | TRENDnet `TEG-3102WS` | Connects both non-SFP NICs from each MS-02 for management/OOB traffic | ## External dependency @@ -18,5 +19,14 @@ description: Core network devices and their assigned roles. | --- | --- | --- | | Home router | MikroTik CCR2004 | Routes the home network, provides internet access, and terminates the upstream side of the routed lab transit | +## Connections + +| Endpoints | Connection | +| --- | --- | +| CCR2004 to VP6630 | Routed lab transit | +| VP6630 to CRS309-1G-8S+IN | 802.1Q trunk for lab VLANs | +| VP6630 to TEG-3102WS | Direct management/OOB uplink | +| TEG-3102WS to each MS-02 | Two non-SFP NICs carrying management/OOB traffic | + See the [Lab v2 core network design](../designs/drafts/lab-v2-core-network.md) for the topology and device boundaries. diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index a2047b4..2f1f869 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -32,7 +32,7 @@ theme: nav: - Home: index.md - Decisions: - - "ADR-0001: Use VyOS for Layer 3 and MikroTik for Layer 2": decisions/0001-use-vyos-for-layer-3-and-mikrotik-for-layer-2.md + - "ADR-0001: Use VyOS for Layer 3 and Dedicated Switches for Layer 2": decisions/0001-use-vyos-for-layer-3-and-switches-for-layer-2.md - Designs: - Drafts: - Lab v2 core network: designs/drafts/lab-v2-core-network.md