What
docs/dev/testing-strategy.md is the repo's map of what is tested where, and CLAUDE.md calls it "the fixed four-tier testing model — do not invent a new one". The appliance work has landed a substantial test surface that appears nowhere in it:
tests/os/run.sh (~1600 lines) — a KVM battery with five phases (boot, update, install, provision, fault), totalling 81 assertions per docs/dev/appliance-release.md's table.
tests/os/verify-image.sh — static image verification, no KVM needed.
build/dashboard/tests/web/test_wizard.py (68 tests) and tests/frontend/wizard.test.mjs, which are tier 1 but have no row in the scenario catalog.
The strategy doc's four tiers are Unit / Contract / Mini-stack / Live matrix, and its tier-4 row names only tests/integration/run.sh. tests/os/README.md self-describes as "Tier-4", so today there are two different things called tier 4 with different entry points, different prerequisites (KVM + libvirt + root vs. a synced box), and different release gates.
Why it matters
The doc's stated contract is "each situation is tested once, at the lowest tier that is honest", and its "Adding a scenario" section tells a contributor where new coverage goes. Neither works for appliance behavior right now — there is no rule saying whether a new appliance behavior belongs in the KVM battery, the stack suite, or a unit test.
That ambiguity has a cost the audit could see: the KVM battery is the most expensive harness in the repo and the easiest place to accidentally park logic that a unit test could prove. It is currently well-disciplined (the egress leg at tests/os/run.sh:1202-1230 proves real kernel drops, which genuinely needs a real kernel; the rule rendering is unit-tested separately in tests/stack/run.sh), but nothing written down keeps it that way.
Suggested shape
Decide and document one of:
- The appliance battery is tier 4 alongside the live matrix — then say so explicitly, with its own prerequisites and gate, and split the tier-4 row in the "What runs where" table.
- It is a fifth tier — then the "fixed four-tier" language in
CLAUDE.md and this doc needs updating together.
Either way, add a scenario-catalog section for the appliance covering at least: EFI boot and the first-boot wizard window; install-to-disk and the reinstall/wipe legs; A/B update, commit gate and rollback; Tor-only egress enforcement under netavark; the wizard's server-owned stage machine. And extend "Adding a scenario" with the appliance rule, so the next contributor knows what belongs in KVM and what does not.
Found during the develop-v2 quality pass (PR #867).
What
docs/dev/testing-strategy.mdis the repo's map of what is tested where, andCLAUDE.mdcalls it "the fixed four-tier testing model — do not invent a new one". The appliance work has landed a substantial test surface that appears nowhere in it:tests/os/run.sh(~1600 lines) — a KVM battery with five phases (boot,update,install,provision,fault), totalling 81 assertions perdocs/dev/appliance-release.md's table.tests/os/verify-image.sh— static image verification, no KVM needed.build/dashboard/tests/web/test_wizard.py(68 tests) andtests/frontend/wizard.test.mjs, which are tier 1 but have no row in the scenario catalog.The strategy doc's four tiers are Unit / Contract / Mini-stack / Live matrix, and its tier-4 row names only
tests/integration/run.sh.tests/os/README.mdself-describes as "Tier-4", so today there are two different things called tier 4 with different entry points, different prerequisites (KVM + libvirt + root vs. a synced box), and different release gates.Why it matters
The doc's stated contract is "each situation is tested once, at the lowest tier that is honest", and its "Adding a scenario" section tells a contributor where new coverage goes. Neither works for appliance behavior right now — there is no rule saying whether a new appliance behavior belongs in the KVM battery, the stack suite, or a unit test.
That ambiguity has a cost the audit could see: the KVM battery is the most expensive harness in the repo and the easiest place to accidentally park logic that a unit test could prove. It is currently well-disciplined (the egress leg at
tests/os/run.sh:1202-1230proves real kernel drops, which genuinely needs a real kernel; the rule rendering is unit-tested separately intests/stack/run.sh), but nothing written down keeps it that way.Suggested shape
Decide and document one of:
CLAUDE.mdand this doc needs updating together.Either way, add a scenario-catalog section for the appliance covering at least: EFI boot and the first-boot wizard window; install-to-disk and the reinstall/wipe legs; A/B update, commit gate and rollback; Tor-only egress enforcement under netavark; the wizard's server-owned stage machine. And extend "Adding a scenario" with the appliance rule, so the next contributor knows what belongs in KVM and what does not.
Found during the
develop-v2quality pass (PR #867).