Skip to content

testing-strategy.md does not place the appliance in the four-tier model #869

Description

@VijitSingh97

What

docs/dev/testing-strategy.md is the repo's map of what is tested where, and CLAUDE.md calls it "the fixed four-tier testing model — do not invent a new one". The appliance work has landed a substantial test surface that appears nowhere in it:

  • tests/os/run.sh (~1600 lines) — a KVM battery with five phases (boot, update, install, provision, fault), totalling 81 assertions per docs/dev/appliance-release.md's table.
  • tests/os/verify-image.sh — static image verification, no KVM needed.
  • build/dashboard/tests/web/test_wizard.py (68 tests) and tests/frontend/wizard.test.mjs, which are tier 1 but have no row in the scenario catalog.

The strategy doc's four tiers are Unit / Contract / Mini-stack / Live matrix, and its tier-4 row names only tests/integration/run.sh. tests/os/README.md self-describes as "Tier-4", so today there are two different things called tier 4 with different entry points, different prerequisites (KVM + libvirt + root vs. a synced box), and different release gates.

Why it matters

The doc's stated contract is "each situation is tested once, at the lowest tier that is honest", and its "Adding a scenario" section tells a contributor where new coverage goes. Neither works for appliance behavior right now — there is no rule saying whether a new appliance behavior belongs in the KVM battery, the stack suite, or a unit test.

That ambiguity has a cost the audit could see: the KVM battery is the most expensive harness in the repo and the easiest place to accidentally park logic that a unit test could prove. It is currently well-disciplined (the egress leg at tests/os/run.sh:1202-1230 proves real kernel drops, which genuinely needs a real kernel; the rule rendering is unit-tested separately in tests/stack/run.sh), but nothing written down keeps it that way.

Suggested shape

Decide and document one of:

  1. The appliance battery is tier 4 alongside the live matrix — then say so explicitly, with its own prerequisites and gate, and split the tier-4 row in the "What runs where" table.
  2. It is a fifth tier — then the "fixed four-tier" language in CLAUDE.md and this doc needs updating together.

Either way, add a scenario-catalog section for the appliance covering at least: EFI boot and the first-boot wizard window; install-to-disk and the reinstall/wipe legs; A/B update, commit gate and rollback; Tor-only egress enforcement under netavark; the wizard's server-owned stage machine. And extend "Adding a scenario" with the appliance rule, so the next contributor knows what belongs in KVM and what does not.

Found during the develop-v2 quality pass (PR #867).

Metadata

Metadata

Assignees

No one assigned

    Labels

    appliancePithead OS appliance work — lands on feat/phase2-bakery-imagedocumentationImprovements or additions to documentationtestingTests, CI, and test infrastructure

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions