Skip to content

docs: require a scenario for every detection rule #140

Description

@sepehr-safari

Three of the 16 detection rules have no scenario exercising them:
TIMEOUT_NO_HEARTBEAT, FIRMWARE_UPDATE_FAILURE and
REPEATED_BOOT_NOTIFICATION. A rule with no scenario has no regression guard,
so nothing in the corpus would catch it breaking.

REPEATED_BOOT_NOTIFICATION shows how the gap opens. It shipped in #114 and no
scenario followed, because nothing in the contributor guide asks for one. The
"How to Add a Detection Rule" checklist ends by auditing existing scenarios for
false positives, which is the negative half of the coverage question. The
positive half, a scenario whose expectedFailures contains the new code, is
never requested.

Fix

State the invariant in the "Contributing Detection Rules" section of
CONTRIBUTING.md: every detection rule ships with a scenario that exercises it,
added in the same pull request as the rule.

Add the corresponding step to the "How to Add a Detection Rule" list, next to the
existing false-positive audit, so both directions of coverage are asked for in
the same place.

Why write it down

  • It keeps the corpus honest as rules are added. The scenario count is a v1.0
    exit criterion (ROADMAP.md line 157), and rule coverage is what makes that
    count mean something.
  • It turns the gap into a well-specified newcomer task automatically. Whenever a
    rule lands without a scenario, the follow-up is a good-first-issue that writes
    itself.
  • It is the rule that would have prevented the three issues above from being
    needed at all.

Documentation only. No changeset: CONTRIBUTING.md is not in the package files
list, so it does not ship to npm.

Related

The three uncovered rules are tracked in #137, #138 and #139. This issue is the
process change that stops a fourth appearing.

Metadata

Metadata

Assignees

No one assigned

    Labels

    package:repoRepository-level changespriority:mediumShould be done in current milestonetype:docsDocumentation improvements

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions