Three of the 16 detection rules have no scenario exercising them:
TIMEOUT_NO_HEARTBEAT, FIRMWARE_UPDATE_FAILURE and
REPEATED_BOOT_NOTIFICATION. A rule with no scenario has no regression guard,
so nothing in the corpus would catch it breaking.
REPEATED_BOOT_NOTIFICATION shows how the gap opens. It shipped in #114 and no
scenario followed, because nothing in the contributor guide asks for one. The
"How to Add a Detection Rule" checklist ends by auditing existing scenarios for
false positives, which is the negative half of the coverage question. The
positive half, a scenario whose expectedFailures contains the new code, is
never requested.
Fix
State the invariant in the "Contributing Detection Rules" section of
CONTRIBUTING.md: every detection rule ships with a scenario that exercises it,
added in the same pull request as the rule.
Add the corresponding step to the "How to Add a Detection Rule" list, next to the
existing false-positive audit, so both directions of coverage are asked for in
the same place.
Why write it down
- It keeps the corpus honest as rules are added. The scenario count is a v1.0
exit criterion (ROADMAP.md line 157), and rule coverage is what makes that
count mean something.
- It turns the gap into a well-specified newcomer task automatically. Whenever a
rule lands without a scenario, the follow-up is a good-first-issue that writes
itself.
- It is the rule that would have prevented the three issues above from being
needed at all.
Documentation only. No changeset: CONTRIBUTING.md is not in the package files
list, so it does not ship to npm.
Related
The three uncovered rules are tracked in #137, #138 and #139. This issue is the
process change that stops a fourth appearing.
Three of the 16 detection rules have no scenario exercising them:
TIMEOUT_NO_HEARTBEAT,FIRMWARE_UPDATE_FAILUREandREPEATED_BOOT_NOTIFICATION. A rule with no scenario has no regression guard,so nothing in the corpus would catch it breaking.
REPEATED_BOOT_NOTIFICATIONshows how the gap opens. It shipped in #114 and noscenario followed, because nothing in the contributor guide asks for one. The
"How to Add a Detection Rule" checklist ends by auditing existing scenarios for
false positives, which is the negative half of the coverage question. The
positive half, a scenario whose
expectedFailurescontains the new code, isnever requested.
Fix
State the invariant in the "Contributing Detection Rules" section of
CONTRIBUTING.md: every detection rule ships with a scenario that exercises it,added in the same pull request as the rule.
Add the corresponding step to the "How to Add a Detection Rule" list, next to the
existing false-positive audit, so both directions of coverage are asked for in
the same place.
Why write it down
exit criterion (
ROADMAP.mdline 157), and rule coverage is what makes thatcount mean something.
rule lands without a scenario, the follow-up is a good-first-issue that writes
itself.
needed at all.
Documentation only. No changeset:
CONTRIBUTING.mdis not in the packagefileslist, so it does not ship to npm.
Related
The three uncovered rules are tracked in #137, #138 and #139. This issue is the
process change that stops a fourth appearing.