feat: persist PASS/FAIL/ERROR/NOT_APPLICABLE per rule per resource, fix compliance score

[parthrohit22]

What problem does this solve?

Two coupled problems, one fix:

1. The scanner only persists failing findings. scanner/engine.py::run_scan() collects a flat
   findings list — there is no record of which rules ran clean, which errored, or which
   resources were never evaluated.
2. As a direct consequence, api/models/finding.py::get_compliance_score() computes control
   status as status = "FAIL" if rule_id in failed_rule_ids else "PASS" — any rule that errored,
   whose resource was never observed, or that never ran is counted as PASS by default. This
   inflates every reported compliance score.

Both block every downstream reasoning-layer feature (drift history, campaign detection,
attack-path confidence) and the eight already-open "enterprise rules" issues (#253, #255, #257,
#258, #259, #261, #262, plus closed #256/#260) that already assume a PASS/FAIL/UNKNOWN/
NOT_APPLICABLE contract exists, with no shared engine support for it.

PR #250 ("Azure Resource Graph inventory foundation") already lands Phase 1 of the underlying
evidence model — normalised, tenant-scoped resource snapshots with complete/partial/failed
collection state — but explicitly does not persist rule-outcome data. This issue is Phase 2: the
rule-outcome side, built on top of #250's snapshots.

Describe the solution

• New table (e.g. rule_evaluations) keyed on (scan_id, rule_id, resource_id) with a status
  enum: PASS / FAIL / ERROR / NOT_APPLICABLE, plus a free-text reason for ERROR/NOT_APPLICABLE.
• ScanEngine.run_scan() writes an evaluation row for every resource a rule was applicable to,
  not just the ones that failed.
• Update get_compliance_score() to compute PASS only from persisted successful evaluations in
  this table — remove the absence-implies-PASS default entirely.
• Document the contract in docs/adding-a-rule.md so new rules (including the 8 open enterprise
  issues) target it directly instead of inventing ad hoc indeterminate handling (see
  scanner/rules/az_cmp_002.py's own LOW-severity indeterminate workaround for what happens
  without a shared model).

Alternatives considered

Ship the schema and the compliance-score fix as two separate PRs — rejected; the fix has no
correct implementation without the schema existing first, and reviewing them apart just means
merging a known-broken intermediate state.

Additional context

Depends on: PR #250 (merge first). Blocks: issue 2 below, and all eight open enterprise-rules
issues. Exit test: a forced API error or unseen resource never appears as PASS.

Coordination

This issue is the shared evaluation-contract foundation for the open enterprise-rule packs. It
does not duplicate them — it provides the engine support (per-resource PASS / FAIL / ERROR /
NOT_APPLICABLE) they assume but do not themselves build. It explicitly blocks and should be
linked from: #253, #254, #255, #257, #258, #259, #261, #262. Enterprise-rule PRs should consume
this persisted evaluation contract instead of re-implementing ad hoc indeterminate handling.

[parthrohit22]

Coordination note — foundation for the enterprise-rule packs (not a duplicate).

This issue is the shared rule-evaluation contract the open enterprise-rule packs depend on. It provides the per-resource PASS / FAIL / ERROR / NOT_APPLICABLE persistence that those packs assume but do not themselves build. It does not duplicate any enterprise-rule issue; rather, it explicitly blocks and should be linked from:

#253, #254, #255, #257, #258, #259, #261, #262

Enterprise-rule PRs should consume this persisted evaluation contract instead of re-implementing ad hoc indeterminate handling. Added a "Coordination" section to the issue body for reference.

[dipeshrayg]

I'd like to take this on. Before writing code, here's the design I'm planning, since the resource-level part isn't fully mechanical from the current rule contract.

The core problem: today, every rule's scan() only returns a findings list (violations). The engine has no visibility into which resources a rule actually examined, so there's no denominator for a true per-resource PASS, and no way to distinguish "ran clean" from "never ran" or "not applicable."

The plan: a new Alembic migration adds a rule_evaluations table:

rule_evaluations
  id, scan_id (FK scans), rule_id, resource_id, resource_type,
  status (PASS | FAIL | ERROR | NOT_APPLICABLE),
  reason (nullable, populated for ERROR/NOT_APPLICABLE),
  evaluated_at

Each rule gets one new module constant, RESOURCE_TYPE, declared statically (most rules already embed this string in their finding dicts on FAIL, this just makes it available up front instead of only on failure). ScanEngine.run_scan() cross-references each rule's RESOURCE_TYPE against the subscription's resource inventory from PR #250's InventorySnapshot once it's merged: findings become FAIL, applicable resources with no finding become PASS, an exception during scan() becomes ERROR for the whole applicable set, and a resource type absent from the inventory becomes NOT_APPLICABLE. Rules that haven't declared RESOURCE_TYPE yet keep working exactly as they do today, rule-level, absence-implies-PASS, with a logged warning, so nothing breaks and no 71-rule rewrite is required up front. Rules migrate incrementally. get_compliance_score() reads from rule_evaluations for migrated rules and falls back to today's logic for the rest until migration completes.

First PR scope: the migration, the engine wiring, the get_compliance_score() rewrite, and RESOURCE_TYPE added to a small reference set of rules (the two I've already authored), not a full 71-rule migration in one PR.

Question for maintainers: should NOT_APPLICABLE resolve against PR #250's ARG inventory specifically, or against the existing per-service AzureClient collectors directly?

[m-khan-97]

Thanks for volunteering to take this on. The overall direction makes sense, but I want us to adjust a few parts of the design before implementation so we do not carry the current scoring bug into the new model.

• We should not keep the old “absence means PASS” fallback for rules that have not migrated yet. That is the main defect this issue is meant to remove. If a rule has not produced enough evidence for a real PASS, it should remain UNKNOWN or ERROR, or be explicitly marked NOT_APPLICABLE.
• RESOURCE_TYPE plus the ARG inventory is not enough by itself to decide whether a rule applies. Applicability may depend on tags, policy, child resources, service tier, or evidence that ARG does not contain. I would prefer the rule contract to return typed evaluation results for the resources it actually assessed, including the reason and evidence behind PASS, FAIL, ERROR, or NOT_APPLICABLE.
• PR feat(scanner): add Azure Resource Graph inventory foundation #250 can provide the discovery baseline when its inventory is complete. Please preserve its complete, partial, and failed states: partial or failed discovery must not be used to infer NOT_APPLICABLE. Targeted collectors can add evidence, but collector failures should remain UNKNOWN or ERROR.
• The schema migration and compliance-score correction should land together. Please include tests proving that rule exceptions, missing evidence, partial inventory, and unmigrated rules never improve the reported score.

Could you post a revised schema and rule-result contract here before opening the implementation PR? That will let the leads agree on the denominator and migration behavior early.

I also tried to assign the issue to you, but GitHub currently reports your account as ineligible for assignment. A repository owner may need to restore or grant triage access first.