Roll standalone quality checks into CI with baseline-aware ratchets

[rochecompaan]

Context

PR #192 adds standalone local quality checks and records their initial results. It intentionally leaves CI unchanged so the checks can be studied before they become required.

This issue tracks the follow-up work needed to adopt those checks without making existing repository debt block unrelated development.

Decisions

• Keep PR feat(quality): add standalone code quality checks #192 local-only.
• Add checks to CI one at a time after their output is stable and useful.
• Use baseline-aware ratchets instead of enforcing repository-wide strict thresholds immediately.
• Keep the current coverage and CRAP results identified as the core bin/src baseline. Shipped extensions code needs a separate owned baseline rather than being blended into the core result without notice.
• Do not run repository-wide mutation testing on every PR.

Initial evidence

Core baseline from PR #192:

• Lines/statements: 91.76%
• Branches: 85.95–85.96%
• Functions: 94.80%
• CRAP: 1,615 functions checked; 12 above 30
• Strict typing: 214 diagnostics across 62 files
• Property tests: 3 passing
• Mutation pilot: 38 killed, 19 survived; 66.67%
• Architecture: one unresolved extensions/todos.ts -> typebox dependency

The current coverage, CRAP, and strict-type configurations do not include extensions. Coverage also needs review for type-only files that appear in the executable-line denominator.

CI rollout

1. Blocking immediately

   • Extend normal TypeScript linting to quality/**/*.ts without adding a duplicate ESLint pass.
   • Run the property tests as a required PR check.

2. Blocking after the current defect is fixed

   • Declare typebox as a direct runtime dependency.
   • Configure Dependency Cruiser to resolve package exports using suitable import/node/default/types conditions.
   • Require the architecture check with a zero-violation baseline.

3. Report first, then ratchet

   • Publish machine-readable core and extension coverage summaries.
   • Generate CRAP from the same coverage run rather than rerunning the full test suite.
   • Publish normalized strict-TypeScript diagnostics.
   • Observe about ten default-branch runs, including a cold runner, before making metric gates required.

4. Mutation

   • Run the focused pilot on PRs only when its policy module, focused tests, important callers, or Stryker configuration changes.
   • Run a fresh, non-incremental pilot weekly and upload JSON/HTML reports.
   • Expand only to curated modules with focused test suites.

Ratchet policy

Coverage

• Define core and extension scopes explicitly.
• Exclude only genuinely non-runtime/type-only modules before freezing the denominator.
• Use a global safety floor plus changed-executable-line coverage.
• Allow baseline percentages to rise and uncovered caps to fall; require a dedicated reviewed reset for movement in the other direction.
• Account for the observed 0.01 percentage-point branch variation.

CRAP

• Commit an exact allowlist keyed by file and function with the current score as a cap.
• Fail on a new function above 30 or an existing hotspot exceeding its cap.
• Permanently remove entries when they reach 30 or lower.
• Refactor high-complexity, fully covered functions instead of adding tests that cannot reduce their score.

Strict typing

• Store diagnostics as a multiset of file, TS code, and flattened message, excluding line and column positions.
• Fail on new diagnostics or increased occurrence counts.
• Require the baseline to shrink when errors are removed.
• Gate directories at zero as they become clean; replace the comparator with raw strict tsc when the total reaches zero.

Architecture

• Resolve the current typebox edge rather than suppressing it.
• Keep a zero-violation baseline.
• Introduce future layer rules one at a time, initially report-only when necessary.

Mutation survivor work

Classify survivors by source location, mutator, and replacement because numeric IDs are unstable.

Current findings:

• Meaningful focused gaps include completion-status preference and valid JSON non-array fallback behavior.
• Several behaviors are already covered by broader tests but are absent from test:mutation, including the public environment handoff, completion selection, and undefined parser input.
• Several survivors are equivalent because weakened validation reaches the same broad catch/fallback path. Do not add tests that distinguish only internal implementation paths.

For each survivor: run it alone without the cache, construct the smallest distinguishing input, reuse an existing behavior test when possible, and document equivalent mutants with a rationale. Do not use the raw mutation percentage as the sole gate.

Next property-test targets

In priority order:

1. Task-contract title/template/regex/glob laws.
2. Run-cost aggregation permutation, deduplication, total-conservation, and parse round-trip laws.
3. Published-artifact format/extract, checksum-tamper, and trust-ordering laws.
4. Triage-state precedence and clone-independence laws.

Each property must prove a domain behavior that can fail under a meaningful regression; do not add properties merely to restate implementation.

Follow-up checklist

• Add fast blocking lint/property checks.
• Declare and verify the direct typebox dependency; enable package-exports resolution; make architecture required.
• Define core and extension metric scopes and produce stable CI artifacts.
• Add baseline-aware coverage, CRAP, and strict-type comparators.
• Triage the focused mutation survivors and add path-filtered PR plus weekly scheduled mutation runs.
• Add the next property-test modules.
• Reduce strict-type debt by diagnostic class and directory.
• Refactor CRAP hotspots one at a time.

Acceptance criteria

• Required PR checks block only new regressions or failures in checks that already have a zero baseline.
• Existing typing and CRAP debt does not freeze unrelated development.
• Core and extension metrics are reported separately and accurately named.
• Architecture has zero unresolved, circular, or active layer violations.
• Mutation survivors are killed or justified, with fresh scheduled reports retained as artifacts.
• Baseline updates are monotonic unless a dedicated reviewed reset explains a legitimate denominator or policy change.

Non-goals

• Adding thresholds to PR feat(quality): add standalone code quality checks #192.
• Making all current strict typing or strict CRAP failures block immediately.
• Running repository-wide mutation testing on every PR.
• Adding tests only to increase a score or assert static configuration values.