Skip to content

fix(test): make getRecordCount within-budget cases immune to CI timing - #2159

Open
kriszyp wants to merge 2 commits into
mainfrom
fix/getRecordCount-flaky-timeLimit-default
Open

fix(test): make getRecordCount within-budget cases immune to CI timing#2159
kriszyp wants to merge 2 commits into
mainfrom
fix/getRecordCount-flaky-timeLimit-default

Conversation

@kriszyp

@kriszyp kriszyp commented Aug 13, 2026

Copy link
Copy Markdown
Member

Summary

Two cases in unitTests/resources/getRecordCount.test.js asserted the within-budget
branch of Table.getRecordCount (exact count, no estimatedRange, zero calls to the
engine's entry-count source) while relying on the ambient 500ms default budget
(resources/Table.ts:4772) to guarantee that branch is taken. On a contended CI runner a
30-row scan — which yields once per row via await rest() — can exceed 500ms, flipping
into the estimator branch and failing assertions that were never about timing. This is a
pure test flake, confirmed by the same assertion failing on two unrelated branches within
hours of each other (main and kris/2153-audit-only-has-record, neither of which touches
getRecordCount or the storage layer).

Both cases now pass an explicit timeLimit that never exhausts (Infinity, hoisted to a
WITHIN_BUDGET_TIME_LIMIT constant), mirroring the existing timeLimit: -1 idiom the
sibling estimator cases already use to force the opposite branch deterministically. No
production code changed.

For the human reviewer

  • Deferred, not fixed here: the edited cases no longer exercise the 500ms default
    itself — that coverage was incidental and is exactly what made them flaky. Nothing else
    in the suite asserts the default's value is workable for a small table; adding that
    back deliberately (e.g. a soft/warn check) is a reasonable follow-up but would
    reintroduce the same flakiness if written as a hard timing assertion.
  • Chose timeLimit: Infinity over a large finite value (e.g. 60_000) so "budget never
    exhausts" is expressed structurally rather than by asserting a number is big enough on
    any runner.
  • Gemini's delta-round pass re-raised "this bypasses coverage of the ambient 500ms
    default" as major. Same finding as round 1, already adjudicated there and dropped:
    the pre-edit tests never caught a broken default either (options?.timeLimit ?? 500
    with a corrupted default still yields exact-count assertions passing), so this is a
    pre-existing, stated gap, not a regression from this diff. Domain adjudication was
    auto-pruned this round (narrow low-risk delta) so the tool's receipt shows it
    unresolved; the round-1 reasoning still applies unchanged.

Verification

  • unitTests/resources/getRecordCount.test.js — all 6 cases pass under both storage
    engines: npx mocha unitTests/resources/getRecordCount.test.js and
    HARPER_STORAGE_ENGINE=lmdb npx mocha unitTests/resources/getRecordCount.test.js.
  • Full npm run test:unit:resources (RocksDB): 1509 passing, 3 pre-existing failures
    unrelated to this change (randomAccessFieldsDirective.test.js,
    replayStructures.test.js) — not touched by this diff, not present in the affected
    file.

Review coverage

Pre-push review via prepush-review.mjs, two rounds:

  • Round 1 (full): codex (graded) + gemini + cursor-composer + Harper domain adjudication.
    Adjudicated finding (nit: duplicated timeLimit/comment across the two cases) fixed in
    a follow-up commit.
  • Round 2 (delta, narrow fix inside already-reviewed code): codex (graded, verdict LGTM,
    no findings) + gemini. Gemini's major finding is a repeat of a round-1 finding already
    adjudicated and dropped there (see "For the human reviewer" above); domain adjudication
    was auto-pruned this round as narrow/low-risk, so it wasn't re-adjudicated by the tool.

Refs https://github.com/HarperFast/harper/actions/runs/31648388950

🤖 Generated with Claude Code

Human-Review-Need: 4 @ 40e961b

@kriszyp
kriszyp requested review from dawsontoth and heskew August 13, 2026 11:57

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a WITHIN_BUDGET_TIME_LIMIT constant set to Infinity in the getRecordCount unit tests. This constant is passed as the timeLimit option to getRecordCount calls in tests that expect exact counts within the budget. This change prevents slow or contended CI runners from exceeding the default 500ms budget and inadvertently triggering the estimator path. There are no review comments, and I have no additional feedback to provide.

@claude

claude Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Reviewed; no blockers found.

@kriszyp
kriszyp marked this pull request as ready for review August 15, 2026 00:25
kriszyp and others added 2 commits August 15, 2026 05:54
Two cases in unitTests/resources/getRecordCount.test.js relied on the
ambient 500ms default time budget to exercise the within-budget path
(exact count, no estimator, zero entry-count-source calls). On a
contended CI runner the 30-row scan can miss that window, flipping the
early-exit/sampling branch on and failing assertions that expect it
off.

Pass an explicit, generous timeLimit (60s) in both cases instead,
mirroring the existing `timeLimit: -1` idiom already used by the
sibling estimator cases to force the opposite branch deterministically.

Refs https://github.com/HarperFast/harper/actions/runs/31648388950

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Address pre-push review nit: the explicit timeLimit and its rationale
comment were duplicated across both within-budget cases. Hoist to a
single WITHIN_BUDGET_TIME_LIMIT constant (Infinity, which structurally
expresses "budget never exhausts" rather than asserting a magic number
is big enough) with one comment at the declaration.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@kriszyp
kriszyp force-pushed the fix/getRecordCount-flaky-timeLimit-default branch from 35048dd to 40e961b Compare August 15, 2026 11:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants