Skip to content

Pin the load-bearing native and encoder dependencies - #2179

Merged
kriszyp merged 9 commits into
mainfrom
kris/pin-load-bearing-deps
Aug 18, 2026
Merged

Pin the load-bearing native and encoder dependencies#2179
kriszyp merged 9 commits into
mainfrom
kris/pin-load-bearing-deps

Conversation

@kriszyp

@kriszyp kriszyp commented Aug 15, 2026

Copy link
Copy Markdown
Member

A caret range does not bind the version that reaches a running Harper node. harper-pro, a rebuilt container, and anyone installing published harper without this lockfile can resolve a newer native or encoder dependency without a Harper PR or human merge.

This PR converts the load-bearing ranges to the versions the lockfile already resolved. No installed dependency moves; only the allowed resolution narrows. The pins cover rocksdb-js, the encoder/iterator modules whose identity crosses its boundary, structon, and the optional native addons.

The module identity is now enforced rather than documented as an assumption. Docker smoke requires the root msgpackr and @harperfast/extended-iterable specs to be exact and byte-equal to rocksdb-js's requirements, verifies their installed root versions, and rejects a nested copy under rocksdb-js. A rocksdb-js automation or Renovate PR that moves only one side will therefore fail before it can merge.

The shrinkwrap canary remains meaningful after rocksdb-js becomes exact: exact specs still participate in installed-versus-packed comparison but not range discrimination, while ranged fastify and @aws-sdk/client-s3 provide the discrimination proof. Registry queries retry three times with 1s/2s backoff. If the surviving canaries are current and a query remains unresolved, the check fails closed with a retry-specific error; Kris selected that behavior rather than treating “unknown” as “current.”

For the human reviewer

  1. Exact pins in a published library. These constraints propagate to consumers, which is the intended guarantee. The cost is that a consumer needs a Harper release to take a patch of these dependencies.
  2. Optional native scope. bufferutil, segfault-handler, and utf-8-validate are exact because their prebuild/platform behavior is load-bearing, but exact optional pins also remove the in-range escape when a new Node ABI needs a newer prebuild. Their absence is largely silent today.
  3. Consumer layout. Docker smoke proves the Harper image has one rocksdb-js-facing encoder/iterator instance. It cannot prove every downstream harper-pro or npm-consumer tree has the same layout; that remains the main boundary a human should consider.

The pre-existing lock entries for @harperfast/extended-iterable and segfault-handler still lack resolved/integrity. That pattern is lockfile-wide rather than introduced here, but an exact version without a tarball hash is a weaker supply-chain guarantee.

Verification

  • npm run build
  • npm run lint:required
  • npx mocha unitTests/build-tools/checkShrinkwrapPins.test.mjs — 13 passing
  • npm run test:unit:resources — 1,556 passing, 15 pending
  • Regression proof: the three retry/fail-closed cases fail against the pre-fix script and pass at this head.
  • The focused suite spawns the real checker process and covers exact/ranged canaries, transient and exhausted queries, empty registry results, real-manifest viability, lockstep divergence, ranged encoder regressions, and nested module copies.
  • GitHub: 42 checks passed and two intentionally skipped. This includes Docker smoke, Node 22/24/26 unit tests, adapters, and the complete Linux/Windows/Bun/uWS integration matrix. The first Node 22 attempt hit the unrelated immediate-expiration cache flake in unitTests/apiTests/cache-test.mjs:69; its isolated rerun passed.

The local umbrella unit/integration reruns were not clean under this machine's forced isolated Harper root: unrelated config/logger tests expected the user boot configuration, and one high-volume integration run returned nonzero without preserving the failing case in its truncated output. The prior PR head passed both umbrella suites, and the authoritative current-head GitHub matrices are green.

Review coverage

Authored by GPT-5.6 Codex. Independent pre-push review covered the full PR with Claude and the Harper-domain adjudicator, plus delta rounds with Claude and Gemini. Earlier rounds also included Cursor Grok; Cursor Composer was pruned. The final delta review found the build tooling complete and carried only the dependency-policy decisions above.

Human-Review-Need: 4 @ 3e2adc8

A caret range means the manifest gate does not bind. harper-pro, a container
rebuild, and anyone installing published harper without this lockfile all
resolve to whatever is newest at install time, so a rocksdb-js or msgpackr
minor reaches a running node with no Harper PR and no human merge.

That is the mechanism behind 5.1.22 shipping rocksdb-js 2.4.0 while the
cross-column-family read fix was in 2.5.0: the pin permitted the fix and the
image predated it. The same latitude equally admits a regression.

extended-iterable is pinned for a sharper reason than the rest. rocksdb-js
requires exactly 1.0.3 while the root asked for ^1.0.1, so the day 1.0.4
publishes a fresh resolution hoists 1.0.4 for the root and nests 1.0.3 under
rocksdb-js — two modules, two SKIP sentinels. A vector query whose candidate
record was deleted then returns harper's SKIP into a map() belonging to
rocksdb-js's ExtendedIterable, which does not recognise it and emits the
sentinel as a result row: a phantom record on the read path, no exception and
no log line. msgpackr has the same shape via its extension registry, where the
nested copy never saw addExtension for Blob.

update-rocksdb-js.yml gains --save-exact. It ran `npm install --save`, and with
no .npmrc the default save-prefix of ^ applies, so the next rocksdb-js release
would have rewritten 2.7.1 back to ^2.8.0 and reverted this commit unattended.

Every pin is the version the lockfile already resolved, so no dependency moves
here; only the range narrows. lmdb, cbor-x, ordered-binary, alasql and argon2
were already exact.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@kriszyp
kriszyp requested a review from dawsontoth August 15, 2026 00:01

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates and pins several dependencies in package.json, including @harperfast/extended-iterable, @harperfast/rocksdb-js, msgpackr, structon, bufferutil, segfault-handler, and utf-8-validate. There are no review comments, and I have no feedback to provide.

@claude

claude Bot commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Reviewed; no blockers found.

@kriszyp
kriszyp marked this pull request as ready for review August 15, 2026 00:21
@kriszyp
kriszyp requested a review from cb1kenobi August 15, 2026 00:21
Comment thread package.json
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
@cb1kenobi

Copy link
Copy Markdown
Member

Re-reviewed da2acbc2 (incremental since 35d07599) — no new issues found.

The prior Medium finding is resolved: @aws-sdk/client-s3 (^3.1012.0) joins the canary set as a genuinely discriminating one, the discrimination check now skips exact manifest specs instead of counting them, and an all-exact canary set fails loudly with a clear "add a ranged canary" error. The new unitTests/build-tools/checkShrinkwrapPins.test.mjs covers all five paths and is picked up by test:unit:main; all 5 pass locally.

Worth noting why the new canary holds up: Renovate's minimumReleaseAge: "7 days" combined with the AWS SDK's near-daily publish cadence means the lock pin can't realistically catch up to max-in-range, so this canary is structurally more durable than fastify.

This PR looks good, nice job!


Generated by Barber AI

kriszyp and others added 5 commits August 16, 2026 18:35
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Comment thread build-tools/check-shrinkwrap-pins.mjs
kriszyp and others added 2 commits August 16, 2026 22:07
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
Co-Authored-By: GPT-5 Codex <noreply@openai.com>
@cb1kenobi

Copy link
Copy Markdown
Member

Reviewed 3e2adc8 — no issues found. This PR looks good, nice job!


Generated by Barber AI

@kriszyp
kriszyp merged commit 058ba37 into main Aug 18, 2026
72 of 73 checks passed
@kriszyp
kriszyp deleted the kris/pin-load-bearing-deps branch August 18, 2026 02:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants