Make per-table record-structure dictionary size observable

[kriszyp]

Answers the observability half of #2220 — Per-table msgpackr structure dictionaries grow unbounded and unobservably, with a wire-format cliff at id 63.

The typed (random-access) structure dictionary has been bounded at 256 since v5.1.0 (c93a72611), and storage.randomAccessFields has defaulted off since the same release (6ebe45474). What was still missing is that nothing reported the dictionary's size, and nothing signalled when it saturates — past the bound, novel record shapes are permanently stored without random-access field encoding, with no log line, metric, or error. The reporter found a 1,297-structure table by counting entries in a decoder dump inside an unrelated error message.

surface	what
describe_table	typed_structures_enabled, typed_structure_count, typed_structure_limit, classic_structure_count
log	one warning per encoder, on the transition to saturation
API	Table.getStructureCounts() / RecordEncoder.getStructureCounts()

No wire-format change, no bound change, no behavior change for a table below the bound.

For the human reviewer

Read resources/RecordEncoder.ts first — everything else adapts what it returns.

• The counts come from the durable shared-structures payload, not from this thread's encoder arrays. That is the one non-obvious decision here, and the reason getStructureCounts() is not a field read. Each worker owns its own encoder and loads/mints lazily, so a worker that never encoded for a store holds empty arrays for a store whose dictionary is full — reporting those would tell an operator there is full headroom when there is none. countDurableStructures() mirrors structon's own onLoadedStructures payload handling; look hardest here. The bare-array and cbor-x {structures} forms carry named structures only and structon keeps the local typed dictionary for them, so those two branches deliberately return the encoder's count rather than zero. Getting that backwards inverts the whole point of the change, and an earlier revision did.

• The warning deliberately reads the local array instead. It fires only from the post-save path, where this encoder's array is exactly what was just persisted — canonical at that moment, and free. The consequence is that its scope is per encoder, not per durable store: several workers can each warn, and a node restarted against an already-saturated dictionary warns not at all (saturation stops minting, so it stops saving). That asymmetry is intentional — the counts are the reliable signal and the warning is the heads-up — and it is stated in the docs companion rather than left to be discovered.

• typed_structures_enabled exists because typed 0 / limit 256 is ambiguous. Random-access fields default off, so most tables report zero typed structures — that is the feature being disabled, not spare headroom, and an alert on count / limit would read 0% either way.

• describe_table now reads storage. getStructureCounts() does one getBinarySync plus a decode of the shared-structures payload, which two review rounds flagged as a possible describe_all regression. Measured rather than argued — describe_all with skip_record_count: true over 200 tables, 5 rounds each, with the read and with it stubbed out:

  	round 1	2	3	4
  with counts	11.6 ms	16.2	17.6	11.5
  without	14.6 ms	14.6	19.6	14.5

  The distributions overlap, so no cache was added. Worst case is bounded: a fully saturated dictionary is ~7 KB and decodes in ~0.03 ms, and a pre-v5.1.0 table with a legacy over-cap dictionary (e.g. the issue's 1,297 entries) is ~5× that. The stale "the remaining stats below are O(1)" comment was wrong before this change too — getSize() and getAuditSize() read storage — and is corrected.

• No analytics metric, deliberately. The first revision folded these counts onto the table-size record. The review caught that table-size has no RocksDB emitter at all — storeTableSizeMetrics is called only from the non-RocksDB branch of storeDBSizeMetrics — so on the default engine the fields would never have been written. Rather than ship a dead path or resurrect the metric as a rider here (it is a real cardinality decision), that hunk is gone and the gap is filed as The table-size analytics metric is never emitted on RocksDB #2249.

• What this deliberately does not do: no pruning and no renumbering. Structure ids are positional and referenced by every stored record, every transaction-log entry, and every unapplied replication backlog entry, so deletion silently reinterprets records; renumbering is only safe inside a full table re-encode, which is a migration feature rather than a fix. Rationale is written up in the issue reply.

Verification

• Unit — unitTests/resources/recordEncoder.test.js, thirteen cases: the bound holds; a past-cap record still round-trips; counts come from the durable payload even when this encoder's arrays are empty; named-only payloads do not zero the typed count; zero for a never-saved store; the enabled flag; typed vs classic separation; warn-exactly-once; no warn with headroom; a throwing log sink does not fail the write; a declined save does not warn; zeros rather than a throw for a store with no shared-structures mechanism; and the growth model itself (key order and per-field value width each mint distinct shapes) asserted against the installed dependency behavior.
• Fails-on-base — four checks, each red before / green after: the durable-vs-local test against the local-array implementation; the named-only-payload test against a typed: 0 return; the saturation-warning unit test and the integration read_log assertion, both with the two checkStructureCapacity() call sites removed.
• End-to-end — two suites, both green locally:
  • integrationTests/apiTests/structure-dictionary.test.mjs (new) runs with storage.randomAccessFields: true — the only configuration in which a table mints typed structures — writes 400 distinct record shapes, and asserts the dictionary saturates exactly at its limit, that the saturation warning comes back through read_log, and that past-bound records still read back. Measured live: typed 256/256, classic 32, the latter matching msgpackr's own maxSharedStructures bound. The read_log leg matters because the unit tests stub harperLogger, so nothing else proves the operator-facing signal survives a real logger binding.
  • integrationTests/apiTests/configuration.test.mjs asserts the four fields on a default-config table that has rows, so a nonzero classic_structure_count proves real values rather than plumbed-through defaults (26/26).
• Suites — test:unit:resources 1654 passing. test:unit:main could not complete locally (~/harper/database/system/LOCK held by another local instance — known shared-lock contention, not a regression); relying on CI.
• Probes against the installed deps (msgpackr 2.0.5 / structon 1.0.8) confirmed an over-cap dictionary (1,297 entries) loads in full under a 256 cap, keeps encoding existing shapes as structs, and only stops minting new ones — so the bound is safe for the already-grown table that prompted the issue.

Docs companion: HarperFast/documentation#633 — Document record-structure dictionary counts on describe_table

[claude]

Reviewed; no blockers found.

[gemini-code-assist]

Code Review

This pull request introduces tracking and observability for record-structure dictionary counts (both typed and classic) within the RecordEncoder and exposes them via the describe_table operation. It establishes a default maximum limit of 256 typed structures, warns when this limit is reached to alert operators of saturation, and adds comprehensive integration and unit tests to verify correct behavior and round-tripping of records beyond the cap. There are no review comments, so I have no feedback to provide.