Hash-field TTL reap (sweep 2) is unbudgeted — iterates every HashWithTtl key per 100ms tick on the shard loop

[TinDang97]

Summary

The active-expiry tick's 1ms budget covers only sweep 1 (whole-key probabilistic sampling). Sweep 2 — the hash-field TTL reap — then iterates every HashWithTtl key in the db and calls reap_expired_fields_one_hash on each, with no time budget, no sampling, and no batch cap (src/server/expiration.rs:135-143).

A db with many field-TTL'd hashes stalls the shard event loop for the full reap, every 100ms tick. The reap itself can be arbitrarily expensive per key (draining expired fields, possibly downgrading HashWithTtl → Hash or deleting the key).

Compounding it, the key list is collected by a full table scan each tick (hashes_with_field_expiry, src/storage/db/accessors.rs:408 — see #541).

Fix direction

Extend the sweep-1 pattern: carry the same Instant budget into sweep 2 and resume from a cursor next tick, or sample like sweep 1 does. Should ride the expiry-index work in #541 (the HashWithTtl set becomes an indexed population instead of a scan).

Evidence anchors

• src/server/expiration.rs:130 (budget check confined to sweep 1's loop), :135-143 (unbounded sweep 2)

[TinDang97]

Design template from #549's landing: HashWithTtl already maintains min_expiry_ms per value, so a second deadline index keyed (min_expiry_ms, key) would give sweep 2 the same pop-exactly-due shape as whole-key expiry — no scan while the latch is up. The hard part remains the transition surface: value-TYPE changes happen through get_mut/encoding upgrades in more places than TTL writes do, so the writer-sweep + full-scan consistency oracle discipline from #549 (see debug_expiry_index_consistent) is the method to apply, with the oracle extended to compare the hash index against a kind-scan.