fix(invertedindex): stamp flushed posting keys with the deduped doccount

[oc-engteam]

What & why

flushPendingWrites stamped a posting row's on-disk key doccount with the pre-dedup buffer
length. updateIndex appends a docid to a keyword's buffer once per occurrence without dedup (for
ingest speed), so a hot keyword (e.g. every function name in a file contains get) accumulates a flush
row whose buffer length ≫ its unique docid count. But the row VALUE is encodeInvertedValue(docids),
which sorts and dedups. So the key's doccount could be many times the number of docids actually
stored.

The background merger uses that doccount to decide "already well batched": a row with
doccount > maxInvertedIndexSize/2 is skipped from merge groups (keywords_merger.go:242), and a
keyword group with a high DocCount/len(rows) is early-returned (rewriteIndex). An inflated doccount
therefore permanently quarantines hot-keyword rows from compaction — inflating index disk usage
and the number of rows a hot-term Search must scan.

• Search results are unaffected — doccount (3rd return of decodeInvertedKey) is read only by
  the merger; Search/GetDocs read the value bytes. This is a build-throughput / footprint /
  defeated-compaction defect, not a wrong-results bug.
• The delete and merge paths already stamped the true deduped count (the comment at
  invertedindex_internal.go even names this exact "inflated doccount → quarantined forever" hazard) —
  only the common flush path was wrong.

Fix

Make the stamped doccount always equal the value's post-dedup unique count, computed in the same
pass that encodes the value, and route all three writers through one seam:

• encodeInvertedValueCounted(docids) ([]byte, int) — the existing encoder body, now also returning the
  post-Compact unique count (len(us) == the varint count). encodeInvertedValue delegates to it.
• defaultWriteInvertedIndex(idx, batch, tableId, kw, docids, tick) — a single named seam that builds
  the key from that unique count, so a stamped doccount can never again disagree with the row it
  labels. flush / delete / merge all call it (delete & merge are behavior-identical — they already
  passed a deduped set).

No on-disk format change (the key layout is unchanged; only the doccount value in new keys becomes
correct), no reindex, no StorageVersion bump. Existing inflated rows on disk heal only on a natural
re-index — merger self-heal was considered and deliberately left out of scope.

Tests

Two new behavior tests (real Index on a t.TempDir KV, real on-disk format):

• TestFlush_StampsDedupedDoccount — a keyword flushed with duplicate occurrences: asserts the stamped
  doccount equals the value's unique docid count. RED before (3 ≠ 1), GREEN after.
• TestMerge_CompactsHotKeywordRows — several inflated hot-keyword rows (MaxInvertedIndexSize=10,
  each doccount 6 > 5): asserts the merger compacts them and the searchable docid set is preserved.
  RED before (rows quarantined, count unchanged), GREEN after.

encodeInvertedValueCounted and defaultWriteInvertedIndex are both 100% covered; the seam's 10 test
overrides + 1 direct caller were migrated; full package passes with -race.

🤖 Generated with Claude Code
via Happy