Skip to content

perf: represent fort ids as a 17-byte value type - #399

Open
jfberry wants to merge 18 commits into
c/golbat-memory-persistence-6846ccfrom
perf/fortid-value-type
Open

perf: represent fort ids as a 17-byte value type#399
jfberry wants to merge 18 commits into
c/golbat-memory-persistence-6846ccfrom
perf/fortid-value-type

Conversation

@jfberry

@jfberry jfberry commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Supersedes #395. Stacked on c/golbat-memory-persistence-6846cc (#394) — the diff below is this branch's 17 commits only.

Replaces string interning with a fixed-width value type for fort ids, and makes pokemon username persistence opt-in.

Why this and not interning

#395 proposed 4-byte handles into a global intern table. Before committing to that, both shapes were benchmarked against each other at production scale (2M forts, 3.25M pokemon, this repo's own rtree/xsync versions). Results are in the comment on #395; the short version:

string uint32 handle [16]byte+suffix
Forced-GC median 177 ms 128 ms 126 ms
Heap objects 20.6M 19.4M 15.3M
Scan inner loop (2M forts) 181 µs 99 µs 162 µs

On GC — the metric this work exists for — the two options tie. So the decision came down to structure: the value type deletes a category of machinery (global table, unbounded growth, deleted-fort leak, resolve-failure paths at ~35 emit sites) rather than engineering around it, and drives heap objects to the floor. The handle's real advantage is scan throughput, and fort scans are low-traffic today; interning on top of value ids stays available if that changes.

The type

Fort ids are a 128-bit hex GUID plus an optional two-hex-digit suffix, stored as varchar(35):

type FortId struct {
	Guid   [16]byte
	Suffix uint8 // parsed value, raw; 0 = the bare 32-char form
}

17 bytes, comparable, pointer-free — so it is used directly as cache keys, map keys and R-tree payloads, with string conversion only at DB/JSON/webhook/proto boundaries.

Three properties are load-bearing and each is pinned by a test:

  • The suffix is not enumerated. Any two lowercase hex digits parse, so a new Niantic suffix needs no code change. A census of a production instance found .16/.11/.12 on pokestops and gyms and .23/.16/.11/.12 on stations — the set is not per-fort-type.
  • Bare 32-char ids are a live shape, not legacy junk. They are sponsored forts (EE, Community Ambassador Location, Play! Pokémon Store), several updated within the hour of the census. They are treated as Niantic's stripped null suffix, which is also what makes byte order match varchar order: the bare form sorts first, and lowercase hex ascends in ASCII exactly as it ascends in value. That congruence is what lets the write-behind deadlock-avoidance sort stay consistent with ORDER BY id.
  • The zero value is the absent/"None" sentinel, so ParseFortId refuses to produce it — an all-zero GUID with no suffix is rejected, as is "". Exactly one nonconforming id exists in the censused production database (a pokestop with id = ''); it now fails to load with an error log rather than aliasing "no fort".

Wire and schema compatibility

Nothing observable changes. Schema untouched (git diff -- sql/ is empty). API and webhook DTO fields stay string/*string with conversion at the assignment site, so the JSON contract is unchanged — including "pokestop_id": "None" in the pokemon webhook and null (never "") in API responses, both pinned by tests.

Value()/Scan() render and read the same varchar. TestPokemonFullRowRoundTrip reads pokestop_id back as a raw string rather than through FortId.Scan — the only check that catches a Valuer regression, since a self-consistent Scanner/Valuer pair could write garbage into the column and never notice. sqlx.In's reflective expansion of []FortId is pinned separately, because a mistake there is a silently wrong DELETE.

Username

username is no longer persisted by default. New store_username option (default false); the reporting account is threaded from the decode context to its two real consumers — the webhook payload and the shiny/duplicate-encounter dedup.

Both consumers now prefer the live account over the stored one. This is a fix, not just plumbing: the dedup needs the account reporting now, and preferring a frozen stored value would make SetAccountSeen report "already seen" for every distinct account after the first, dropping their shiny checks from the odds statistics before the counters ran.

With the option off, webhooks carry the account that triggered the save rather than the first account to see the pokemon, and ProactiveIVSwitch webhooks (which have no account context) carry "username": null.

Measured size effect

PokemonData 256 → 248 bytes, Pokemon 352 → 344, allocator class unchanged at 352. PokestopId moved out of the pointer group into the 1-byte group, where 17 bytes fill existing padding rather than create it. Station and Incident each grew 8 bytes, also without changing class. The struct-level effect is a wash; the win is ~3.25M fewer heap-allocated strings, which unsafe.Sizeof cannot see.

Testing

Green under both build tags, -race, and golangci-lint v2.12.2. Three MariaDB round-trip tests verified against a real 11.4 instance with migrations applied, including that rows written before this change still load. Every task's diff was reviewed independently, and a whole-branch review re-ran all gates.

Before deploying

FortId.Scan now gates six more columns than the census covered: pokemon.pokestop_id, incident.pokestop_id, route.start_fort_id, route.end_fort_id, tappable.fort_id, station_battle.station_id. A malformed value makes that row unloadable — logged and skipped, blast radius one row. pokemon.pokestop_id is the one to check: the previous code could write '' there unguarded, and with preserve_pokemon enabled every such row is skipped at startup. Worth a SELECT COUNT(*) WHERE <col> = '' per column so any startup noise is a known quantity.

Design doc: docs/superpowers/specs/2026-08-18-fortid-value-type-design.md.

🤖 Generated with Claude Code

jfberry and others added 18 commits August 19, 2026 17:35
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fort ids are a 128-bit hex GUID plus an optional two-hex-digit suffix.
As a fixed-width value they are pointer-free and live inline in caches,
map keys and R-tree payloads instead of costing a heap object per copy.
The zero value is the absent sentinel; parse refuses to produce it.
No callers yet.
Empty input is a real production id (one pokestop row) that must never
alias the zero/absent sentinel: UnmarshalText's len(b)==0 special case
is deleted, so Scan("") and UnmarshalText([]byte{}) now error like any
other malformed id, while Scan(nil) still yields the sentinel with no
error (SQL NULL genuinely means absent).

Also: the one-shot ".00" warning now only fires once a parse is known
to succeed, so an all-zero-GUID ".00" (rejected as the sentinel) can no
longer burn the slot a genuine occurrence needs; UnmarshalText's error
now carries the method name for attribution; and the Compare/string-
order fuzz test now includes the zero FortId against its "" string form.
The deadlock-avoidance sort now takes an explicit KeyCompare instead of
cmp.Compare, so a fixed-width struct key can be used. Extracted the sort
into a named sortEntriesForLockOrder method so it's directly testable.
Every existing queue passes cmp.Compare and sorts exactly as before. A
nil comparator is fatal at construction rather than a panic inside a
later flush.

Also fixes decoder/station_battle_test.go, a second NewTypedQueue
construction site the brief's writebehind_batch.go-scoped grep missed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lookup map, spatial tree, snapshot and evictor now hold 17-byte
values instead of ~35-byte string keys: no per-entry heap object, no
pointer for the GC to mark in tree nodes, no string hashing per scan
candidate. Entity Id fields are still strings, so calls in from the
entity layer parse through the temporary fortid_bridge.go, which Task 7
deletes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every live fort appears twice in the tracker (the forts map and its cell's
pokestop/gym set), so this removes two more string references per fort.
GMO ingest parses once, at the decode boundary. The keyset-pagination
cursor stays a string — it is the database's ordering — and advances on the
raw value so a junk row cannot stall the loader.

Also fixes two pre-existing warts in fortKindOps while editing it: the
comparable constraint was vestigial (an explicit isNil field exists
precisely because comparable cannot compare T against nil), and the
lock-contention caller name was built by concatenation into
'cleargymWithLock', which matches no identifier in the tree — it now
names clearFortWithLock so a contention log can be grepped.

Also converts the four RegisterFort/RestoreFort call sites the compiler
surfaced beyond the brief's file list (preload.go x2, gym_decode.go,
pokestop_decode.go), which still hold entity.Id as a string pending
Tasks 5/7 — these route through the Task 3 bridge (fortIdFromLegacyString).
TestFortTrackerParseFailureAdvancesCursor never drove the keyset loader; it
only checked ParseFortId in isolation. Extracted loadFortKindFromDB's
per-page row application into applyFortRows(table, rows, isGym, nowMs)
(applied int, cursor string) so a synthetic []fortRow batch can exercise
the real path. Two new tests replace it: a three-row batch with a
malformed middle row (both good rows applied, cursor is the last row's raw
id) and a batch where the LAST row is malformed (cursor is still that raw
id — the exact shape that would otherwise stall the loader on a re-fetch
loop). Behavior is unchanged: same locking (once per page), same lastId
semantics, same log line on a malformed id.

Also: TODO comments on gymClearOps/pokestopClearOps's loadForUpdate
closures noting they collapse back to bare function references once
Task 5 lands, and reworded fortid_bridge.go's doc comment — "do not add
new callers" was already false after this task's four compiler-forced
sites, when the real intent was "do not use this as a general-purpose
parse-or-ignore helper outside the conversion."
Entity fields, cache keys, write-behind queue keys and every record
accessor now carry the 17-byte value. Ids convert to strings only at the
SQL, JSON and webhook boundaries, and proto ingest parses once with an
error log for structurally malformed ids.
dedupeIDs now checks the raw request length against maxQueryIDs before
parsing, so an oversized batch of malformed ids still trips 413 instead of
being silently reduced to an empty, cap-compliant list; parse failures
aggregate into one log line per call instead of one per dropped id.

The "pokestop/id for unknown id" test used a malformed fixture id, which
made it 404 out of ParseFortId instead of exercising PeekPokestopRecord's
cache-miss path — split into a well-formed-but-absent case (the original
intent) and a dedicated malformed-id case. Added a regression test pinning
the restored 413-on-oversized-malformed-batch contract.

Also inlined the single-use `id := pokestop.Id` / `id := gym.Id` aliases
fortRtree.go's bridge-call removal left behind.
Includes the station-battle content hash, which now folds the id's bytes
instead of its string form. Battle snapshot signatures are per-process
(the maphash seed is generated at startup), so nothing persisted changes.
The ORDER BY station_id grouped stream is unaffected: FortId's byte order
matches the varchar collation.

Also closes the query-stations cap-check bypass carried over from the
Task 5 review: the raw request length is checked before parsing/dedup,
matching query-gyms, with a pinning test for both well-formed and
malformed oversized requests. Removed writeString (dead after the hash
change, no other callers in this codebase) to keep golangci-lint's
unused check clean, and updated the entity-size pin for Station's
FortId-driven struct growth (472->480, no allocator class regression).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Incident.PokestopId, Tappable.FortId, and Route.StartFortId/EndFortId are
now FortId. Tappable's optional fort reference drops null.String for
FortId's zero value; its API field stays *string so the JSON keeps
emitting null. Incident and route ids themselves are not fort ids and are
unchanged.

Route's start/end fort ids are NOT NULL columns and functionally identify
the route (unlike Tappable's nullable fort_id), so they're parsed upfront
in UpdateRouteRecordWithSharedRouteProto and the whole update is abandoned
on parse failure, mirroring the existing incident_process.go pattern for
Incident.PokestopId (also NOT NULL) rather than risking a NULL write.

This removes the six incident-path callers of the temporary parse bridge.
The bridge (decoder/fortid_bridge.go) stays in place: two pokemon-path
callers remain (Pokemon.PokestopId), which the next task converts and
which then deletes the file.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Removes one heap object and a ~35-byte duplicated string per cached
pokemon — about 3.25M of each in production, against maybe 500k distinct
forts. The webhook 'None' sentinel and the API's null pokestop_id are
preserved exactly; an absent fort writes SQL NULL, not an empty string.

Also deletes decoder/fortid_bridge.go: its last two call sites
(pokemon_state.go's createPokemonWebhooks, pokemon_decode.go's
updateFromNearby) depended on this conversion and are gone now that
PokestopId is FortId end to end. `grep -rn fortIdFromLegacyString`
returns nothing — the completion check for the whole fort-id conversion.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The account name is threaded from the decode context to the webhook
payload and the shiny/duplicate-encounter dedup, which are its only
consumers — and the dedup wants the account reporting now, not the
first-seen one, so that path is more correct than before. Operators who
want the column populated set store_username = true.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review round 1 on the username-persistence task found that preferring
the stored (first-wins) column value over the live/reporting account
corrupted the shiny/duplicate-encounter dedup: once a second account
touched a pokemon, its snapshot carried the frozen first account's
name, so updateEncounterStats treated it as an already-seen duplicate
and silently dropped it from the stats. The webhook payload had the
same bug in a lower-stakes form (wrong reporter shown).

Extract the shared precedence into resolveUsername (live wins when
present, falls back to stored only when the caller has no live
account — e.g. weather_iv.go's proactive re-save) and use it in both
createPokemonWebhooks and statsSnapshot. setUsernameIfStored's
first-wins gate is unchanged — it correctly governs persistence, just
not what the two consumers read.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CLAUDE.md's fort-scaling caveat pre-scoped interning as the lever; the
benchmarked outcome was the value type instead, which removes the same
costs without a global table. Interning remains available on top if fort
scans ever become hot.

Also corrects the design spec's §1/§5 claim that choosing the value type
over interning cost Pokemon "returning to the 352 allocator class
(~104 MB at 3.25M cached)". That never happened: this branch descends
from PR 394's base (already 352 bytes, no interning ever applied), so
there was no smaller class to give up. Measured sizes actually shrank —
PokemonData 256->248, Pokemon 352->344 — with the allocator class
unchanged at 352.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Struct Pattern illustration and the fort tracker data model diagram
still showed fort ids as string (Id string, TrackedMutex[string],
map[string]*FortTrackerLastSeen) after the FortId conversion. Follow-up
to fc3eb63, per coordinator review.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nine agreed fixes from the whole-branch review of the FortId value-type
conversion (Ready to merge: Yes, no Critical findings):

1. Aggregate decode-path fort-id parse-failure logs via a new
   decoder.FortIdParseDrops (util.DropReporter), routed through the six
   hottest sites across decode.go/gmo_decode.go/pokemon_decode.go —
   bounds log volume to ~1 line/s if Niantic ever changes the id
   structure, matching the house DropReporter pattern.
2. Add FortId.Ptr() (mirrors null.String.Ptr()) and use it at the four
   duplicated absent-fort-to-*string call sites.
3. Assert pokestop_id == null in the existing nullables test.
4. Drop the redundant `id FortId` parameter from
   updatePokestopLookup/updateGymLookup/updateStationLookup — read it
   off the entity instead.
5. Fix fort_tracker.go's startup count to use applyFortRows' `applied`
   return instead of len(rows), so malformed rows aren't counted as
   loaded.
6. Correct three inaccurate comments/spec text: the username test's
   "pre-existing behavior" claim, config.toml.example's store_username
   no-account case, and the design spec's pokemon-preserve JSON claim
   (it's a DB round-trip, not JSON).
7. Document AppendText/MarshalText's zero-value write-only asymmetry.
8. Fix two stale/duplicate comments in writebehind/typed_queue.go.
9. Add a shared "unparseable" token to FortId.UnmarshalText's error so
   ingest-site and DB-scan-wrapper failures share one grep.

Baseline vs final: 239 -> 240 PASS by name (added TestFortIdPtr), 0
FAIL, same 3 SKIP (MariaDB-only tests). go build (both tags), go vet,
golangci-lint, and go test ./decoder/ -race all clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two follow-ups from review of the store_username option.

setUsernameIfStored now requires the record to be new or already dirty.
SetUsername has never set the dirty flag, so a username could not trigger
a write by itself — but the field was still claimed in memory regardless,
and being first-wins that let a sighting which changed nothing take the
field and block the account whose update actually got written. A row could
end up storing an account that contributed none of its data. The gate
mirrors savePokemonRecordAsAtTime's own entry condition, so the account is
adopted exactly when the row is headed for the queue. The call sites move
to the end of their update functions, since the gate only means anything
after the setters that decide dirtiness have run; the tappable path drops
its own call, as addEncounterPokemon runs immediately after and does it.

The encounter dedup now keys on a per-process hash of the account rather
than the name. It only ever asks whether an account was seen before and
how many were distinct — it never reads a name back — so hashing costs
nothing behaviourally, and it stops a cache from holding caller-supplied
account names for an entry's whole TTL. Retaining them there would have
undercut making persistence opt-in in the first place.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant