perf: represent fort ids as a 17-byte value type - #399
Open
jfberry wants to merge 18 commits into
Open
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fort ids are a 128-bit hex GUID plus an optional two-hex-digit suffix. As a fixed-width value they are pointer-free and live inline in caches, map keys and R-tree payloads instead of costing a heap object per copy. The zero value is the absent sentinel; parse refuses to produce it. No callers yet.
Empty input is a real production id (one pokestop row) that must never
alias the zero/absent sentinel: UnmarshalText's len(b)==0 special case
is deleted, so Scan("") and UnmarshalText([]byte{}) now error like any
other malformed id, while Scan(nil) still yields the sentinel with no
error (SQL NULL genuinely means absent).
Also: the one-shot ".00" warning now only fires once a parse is known
to succeed, so an all-zero-GUID ".00" (rejected as the sentinel) can no
longer burn the slot a genuine occurrence needs; UnmarshalText's error
now carries the method name for attribution; and the Compare/string-
order fuzz test now includes the zero FortId against its "" string form.
The deadlock-avoidance sort now takes an explicit KeyCompare instead of cmp.Compare, so a fixed-width struct key can be used. Extracted the sort into a named sortEntriesForLockOrder method so it's directly testable. Every existing queue passes cmp.Compare and sorts exactly as before. A nil comparator is fatal at construction rather than a panic inside a later flush. Also fixes decoder/station_battle_test.go, a second NewTypedQueue construction site the brief's writebehind_batch.go-scoped grep missed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lookup map, spatial tree, snapshot and evictor now hold 17-byte values instead of ~35-byte string keys: no per-entry heap object, no pointer for the GC to mark in tree nodes, no string hashing per scan candidate. Entity Id fields are still strings, so calls in from the entity layer parse through the temporary fortid_bridge.go, which Task 7 deletes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every live fort appears twice in the tracker (the forts map and its cell's pokestop/gym set), so this removes two more string references per fort. GMO ingest parses once, at the decode boundary. The keyset-pagination cursor stays a string — it is the database's ordering — and advances on the raw value so a junk row cannot stall the loader. Also fixes two pre-existing warts in fortKindOps while editing it: the comparable constraint was vestigial (an explicit isNil field exists precisely because comparable cannot compare T against nil), and the lock-contention caller name was built by concatenation into 'cleargymWithLock', which matches no identifier in the tree — it now names clearFortWithLock so a contention log can be grepped. Also converts the four RegisterFort/RestoreFort call sites the compiler surfaced beyond the brief's file list (preload.go x2, gym_decode.go, pokestop_decode.go), which still hold entity.Id as a string pending Tasks 5/7 — these route through the Task 3 bridge (fortIdFromLegacyString).
TestFortTrackerParseFailureAdvancesCursor never drove the keyset loader; it only checked ParseFortId in isolation. Extracted loadFortKindFromDB's per-page row application into applyFortRows(table, rows, isGym, nowMs) (applied int, cursor string) so a synthetic []fortRow batch can exercise the real path. Two new tests replace it: a three-row batch with a malformed middle row (both good rows applied, cursor is the last row's raw id) and a batch where the LAST row is malformed (cursor is still that raw id — the exact shape that would otherwise stall the loader on a re-fetch loop). Behavior is unchanged: same locking (once per page), same lastId semantics, same log line on a malformed id. Also: TODO comments on gymClearOps/pokestopClearOps's loadForUpdate closures noting they collapse back to bare function references once Task 5 lands, and reworded fortid_bridge.go's doc comment — "do not add new callers" was already false after this task's four compiler-forced sites, when the real intent was "do not use this as a general-purpose parse-or-ignore helper outside the conversion."
Entity fields, cache keys, write-behind queue keys and every record accessor now carry the 17-byte value. Ids convert to strings only at the SQL, JSON and webhook boundaries, and proto ingest parses once with an error log for structurally malformed ids.
dedupeIDs now checks the raw request length against maxQueryIDs before parsing, so an oversized batch of malformed ids still trips 413 instead of being silently reduced to an empty, cap-compliant list; parse failures aggregate into one log line per call instead of one per dropped id. The "pokestop/id for unknown id" test used a malformed fixture id, which made it 404 out of ParseFortId instead of exercising PeekPokestopRecord's cache-miss path — split into a well-formed-but-absent case (the original intent) and a dedicated malformed-id case. Added a regression test pinning the restored 413-on-oversized-malformed-batch contract. Also inlined the single-use `id := pokestop.Id` / `id := gym.Id` aliases fortRtree.go's bridge-call removal left behind.
Includes the station-battle content hash, which now folds the id's bytes instead of its string form. Battle snapshot signatures are per-process (the maphash seed is generated at startup), so nothing persisted changes. The ORDER BY station_id grouped stream is unaffected: FortId's byte order matches the varchar collation. Also closes the query-stations cap-check bypass carried over from the Task 5 review: the raw request length is checked before parsing/dedup, matching query-gyms, with a pinning test for both well-formed and malformed oversized requests. Removed writeString (dead after the hash change, no other callers in this codebase) to keep golangci-lint's unused check clean, and updated the entity-size pin for Station's FortId-driven struct growth (472->480, no allocator class regression). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Incident.PokestopId, Tappable.FortId, and Route.StartFortId/EndFortId are now FortId. Tappable's optional fort reference drops null.String for FortId's zero value; its API field stays *string so the JSON keeps emitting null. Incident and route ids themselves are not fort ids and are unchanged. Route's start/end fort ids are NOT NULL columns and functionally identify the route (unlike Tappable's nullable fort_id), so they're parsed upfront in UpdateRouteRecordWithSharedRouteProto and the whole update is abandoned on parse failure, mirroring the existing incident_process.go pattern for Incident.PokestopId (also NOT NULL) rather than risking a NULL write. This removes the six incident-path callers of the temporary parse bridge. The bridge (decoder/fortid_bridge.go) stays in place: two pokemon-path callers remain (Pokemon.PokestopId), which the next task converts and which then deletes the file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Removes one heap object and a ~35-byte duplicated string per cached pokemon — about 3.25M of each in production, against maybe 500k distinct forts. The webhook 'None' sentinel and the API's null pokestop_id are preserved exactly; an absent fort writes SQL NULL, not an empty string. Also deletes decoder/fortid_bridge.go: its last two call sites (pokemon_state.go's createPokemonWebhooks, pokemon_decode.go's updateFromNearby) depended on this conversion and are gone now that PokestopId is FortId end to end. `grep -rn fortIdFromLegacyString` returns nothing — the completion check for the whole fort-id conversion. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The account name is threaded from the decode context to the webhook payload and the shiny/duplicate-encounter dedup, which are its only consumers — and the dedup wants the account reporting now, not the first-seen one, so that path is more correct than before. Operators who want the column populated set store_username = true. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review round 1 on the username-persistence task found that preferring the stored (first-wins) column value over the live/reporting account corrupted the shiny/duplicate-encounter dedup: once a second account touched a pokemon, its snapshot carried the frozen first account's name, so updateEncounterStats treated it as an already-seen duplicate and silently dropped it from the stats. The webhook payload had the same bug in a lower-stakes form (wrong reporter shown). Extract the shared precedence into resolveUsername (live wins when present, falls back to stored only when the caller has no live account — e.g. weather_iv.go's proactive re-save) and use it in both createPokemonWebhooks and statsSnapshot. setUsernameIfStored's first-wins gate is unchanged — it correctly governs persistence, just not what the two consumers read. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CLAUDE.md's fort-scaling caveat pre-scoped interning as the lever; the benchmarked outcome was the value type instead, which removes the same costs without a global table. Interning remains available on top if fort scans ever become hot. Also corrects the design spec's §1/§5 claim that choosing the value type over interning cost Pokemon "returning to the 352 allocator class (~104 MB at 3.25M cached)". That never happened: this branch descends from PR 394's base (already 352 bytes, no interning ever applied), so there was no smaller class to give up. Measured sizes actually shrank — PokemonData 256->248, Pokemon 352->344 — with the allocator class unchanged at 352. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Struct Pattern illustration and the fort tracker data model diagram still showed fort ids as string (Id string, TrackedMutex[string], map[string]*FortTrackerLastSeen) after the FortId conversion. Follow-up to fc3eb63, per coordinator review. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nine agreed fixes from the whole-branch review of the FortId value-type conversion (Ready to merge: Yes, no Critical findings): 1. Aggregate decode-path fort-id parse-failure logs via a new decoder.FortIdParseDrops (util.DropReporter), routed through the six hottest sites across decode.go/gmo_decode.go/pokemon_decode.go — bounds log volume to ~1 line/s if Niantic ever changes the id structure, matching the house DropReporter pattern. 2. Add FortId.Ptr() (mirrors null.String.Ptr()) and use it at the four duplicated absent-fort-to-*string call sites. 3. Assert pokestop_id == null in the existing nullables test. 4. Drop the redundant `id FortId` parameter from updatePokestopLookup/updateGymLookup/updateStationLookup — read it off the entity instead. 5. Fix fort_tracker.go's startup count to use applyFortRows' `applied` return instead of len(rows), so malformed rows aren't counted as loaded. 6. Correct three inaccurate comments/spec text: the username test's "pre-existing behavior" claim, config.toml.example's store_username no-account case, and the design spec's pokemon-preserve JSON claim (it's a DB round-trip, not JSON). 7. Document AppendText/MarshalText's zero-value write-only asymmetry. 8. Fix two stale/duplicate comments in writebehind/typed_queue.go. 9. Add a shared "unparseable" token to FortId.UnmarshalText's error so ingest-site and DB-scan-wrapper failures share one grep. Baseline vs final: 239 -> 240 PASS by name (added TestFortIdPtr), 0 FAIL, same 3 SKIP (MariaDB-only tests). go build (both tags), go vet, golangci-lint, and go test ./decoder/ -race all clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two follow-ups from review of the store_username option. setUsernameIfStored now requires the record to be new or already dirty. SetUsername has never set the dirty flag, so a username could not trigger a write by itself — but the field was still claimed in memory regardless, and being first-wins that let a sighting which changed nothing take the field and block the account whose update actually got written. A row could end up storing an account that contributed none of its data. The gate mirrors savePokemonRecordAsAtTime's own entry condition, so the account is adopted exactly when the row is headed for the queue. The call sites move to the end of their update functions, since the gate only means anything after the setters that decide dirtiness have run; the tappable path drops its own call, as addEncounterPokemon runs immediately after and does it. The encounter dedup now keys on a per-process hash of the account rather than the name. It only ever asks whether an account was seen before and how many were distinct — it never reads a name back — so hashing costs nothing behaviourally, and it stops a cache from holding caller-supplied account names for an entry's whole TTL. Retaining them there would have undercut making persistence opt-in in the first place. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #395. Stacked on
c/golbat-memory-persistence-6846cc(#394) — the diff below is this branch's 17 commits only.Replaces string interning with a fixed-width value type for fort ids, and makes pokemon username persistence opt-in.
Why this and not interning
#395 proposed 4-byte handles into a global intern table. Before committing to that, both shapes were benchmarked against each other at production scale (2M forts, 3.25M pokemon, this repo's own rtree/xsync versions). Results are in the comment on #395; the short version:
[16]byte+suffixOn GC — the metric this work exists for — the two options tie. So the decision came down to structure: the value type deletes a category of machinery (global table, unbounded growth, deleted-fort leak, resolve-failure paths at ~35 emit sites) rather than engineering around it, and drives heap objects to the floor. The handle's real advantage is scan throughput, and fort scans are low-traffic today; interning on top of value ids stays available if that changes.
The type
Fort ids are a 128-bit hex GUID plus an optional two-hex-digit suffix, stored as
varchar(35):17 bytes, comparable, pointer-free — so it is used directly as cache keys, map keys and R-tree payloads, with string conversion only at DB/JSON/webhook/proto boundaries.
Three properties are load-bearing and each is pinned by a test:
.16/.11/.12on pokestops and gyms and.23/.16/.11/.12on stations — the set is not per-fort-type.ORDER BY id.ParseFortIdrefuses to produce it — an all-zero GUID with no suffix is rejected, as is"". Exactly one nonconforming id exists in the censused production database (a pokestop withid = ''); it now fails to load with an error log rather than aliasing "no fort".Wire and schema compatibility
Nothing observable changes. Schema untouched (
git diff -- sql/is empty). API and webhook DTO fields staystring/*stringwith conversion at the assignment site, so the JSON contract is unchanged — including"pokestop_id": "None"in the pokemon webhook andnull(never"") in API responses, both pinned by tests.Value()/Scan()render and read the same varchar.TestPokemonFullRowRoundTripreadspokestop_idback as a raw string rather than throughFortId.Scan— the only check that catches aValuerregression, since a self-consistent Scanner/Valuer pair could write garbage into the column and never notice.sqlx.In's reflective expansion of[]FortIdis pinned separately, because a mistake there is a silently wrongDELETE.Username
usernameis no longer persisted by default. Newstore_usernameoption (defaultfalse); the reporting account is threaded from the decode context to its two real consumers — the webhook payload and the shiny/duplicate-encounter dedup.Both consumers now prefer the live account over the stored one. This is a fix, not just plumbing: the dedup needs the account reporting now, and preferring a frozen stored value would make
SetAccountSeenreport "already seen" for every distinct account after the first, dropping their shiny checks from the odds statistics before the counters ran.With the option off, webhooks carry the account that triggered the save rather than the first account to see the pokemon, and
ProactiveIVSwitchwebhooks (which have no account context) carry"username": null.Measured size effect
PokemonData256 → 248 bytes,Pokemon352 → 344, allocator class unchanged at 352.PokestopIdmoved out of the pointer group into the 1-byte group, where 17 bytes fill existing padding rather than create it.StationandIncidenteach grew 8 bytes, also without changing class. The struct-level effect is a wash; the win is ~3.25M fewer heap-allocated strings, whichunsafe.Sizeofcannot see.Testing
Green under both build tags,
-race, andgolangci-lintv2.12.2. Three MariaDB round-trip tests verified against a real 11.4 instance with migrations applied, including that rows written before this change still load. Every task's diff was reviewed independently, and a whole-branch review re-ran all gates.Before deploying
FortId.Scannow gates six more columns than the census covered:pokemon.pokestop_id,incident.pokestop_id,route.start_fort_id,route.end_fort_id,tappable.fort_id,station_battle.station_id. A malformed value makes that row unloadable — logged and skipped, blast radius one row.pokemon.pokestop_idis the one to check: the previous code could write''there unguarded, and withpreserve_pokemonenabled every such row is skipped at startup. Worth aSELECT COUNT(*) WHERE <col> = ''per column so any startup noise is a known quantity.Design doc:
docs/superpowers/specs/2026-08-18-fortid-value-type-design.md.🤖 Generated with Claude Code