Skip to content

perf: shrink cached Pokemon 800→352 bytes (field packing, plain-struct scan history) - #394

Open
TurtIeSocks wants to merge 74 commits into
mainfrom
c/golbat-memory-persistence-6846cc
Open

perf: shrink cached Pokemon 800→352 bytes (field packing, plain-struct scan history)#394
TurtIeSocks wants to merge 74 commits into
mainfrom
c/golbat-memory-persistence-6846cc

Conversation

@TurtIeSocks

@TurtIeSocks TurtIeSocks commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Shrinks the cached Pokemon struct from 800 bytes to 352, moving it from Go's 896-byte allocator size class to the 352-byte class — 544 fewer bytes per cached pokemon. PokemonLookup, the per-candidate scan struct, also drops 18 → 16 bytes. Measured on production instance under live traffic:

main this branch
pm2 RSS 15 GB 9.2–9.3 GB (−38%)
GC mark CPU 24.22% ~19%
live heap 6.66 GB ~4.2 GB
pokemon 896 B/obj 352 B/obj
spawnpoint 144 B/obj 112 B/obj

String interning was split out to #395 at @jfberry's request, so it is no longer part of this change.

Why 512 bytes

Go's GC treats objects above 512 bytes materially worse. Measured across 5M live entries with forced GC, mean of 5 runs:

object shape GC mark
800 B, 12 pointers (the original Pokemon) 279.7 ms
800 B, 1 pointer 271.6 ms
800 B, 0 pointers 62.4 ms
512 B, 1 pointer 62.6 ms
520 B, 1 pointer 194.4 ms
256 B, 1 pointer 33.2 ms

Dropping pointer count from 12 to 1 buys 3%. Adding 8 bytes across the 512 boundary costs 3.1x. TestPokemonUnderGCThreshold guards the line.

Approach

Nullable fields use null.Value[T]. guregu/null's non-generic types are 16 bytes each because they embed sql.NullInt64, and most nullable pokemon columns are tinyint or smallint. null.Value[T] embeds stdlib sql.Null[T] and measures 2/4/8/16/8/2 bytes for uint8/uint16/uint32/uint64/float32/bool. Scan range-rejects and encoding/json renders float32 at 32-bit precision, so both behaviours an earlier draft hand-rolled come from the library.

cell_id and spawn_id are null.Value[int64]. sql.Null[uint64] cannot scan a negative int64 (cell_id is a signed bigint whose real values are frequently negative), and its Value() rejects any uint64 with the high bit set.

Scan history is a plain Go struct. Pokemon embedded grpc.PokemonInternal, a generated protobuf message used purely as in-memory Ditto state, carrying MessageState/SizeCache/UnknownFields at 64 bytes per pokemon and 88 per scan — and only marshalled behind four ANDed conditions with the controlling flag defaulting off. Now []*pokemonScan, 24-byte header and 44 bytes per entry, with protobuf built on demand at the two DB boundaries. Elements stay behind pointers deliberately: the Ditto code holds a pointer to a history entry across other work and mutates through it, so a value slice would have &s[i] invalidated by append.

Fields ordered by descending alignment, and three deleted: Capture1/2/3 (setters with no callers, in neither the select columns nor the upsert), changedFields (const-folded dead in production builds, but its 24-byte header and one GC-scanned word stayed), and seen_type as a string (an eight-value enum in a 24-byte null.String plus a heap pointer, now a one-byte code).

The debug change accumulator rolled out to every entity. Four crossed an allocator class — Station, Spawnpoint, Incident, Tappable. Pokestop, Gym, Route and Player shed 24 bytes but stayed in the same class, so they save nothing at allocation time.

What the profile shows

The first version of this description estimated the CPU saving at "low single-digit percent," sourced from docs/decode-performance-findings.md. That 5.4% figure is from the protobench harness, not production, and was wrong by roughly 5x. @jfberry's "absolutely will need to be benchmarked" was the right call.

Measured on main: GC mark 24.22% of CPU, live heap 6.66 GB against ~14 GB RSS — so about half the process was collector headroom, with GOGC unset. Pokemon were 2.71 GB of that, 40.8%, at ~3.25M cached.

On this branch, per-entity bytes match the predicted size-class ratios to three decimals:

pokemon      896.0 -> 352.0 B/obj   ratio 0.393   size-class predicts 0.393
spawnpoint   144.0 -> 112.0 B/obj   ratio 0.778   size-class predicts 0.778
scan entry    55.8 ->  25.4 B/obj   ratio 0.455   size-class predicts 0.500

Scan entries beat prediction because they also lost their protobuf machinery, not just a size class.

The RSS win outran the struct arithmetic by 1.6x. The model said 1.84 GB of live heap saved, doubled for GOGC=100 headroom, so ~3.7 GB of RSS. pm2 shows 5.75 GB. The difference is second-order and the model did not include it: fewer and more uniform objects mean less GC metadata and less span fragmentation, and HeapReleased went from 8 MB to 647 MB — the runtime is now returning pages where before it essentially was not.

Caveats on the measurement

The two captures were taken at different points in the day. main at 19:17 (evening peak, 3.25M pokemon), this branch at 07:40 (morning trough, 2.74M). That gap is diurnal, not cache warmth. Adjusting for 0.51M more pokemon at 352 B with the GOGC doubling puts the branch near 9.6 GB at peak — still ~5.4 GB and 36% below baseline, but a capture at ~19:00 would settle it properly.

The GC figure is two samples, 18.56% and 19.58%, against a baseline that was a single 30s sample. Read it as ~19% ± 0.5 and ~5 points saved, not 5.7.

The capture is ~25 commits behind. Every production number here was taken on this branch at bca1669. Four review rounds have landed since, all of them correctness fixes rather than size changes, so Pokemon is still 352 — but the figures have not been re-observed.

Nothing has been measured on go 1.27, whose allocator changes could move all of this. Raised by @jfberry and still open.

Wire changes

API responses: weight, height and iv render with fewer digits — 6.7 rather than 6.699999809265137. They originate as protobuf float fields promoted through float64(), so the extra digits were an artifact of that promotion. Verified byte-identical across 500,018 values through both the stdlib and goccy encoders.

Webhook payloads: same fields, same change.

The OpenAPI schema changed on 19 fields, and separately on four timestamps — 16 integers format: int64int32, 3 floats doublefloat. Worth calling out separately: narrowing to unsigned Go types also made huma emit minimum: 0 on those 16 fields. That is a validation constraint rather than a format label, so a strict client or validating proxy would begin rejecting negatives. Accurate to the domain — none of those fields can legitimately be negative — but it is new contract surface beyond the format change, and it was not part of the original ask. TestApiPokemonResultSchemaWidths now pins the shape so a future width change cannot drift it silently.

A later round then took the four timestamp fields the other way. expire_timestamp and updated had become format: int32, which overflows generated clients in 2038 — and inconsistently, since first_seen_timestamp stayed int64. All four are now format: int64 with minimum: 0. Note the second half of that: first_seen_timestamp and changed gain a floor they never advertised, because huma had nothing to infer one from at int64. Accurate to the domain, but new constraint surface.

seen_type enum strings and the golbat_internal protobuf are unchanged.

Review feedback

Round 1null already supports generics (the decoder/nulltypes package an earlier draft added is deleted, −702 lines); pull the SeenTypeCodeWild = 0 fix in (code 0 is now an explicit Unset sentinel); the webhook re-widening is wasted (gone); the embedded protobuf is ripe for change (done, stacked here); rename the change list and roll it out (done).

Round 2 — four asks, all addressed:

  1. SetSeenType takes a SeenTypeCode, not a string. The constants are typed, so a typo now fails to compile instead of silently no-opping, and the decode path loses a map lookup.
  2. The widen layer is gone. widenPtr/widenFloatPtr deleted across 19 call sites by narrowing the response types.
  3. Clamp comparisons converge. This was a real bug and the most serious thing in the round — see below.
  4. NullSeenType.Scan degrades instead of failing. An unrecognised enum value now warns and yields invalid rather than erroring, because an erroring Scan stranded a newRecord cache entry that re-failed its DB load on every sighting, silently dropping all processing for that pokemon for the duration of a mixed deployment. Value() stays strict — writing an out-of-enum string to a MariaDB ENUM is silently stored as ''.

On #3, since it was introduced by this PR

Several sites compared a clamp-saturated stored value against a raw proto value. With a costume of 300, the stored value saturates at 255 and 255 != 300 on every sighting, forever. The branch that guards nulls out weight, height, size, moves, cp, shiny, ditto and pvp — so every encounter's enrichment was undone by the next sighting, fleet-wide.

An earlier review on this branch flagged it as Minor and "not reachable with today's proto values," and it shipped on that basis. That was wrong twice over: pogo enums are open int32, so unknown and negative wire values pass straight through, and Golbat ingests raw protos from third-party scanners — a malformed packet is enough.

Fixed at 7 sites across 5 functions (3 of which the ask did not name) with non-counting narrowUint8/16/32 helpers on comparison sides, the counting clampUint* delegating to them so golbat_field_clamped_total still fires exactly once per real store. Regression tests cover each site individually — reverting one fails only its own test.

Round 4 — four asks plus a ranked list, all addressed. Two of the four turned out deeper than stated:

  • An unrecognised seen_type was degrading to Unset, which is an active state. updateFromWild downgraded the record, updateFromNearby replaced precise coordinates with cell centres, and the save wrote the damage back over a newer binary's value. Pre-PR main stored the string opaquely and every switch fell through harmlessly, so the degrade shipped worse than what it replaced — in exactly the mixed-deployment case it was built for. Now a distinct unknown code, inert at all 17 switch sites, with COALESCE on both write paths so it refuses to overwrite.
  • calculateIv computed Iv from the raw IV sum while storing clamped values. iv is float(5,2) unsigned, so a sum above 450 produces a value MariaDB rejects under STRICT_TRANS_TABLES — failing the entire multi-row batch upsert. Narrowing the inputs as specified did not close it, because the clamp used the column's 255 rather than the game's 15. Clamping at 15 does, and subsumes it: sum ≤ 45 means Iv ≤ 100.

Chasing that second one surfaced a third bug nobody had connected: PokemonLookup's int8 fields use -1 for "absent", and int8(255) is -1. A clamped value entered the scan index as the sentinel for never encountered, silently changing which DNF filters matched. Every lookup field now saturates at a sentinel-safe ceiling instead of truncating — covering the whole ≥128 band, not just the exact ceiling — and Form gets its own helper because its -1 is the wildcard-form key rather than an absence marker. Atk/Def/Sta/Iv were reachable from rows already in the database, since a write-side clamp cannot heal stored data.

CI also now runs under the production build tag. It was running go test ./... without -tags go_json, so every golden-JSON proof in this PR was verified against stdlib while production serves through goccy. Nothing failed once corrected — goccy matches byte-for-byte across every fixture and adversarial case tested — but the proofs were unenforced against the shipping codec. A MariaDB service container now makes the round-trip tests actually execute rather than skip.

Rounds 5 and 6@Mygod flagged that the protobuf-to-plain-struct conversion drops unknown fields when an older binary rewrites golbat_internal. A guard was built to refuse the overwrite; @jfberry ruled it an overreach — the field is transitory, the flag defaults off, and losing fidelity there is harmless — so it was reverted. Four smaller asks landed instead: the proactive-IV skip now compares weather in one encoding rather than testing the lookup's -1 against the entity's 0; NullSeenType.UnmarshalJSON yields the inert Unknown sentinel rather than the destructive Unset; the warn throttles are reset by tests instead of swapped, closing a latent race; and three stale doc claims are corrected. CI also now vets the untagged build, which nothing else covered.

Testing

  • Live MariaDB round-trip through the production pokemonBatchUpsertQuery: a row with every nullable column NULL, and a fully-populated row covering the full 64-bit cell_id, spawn_id, and float64 lat/lon precision.
  • Wire compatibility for golbat_internal: a hand-written byte literal in the old grpc.PokemonInternal shape decodes through the new path, and the new write path produces bytes byte-identical to it.
  • TestPokemonScanCoversEveryProtoField reflects over both structs and diffs exported field-name sets, so adding a proto field without mapping it fails loudly.
  • Clamp convergence and clamp counting, each pinned separately.
  • TestPokemonEntitySizes pins the sizes; TestPokemonUnderGCThreshold guards the 512 line.
  • Green under both build tags, -race, golangci-lint, and the full suite.

statsCollector became an atomic.Pointer seeded with a noop at package init. A test needed to swap it, and decoder/init_test.go already documented that doing so races the stats aggregation worker, which reads the global on a ticker for the life of the process. This also retires a pre-existing race in station_battle_test.go.

Follow-ups

ProactiveIVSwitch's boosted-weather guard never fires — pre-existing on main, confirmed by @Mygod. boostedWeathers&uint8(1)<<w != 0 parses as (boostedWeathers & 1) << w, since Go gives & and << equal precedence. Every boostedWeatherLookup entry is even, so the guard is unconditionally false and newWeather is always 0 — the switch can only remove boosts, never apply them, and a currently-boosted pokemon whose new weather still boosts it gets wrongly de-boosted. Separately, NewWeather arrives as an unvalidated int32 from an open proto3 enum, and a negative shift count panics in a goroutine with no recover in its chain, so one malformed packet takes the process down. Both belong in their own PR, since the fix changes live behaviour immediately.

  • The same negative-aliasing pattern is still live in fortRtree.go — bare int8/int16 casts on quest reward amounts, team and slots. The lookupInt8/lookupInt16 helpers added here apply verbatim.

  • clampUint's derived ceiling would silently wrap for a future ~uint64 instantiation; a limit < 0 panic guard is cheap.

  • The statsCollectorSet ordering guard doesn't cover the exported InitTypedQueues.

  • TestUpdatePokemonLookupSaturatesClampedFields leaks a phantom entry into pokemonFormCount.

  • The publish workflow's build-gate comment overclaims (branch pushes still publish per-branch images; the load-bearing case is fork-PR tokens), and tag-push releases now block on MariaDB service health.

  • Bit-packing the numeric fields. Deliberately not done here. From 312 bytes, packing the small numerics into shared words reaches roughly 241 (the 256 class); adding a database row shim so Pvp and GolbatInternal can leave the struct reaches roughly 193 (the 208 class) — another ~112 bytes per pokemon. It has to be a build tag rather than a config flag, because Go fixes struct offsets at compile time and a runtime branch keeps the struct at the larger size, saving nothing. The cost is a build matrix of 4+ combinations and converting ~250 direct field reads to accessors in both layouts, so the readable build stops being readable. Worth its own PR against this measured baseline.

  • Pvp and GolbatInternal were assessed and left alone. Two measured negative results, both now pinned in entity_sizes_test.go: Pvp is not dead between its write and its zeroing (the queue snapshot, the direct-write fallback and the webhook builder all read it), and narrowing it alone buys exactly zero allocated bytes, because 296 still lands in the 320 class and the next class down is 288.

  • The intern table wants an alert threshold, not just a dashboard line. The length cap bounds entry size, not table count; a caller sending a million distinct well-formed usernames still grows it. golbat_intern_table_size is the instrument.

  • A clamped value now lands exactly on PokemonLookup's missing-value sentinelint8(255) = -1, and int16(65535) = -1 for Cp and Form. That aliases "clamped" with "unknown" in scan filters, which the pre-narrowing int8(300) = 44 did not. Narrow, latent, introduced here.

  • Off-heap/arena storage for PokemonData deserves another look. It was ruled out earlier using ~5% for GC's share of CPU; at a measured 24.22% that arithmetic does not hold.

  • Pokestop (1152 B) and Gym (968 B) are both far over the 512-byte threshold.

Design: docs/superpowers/specs/2026-08-16-pokemon-struct-packing-design.md

🤖 Generated with Claude Code

TurtIeSocks and others added 19 commits August 16, 2026 12:03
Design for shrinking the cached Pokemon entity below Go's 512-byte
allocation threshold, where GC mark cost jumps 3.1x (measured: 512 B with
one pointer marks in 62.6 ms, 520 B in 194.4 ms, across 5M live entries).

Chooses narrow null wrapper types (NullUint8 et al, each implementing
sql.Scanner and driver.Valuer) over a single validity bitmask. The bitmask
is 56 bytes smaller but requires a parallel DB-shaped struct at all six
sqlx call sites, and fails at runtime rather than compile time when a NULL
column meets a bare uint8.

PokemonData 592 -> 224 bytes; Pokemon 800 -> ~400, which crosses from Go's
896 size class to the 416 class for roughly 480 bytes saved per cached
pokemon.

Also drops four fields that cost memory without earning it: Iv (the column
is GENERATED ALWAYS AS ... VIRTUAL), Capture1/2/3 (no callers, absent from
both the select columns and the upsert), changedFields (const-folded dead
in production builds), and narrows SeenType from a 24-byte string header to
a 2-byte enum.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The spec claimed `iv` is a GENERATED ALWAYS AS ... VIRTUAL column and that
pokemonBatchUpsertQuery writing `:iv` against it was a pre-existing bug to
resolve. Both are wrong.

The schema comment at decoder/pokemon.go:113 is stale.
sql/11_ivchanges.up.sql drops the generated column and adds a plain
nullable float(5,2) in its place, so the column is real and writable and
the upsert is correct as written.

Iv could still be dropped and recomputed from the three IV fields, but that
means changing the upsert, the select columns, and four call sites — one of
them on the public API response path — to save 8 bytes against ~100 bytes
of headroom. Narrowed to NullFloat32 and left alone instead.

PokemonData 224 -> 232 bytes, Pokemon ~402 -> ~410, both still well under
the 512-byte threshold.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
sql/7_add_height_size.up.sql renames the original size double(18,14) to
height and adds a new size tinyint unsigned, so NullUint8 for Size is a
direct column match rather than a judgement call about observed range.

That is the third stale claim found in the schema comment at
decoder/pokemon.go:88-140, after the iv generated-column and the
four-value seen_type enum. Added a section telling implementers to verify
column types against the migrations rather than the comment.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Eight tasks from the approved design. Task 0 is the pprof measurement that
decides whether the work is worth doing at all; tasks 1-6 are the
implementation; task 7 records the measured outcome against the estimate.

Each task ends with an independently testable deliverable. The size
assertion test lands first and its expected values are updated by every
subsequent task, so each task's memory effect is explicit in its own diff.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
unsafe.Sizeof works directly in a table entry; the wrapper functions were
indirection a reviewer would rightly flag.
Instruments struct size assertions that later tasks will update as the
Pokemon type is packed. Pins current reality: PokemonData at 592 bytes,
Pokemon at 800 bytes (above the 512-byte GC threshold). Documents the
gcSizeThreshold constant explaining why staying under 512 bytes matters.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Task 2's review found NullFloat32.MarshalJSON formats at bitSize 32 where
guregu/null.Float formats at 64, so weight emits 6.7 rather than
6.699999809265137. Both documents asserted byte-identical JSON, which is
now knowingly false for that one type.

Ruled: keep bitSize 32. The values are protobuf float fields promoted
through float64() at decoder/pokemon_decode.go:749,751, so the extra digits
were promotion noise. The divergence gets its own test so it stops being
accidental.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ack to NullUint64.Scan

- NullFloat32.MarshalJSON now returns an error for NaN/Inf instead of
  emitting invalid JSON tokens. This prevents client-supplied malformed
  proto floats from corrupting API payloads.
- Updated NullFloat32 doc comment to explicitly document the bitSize 32
  divergence from guregu/null (bitSize 64) and explain why it's correct.
- NullUint64.Scan now has a ParseUint fallback for []byte/string inputs
  to handle unsigned values above MaxInt64 (unlikely but future-proof).
- Updated NullUint64 doc comment to clarify the actual schema precondition
  (signed bigint columns) and that unsigned values are tolerated.
- Extended TestUint64FullRange to test the []byte path.
- Added TestFloat32JSONIsNarrower to document the deliberate JSON divergence.
- Added TestFloat32JSONNaNInf to verify error handling for special float values.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PokemonData 592 -> 280 bytes (the design doc estimated 232; the
"pointer-carrying, last" field group's own 8-byte alignment requirement
reintroduces 7 bytes of padding a fully byte-count-monotonic order would
have avoided — see entity_sizes_test.go and pokemon.go's struct doc
comment for the measured breakdown). Pokemon 800 -> 456 bytes; still
carries changedFields and internal, both left for a later task.

Nullable numerics move from guregu/null's 16-byte wrappers to nulltypes
equivalents sized to the actual columns, and fields are reordered by
descending alignment.

Setters keep their null.X signatures and clamp out-of-range values to the
column boundary, counted by golbat_field_clamped_total. Direct field
assignments in pokemon_decode.go and weather_iv.go move to setter calls
(or, for AtkIv/DefIv/StaIv, direct clamp+assign inside calculateIv, which
remains their sole mutator by design) so they cannot skip the clamp.

Drops Capture1/2/3, which had setters but no callers and appeared in
neither pokemonSelectColumns nor pokemonBatchUpsertQuery. The webhook
payload keeps its capture_* fields at 0, which is what consumers have
always received.

The API response's float fields (weight/height/iv) widen through a
shortest-round-trip string rather than a naive float64 cast, so widening
a narrowed float32 back to *float64 doesn't expose float32 rounding
noise at float64 precision (float64(float32(3.14)) is 3.140000104904175,
not 3.14) - this keeps the golden JSON test's wire format byte-identical.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Task 3 measured PokemonData at 280 bytes, not the 232 the plan estimated.
The estimate was arithmetically impossible: the declared fields carry 273
bytes of payload and the struct aligns to 8, so every ordering rounds to
280 with 7 bytes of mandatory trailing padding.

Field order remains load-bearing — a careless reordering still adds
padding — but 280 is the floor for this field set. Pokemon came out at 456,
inside Go's 480-byte size class and under the 512 threshold.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Same correction as the spec: 273 bytes of payload align to 280, so the
plan's Step 8 constant and commit-message template were both wrong.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review fixes for the PokemonData narrowing commit:

- The struct's doc comment and TestPokemonEntitySizes's wantPokemonData
  comment both claimed a fully-monotonic field order would have avoided
  the 7 bytes of padding before the pointer group. That's arithmetically
  false: the field payload sums to 273 bytes, Go's 8-byte struct
  alignment rounds any ordering of this field set up to 280, and the
  current order already hits that minimum - the 7 bytes just move
  elsewhere (e.g. trailing padding) under any other ordering, they don't
  disappear. Corrected in both places; the design doc's 232/264 estimates
  were arithmetic errors, not a target this order missed.

- The size-class remark ("[257, 288] size class") was attached to
  wantPokemonData, but PokemonData is never independently heap-allocated
  (embedded in Pokemon, copied by value into []PokemonData batches), so
  no size class applies to it. Moved to wantPokemon's comment, where 456
  landing in the 480-byte class under the 512 threshold is the actually
  load-bearing fact.

- Added decoder/pokemon_clamp_test.go: IncFieldClamped and the four clamp
  helpers had no coverage. A small counting fake wrapping the noop
  collector (mirroring the swap/restore pattern already used in
  station_battle_test.go) pins that out-of-range Set* calls land on the
  column boundary and count exactly once, in-range values count zero,
  and clampFloat32 never clamps (weight/height/iv are documented as
  always in range).

- Documented why SetHeight/SetWeight compare the clamped float32 values
  exactly instead of with the old null.Float tolerance: both sides are
  already narrowed to float32 by the time they're compared, so the
  float64-promotion jitter the tolerance existed for is collapsed before
  the comparison runs.

Verified with -race (beyond the task's specified test command) that the
counting-fake swap pattern doesn't introduce a new race: the one race
`go test -race ./decoder/` reports (station_battle_test.go vs the stats
aggregation worker) reproduces identically with pokemon_clamp_test.go
removed from the package, confirming it predates this change. Flagged
separately for a dedicated fix rather than addressed here.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review round 2 fixes:

- The clamp-path test added in the previous fix propagated the exact
  race decoder/init_test.go's comment warned about: reassigning the
  package-level statsCollector from a test races the background stats
  aggregation worker and StartWorkerBacklogReporter's ticker, both of
  which read it continuously for the life of the test binary. Fixed at
  the root instead of narrowing the test: statsCollector is now
  atomic.Pointer[stats_collector.StatsCollector], read through a new
  getStatsCollector() accessor. Converted all ~80 call sites (18
  non-test files, mechanical) and all four test-side swaps (the new
  pokemon_clamp_test.go plus three pre-existing ones in
  station_battle_test.go) to a shared setStatsCollectorForTest helper.
  `go test -race ./decoder/` is now clean across repeated runs, including
  the specific test that used to fail
  (TestCreateStationWebhooksEmitsFutureBattle) - this also resolves the
  pre-existing race filed separately as a background task, which is now
  withdrawn.

- The previous fix round corrected the padding-arithmetic mistake in
  pokemon.go and entity_sizes_test.go but left two copies of the same
  false claim standing in the task report itself (the "Measured sizes"
  narrative and a Concerns bullet), plus a self-audit line claiming they
  were already fixed. Edited both in place to match the corrected
  arithmetic (273-byte payload rounds to 280 under 8-byte alignment
  under any field ordering; this order already hits that floor).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Covers the failure mode the narrow-wrapper approach was chosen to avoid: a
NULL column landing in a narrowed field. Also pins the two precision cases
that must not regress, the full 64-bit cell_id and float64 lat/lon.

Skips when GOLBAT_TEST_DSN is unset, since CI defines no database service.

Two bugs surfaced and fixed while proving this against a live database, both
in the test's own setup rather than in nulltypes:
- shiny has a non-NULL schema DEFAULT ('0'), so omitting it from the insert
  (as done for every other nullable column) inserted 0/Valid instead of NULL,
  defeating that one check. Now bound explicitly to NULL.
- defer db.Close() ran before t.Cleanup's delete (cleanups fire after a
  function's own defers), so the sentinel-row delete silently failed on
  every run. Close is now registered via t.Cleanup, ordered to run last.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
seen_type is an eight-value enum held in a 24-byte null.String with a heap
pointer. NullSeenType stores the code and converts at the database and JSON
boundaries, so both wire formats are unchanged.

An unrecognised enum value is now an error rather than a silently stored
string, since the alternative is corrupting scan statistics when the game
adds a seen type ahead of the migrations.

PokemonData 280 -> 256 bytes, Pokemon 456 -> 416 bytes (measured, not the
design doc's estimate).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 24-byte []string slice header was unconditionally present on every
cached Pokemon even though dbDebugEnabled is a build-tag const, so every
append into it was already const-folded away in production. Replaced it
with `debug pokemonDebugState`, a field whose type is defined once per
build tag (mirroring the existing dbDebugEnabled/dbDebugLog split):
db_debug.go gives it a real []string accumulator under -tags dbdebug,
db_debug_off.go gives it a zero-sized stub otherwise. The debug build's
save-time log output is unchanged — dbDebugLog still gets one aggregated
"changed=[...]" line per save, not per-field logging.

Measured: unsafe.Sizeof(Pokemon{}) goes from 416 to 392 in production.
This does NOT change the allocator's size class: 392 still rounds up to
416, confirmed empirically via runtime.MemStats.TotalAlloc across 200k
live *Pokemon (416.0 bytes/object, unchanged). The win is a dead field
gone from the layout and one fewer pointer word in the GC scan bitmap,
not a reduction in allocated memory - the entity_sizes_test.go comments
spell this out so nobody "fixes" the number later assuming it moved.

Adds TestPokemonUnderGCThreshold, the acceptance test for the whole
packing effort: Pokemon must stay under 512 bytes, above which measured
GC mark cost jumps 3.1x. Also removes the now-stale nolint:unused off
gcSizeThreshold, which this test is the first consumer of.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Added Results section to the design doc capturing what was actually
measured (256 B PokemonData, 392 B Pokemon in production) and
clarifying where the design's predictions were wrong:

- The 232 B estimate for PokemonData was arithmetically impossible
  (minimum achievable 280 B from field payload + alignment).
- The Task 6 field deletion produced no allocator savings (both
  416 B and 392 B land in the same size class).

The measurable outcome: allocator gives back 480 bytes per cached
pokemon (896 class -> 416 class), totaling 2.4 GB at 5M entities,
matching the design's magnitude estimate. CPU impact remains
unverified (required production profile never captured).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five cleanups from the whole-branch review (verdict: ready to merge, no
Critical/Important findings):

- Rewrite PokemonData's load-bearing layout comment, which described the
  pre-task-5 280-byte/273-payload layout. Measured the real numbers via
  unsafe.Offsetof/Sizeof: 256 bytes, 251-byte payload, 25-byte 1-byte
  group, 96-byte pointer group, 5 bytes of padding at offset 155->160.
  Also fixes the comment's stale pointer to the schema comment further
  down the file, which now documents known divergences rather than
  containing stale claims.
- Drop the dangling "REMINDER! Keep hasChangesPokemon updated" comment;
  that function no longer exists anywhere in the codebase.
- NullSeenType.Value() now errors instead of returning "" when Code is
  past the end of seenTypeStrings, closing the path by which a bad code
  could silently write an empty string into the seen_type ENUM column.
  Unreachable today (every construction validates), but Code and
  SeenTypeFrom are exported, so the invariant was held by discipline
  alone.
- TestPokemonFullRowRoundTrip now asserts Weight and adds/asserts Iv,
  pinning NullFloat32's float64-widening Value() against both column
  precisions it writes into (weight is double(18,14), iv is float(5,2)).
  Unverified by execution — GOLBAT_TEST_DSN was intentionally left unset.
- Fix a test comment in nulltypes_test.go that cited the wrong hex value
  for its decimal literal (0xc556d6894e95b000, not 0xC5A0...); the
  decimal literal and assertion were already correct.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jfberry

jfberry commented Aug 16, 2026

Copy link
Copy Markdown
Collaborator

Absolutely will need to be benchmarked; and against go 1.27 which changes allocation to be more efficient anyway.
But the memory saving alone may make this worthwhile

@jfberry

jfberry commented Aug 16, 2026

Copy link
Copy Markdown
Collaborator

Incidentally, since the null package already supports generics you may not need the newly defined nulltypes but rather just right-size in the current package

The embedded protobuf is absolutely ripe for change, it's a smell that it's in there in protobuf form - I did once replace it with a conventional structure and move to protobuf on demand (almost nowhere --> it is only marshalled when writing to disk, and the default is that it isn't).

The pokemonChanged value could be renamed to a generic name to encapsulate the change list, and rolled out to the other objects for a near free improvement.

@jfberry

jfberry commented Aug 16, 2026

Copy link
Copy Markdown
Collaborator

Finally, pull the SeenTypeCodeWild = 0 fix into this PR - and the re-widening of the webhook to float64 and reconversion seem like they are wasted steps for some misattributed compatibility requirement

TurtIeSocks and others added 7 commits August 16, 2026 17:44
The maintainer (jfberry) pointed out that github.com/guregu/null/v6
already ships a generic null.Value[T] (embedding stdlib sql.Null[T])
that does the same job as the hand-rolled decoder/nulltypes package:
identical sizes for every narrowed type, Scan range-rejection out of
the box, and float32 JSON marshalling at native 32-bit precision with
no custom MarshalJSON needed. Confirmed all of this empirically before
touching any code.

Deletes decoder/nulltypes entirely (702 lines) and converts every
field, clamp helper, and constructor call across 11 files to
null.Value[T]. PokemonData/Pokemon still measure 256/392 bytes
(TestPokemonEntitySizes, unchanged) and the golden API JSON string in
api_pokemon_response_test.go is untouched.

CellId moves from a NullUint64 storing a bit-reinterpreted S2 cell id
to null.Value[int64], matching cell_id's actual signed-bigint column
type. null.Value[uint64] can't Scan a negative int64 (it round-trips
the driver value through a string and ParseUint rejects the leading
"-"), which is exactly the case cell_id needs since real S2 cell ids
are frequently negative. Giving CellId the correct signed type turns
two former workarounds into straight-line code: SetCellId no longer
needs its uint64(v.Int64) bit-reinterpret cast, and the API response
builder's widenPtr[uint64, int64] call collapses to a plain .Ptr().
One read site (pokemon_state.go's MatchStatsGeofenceWithCell call)
picked up an explicit uint64(...) cast instead, matching the pattern
gym/pokestop/station already use for their own CellId fields.

SpawnId stays null.Value[uint64]: go-sql-driver/mysql returns a native
uint64 for the unsigned bigint column, and pokemon_state.go's existing
strconv.FormatUint(pokemon.SpawnId.ValueOrZero(), 16) call requires a
uint64 argument, so it was both the type the driver already produces
and the type every downstream reader wants.

Verified via go build/test under both go_json and go_json dbdebug,
-race, golangci-lint, and TestPokemonNullColumnRoundTrip /
TestPokemonFullRowRoundTrip against a live MariaDB dev instance (the
latter now also exercises a negative CellId, -16, in place of the old
hex bit-pattern fixture that no longer type-checks against int64).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review finding: sql.Null[uint64].Value() delegates to
driver.DefaultParameterConverter.ConvertValue, which explicitly refuses
any value with the high bit set ("uint64 values with high bit set are
not supported"). The nulltypes.NullUint64.Value() this replaced did a
bit-reinterpret (return int64(n.V), nil) that always succeeded, so the
type swap silently dropped a real capability: null.Value[uint64] can
Scan the full unsigned range of spawn_id (bigint unsigned) but cannot
always Value() it back. Unreachable today only because both
SetSpawnId call sites route through strconv.ParseInt(s, 16, 64), which
can't produce a sign-bit value — an upstream accident, not a proof of
equivalence.

Switches SpawnId to null.Value[int64], matching CellId's existing
type and reasoning: int64 has no such restriction, and real spawn ids
are ~48-bit, far below where either type's range would matter.
SetSpawnId now stores v.Int64 directly with no cast, same shape as
SetCellId. Downstream: pokemon_state.go's
strconv.FormatUint(..., 16) call picked up an explicit uint64(...)
cast (matching the pattern already used for CellId at pokemon_state.go
:282), and three call sites building *int64 API fields
(api_pokemon_response.go, api_pokemon_scan_v2.go, api_pokemon_scan_v3.go)
simplified from widenPtr[uint64, int64](...) to a plain .Ptr(), since
SpawnId.Ptr() now already returns *int64.

Also corrects a gap the previous commit's report should have caught:
TestPokemonFullRowRoundTrip never actually set or asserted SpawnId
against the live database - the value it cited as proof
(SpawnId: null.ValueFrom(uint64(7777))) lives in the unrelated
JSON-only golden test in api_pokemon_response_test.go, which never
touches the database. Added a real SpawnId round-trip assertion here
so the type this commit picks is actually exercised against MariaDB,
not just proven to compile.

Sizes unchanged (PokemonData=256, Pokemon=392): int64 and uint64 are
both 8-byte payloads, so null.Value[int64] and null.Value[uint64] are
both 16 bytes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
SeenTypeCodeWild = iota made 0 - the Go zero value - the enum's most
common real value. Any code reading .Code without first checking
.Valid would silently treat a NULL seen_type as wild; one such site
existed in the write-behind delay logic and was caught during review.

Insert SeenTypeCodeUnset at 0 and shift the eight real codes up by
one. seenTypeStrings gets a leading "" entry so String() reports ""
for both Unset and any out-of-range code via the same lookup;
seenTypeCodes excludes that entry so ParseSeenType can never produce
Unset from a real string. Value() now rejects any code whose String()
is empty, which additionally closes off Unset - writing "" to the
seen_type ENUM column is accepted silently by MariaDB.

Searched every use of a SeenTypeCode value: all go through named
constants, String()/Value()/MarshalJSON(), or ParseSeenType - never a
raw code number persisted, transmitted, or compared - so the shift is
safe. Sizes (256/392) and the API golden JSON are unaffected, as
expected: this changes which integer represents each seen type, not
any string or column width.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
nullFloatFromFloat32 widened the stored null.Value[float32] Weight/
Height back to guregu/null.Float at the webhook payload boundary, to
keep webhook bytes identical to before the struct-packing PR. The
maintainer judged that requirement misattributed: webhook consumers
are this project's call, not a wire contract owed byte-identity.
PokemonWebhook's Weight/Height now hold null.Value[float32] directly
and nullFloatFromFloat32 is deleted, its only caller gone.

This changes webhook output: weight and height now emit 6.7 where
they previously emitted 6.699999809265137 - the short form the API
endpoint already emits, for the same reason - both originate as
protobuf float fields, and the extra digits were an artifact of a
float64() promotion that never carried information.

Narrowed the integer fields (Gender, Cp, Form, Costume, the three
IVs, PokemonLevel, Move1/2, Size, Weather, DisplayPokemonId/Form,
LastModifiedTime) the same way, deciding to treat this as one
consistent policy rather than a case-by-case call: the widening
existed for one reason (byte-identical webhooks) that no longer
applies to any field, narrowing an int is a JSON no-op (guregu/null's
generic Value[T] marshals via encoding/json, which renders 15 as "15"
regardless of int width - verified against the package source), and a
struct where the float fields mirror storage width but the int fields
still widen for no live reason is an inconsistency with no upside.
nullIntFromUint survives for its two other callers (a setter-feeding
reverse conversion in pokemon_decode.go, and the stats collector's
UpdateVerifiedTtl, which is guregu/null throughout).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pokemon embedded `internal grpc.PokemonInternal` purely to hold
Ditto-detection scan history in memory. That carried 64 bytes of protobuf
machinery (MessageState, sizeCache, unknownFields, slice header) on every
cached pokemon, plus 48 bytes of the same overhead on every history entry,
for a message that is only ever marshalled when writing the golbat_internal
column — which is gated behind pokemon_internal_to_db, off by default.

The field becomes `scanHistory []*pokemonScan`: a plain 44-byte struct with
the same twelve int32/bool fields as the proto. Elements stay behind
pointers because the Ditto code takes a pointer to a history entry, holds it
across other work and mutates through it; a value slice would have `&s[i]`
invalidated by any reallocating append.

The protobuf remains the wire format and is untouched. Conversion happens at
exactly two boundaries: populateInternal unmarshals the stored bytes into a
temporary grpc.PokemonInternal, and savePokemonRecordAsAtTime builds one to
marshal. Bytes written before this change decode identically, and bytes
written after it are byte-identical to what the old code emitted — both
pinned by a golden payload in decoder/pokemon_scan_test.go.

Everything else is a type swap: locateScan, locateAllScans, checkScans,
setDittoAttributes, resetDittoAttributes, confirmDitto, detectDitto,
addEncounterPokemon and recomputeCpIfNeeded keep their logic verbatim, and
the five helper methods move from grpc/helper.go (hand-written, now dead) to
the new type unchanged. pokemonScan grows a String() so the Ditto debug logs
and error messages keep the proto-text content they had, including "<nil>".

Pokemon: 392 -> 352 bytes, and unlike the previous task this one moves the
allocator. 352 is itself a Go size class, so the bytes handed out per cached
pokemon go 416 -> 352 (measured, n=200000: 352.0 exactly, against 416.0 for
a 392-byte control). History entries go 96 -> 48. PokemonData is untouched
at 256.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…dary

TestPokemonEntitySizes' doc comment still described the pre-task-7 struct:
392 bytes, mid-class, "24 bytes of free space ... so a modest field addition
is free in allocator terms". At 352 that is exactly inverted. 352 is itself a
Go size class, so the next byte added — a single bool — costs 32 per cached
pokemon, roughly 160 MB at 5M, about half of what removing the embedded
protobuf just won. Someone hitting the failed assertion would have read the
old comment, concluded the field was free, and bumped the constant.

Also state in pokemon_scan.go that a field added to grpc.PokemonScan must be
added to pokemonScan and to BOTH converters, and back that with a test rather
than a comment: TestPokemonScanCoversEveryProtoField compares the exported
field sets of the two structs and cross-checks the count against the wire
descriptor. The golden-bytes fixture cannot catch this — a fixture only knows
about the fields it was built with, so a thirteenth proto field would decode
into the temporary at the read boundary and never be written back.

Verified the new test fires: an extra field on pokemonScan fails it by name,
and a missing one fails the build.

The rest is small. scanHistoryFromProto uses the nil-safe getters on repeated
elements. The round-trip test indexes its fixture slice instead of taking the
address of a range variable, which is correct under Go 1.22+ loop semantics
but reads like the classic aliasing bug. The design doc's results sections
said 392/416 and "120 bytes of headroom" as though they were still current.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Renames Pokemon's pokemonDebugState to the entity-neutral
debugChangeAccumulator and rolls it out to the eight other entities that
still carried a raw `changedFields []string` (Pokestop, Gym, Station,
Spawnpoint, Incident, Tappable, Route, Player), replacing an
unconditional 24-byte slice header with a build-tag-split type that
costs zero bytes in production builds.

Station, Spawnpoint, Incident, and Tappable cross a Go allocator size
class boundary from this (Spawnpoint 136->104, dropping from the
144-byte class to 112 — the one flagged as mattering most, since it's
cached in the millions). Pokestop, Gym, Route, and Player shrink the
same 24 bytes but land in the same allocator class as before, so they
save GC scan-bitmap pressure but nothing at allocation time; measured
both empirically (TotalAlloc delta / n) rather than assumed from the
static size-class table, which undercounts for pointer-containing
("scan") structs at this size range on this Go toolchain.

Tappable and Player had `changedFields` as their last struct field, so
renaming in place would have hit the zero-sized-last-field padding trap
(dodged the same way Pokemon's original conversion did); both got the
field reordered ahead of `newRecord` with a comment explaining why.

Adds TestClassMovedEntitySizes pinning the four entities whose class
actually moved, skipping the four that didn't per the same reasoning
TestPokemonEntitySizes already established.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@TurtIeSocks TurtIeSocks changed the title perf: pack PokemonData below Go's 512-byte GC threshold perf: shrink cached Pokemon 800→352 bytes (field packing, plain-struct scan history, entity rollout) Aug 16, 2026
TurtIeSocks and others added 10 commits August 17, 2026 13:55
The comments explaining jsonenc said -tags go_json was what makes
huma_api.go select goccy for API responses ("the codec huma_api.go
uses to serve every API response" — under a build-tag heading). That
gets the causality backwards: huma_api.go's newHumaConfig imports
goccy directly with no build constraint, so every huma-registered
route — everything the golden tests here pin — is marshaled through
goccy unconditionally, tag or no tag. What -tags go_json actually
gates is gin's own internal JSON codec (github.com/gin-gonic/gin/
codec/json), used by the raw c.JSON() calls outside huma (routes.go's
PokemonScan and GetHealth).

No behavioral gap results — the Dockerfile and Makefile both default
to building with the tag, so huma's unconditional goccy and gin's
tag-selected goccy never disagree in a real build — but the comments
described a mechanism that doesn't exist. Rewrote jsonenc.go's package
doc as the one authoritative explanation of what the tag does and
doesn't gate, and pointed every test file's shorter comment at it
instead of repeating (and this time getting wrong) the same claim six
times. jsonenc's tag-gated design is unchanged and still correct: it
makes each test track whichever codec the build it's compiled into
actually selected, which is the real reason for it.

Comment-only change, confirmed via diff — no marshal calls, no golden
strings, no test logic touched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review round 4, tail items 1-2 (combined: both edit the same clamp/narrow
doc-comment block in decoder/pokemon.go, and the shared `saturate` comment
references facts from each — splitting them would leave one commit's
comment referring to an identifier the other hadn't introduced yet).

narrowUint32 had no production caller: unlike narrowUint8/16, which compare
raw proto values against stored Form/Weather/Costume/Gender columns
throughout pokemon_decode.go and weather_iv.go, nothing does the same for
either uint32 column (expire_timestamp, updated). Its only callers were its
own test cases. Removed the function and the three cases, and updated the
doc comments that used to enumerate it alongside narrowUint8/16.

clampUint's `limit` parameter was always the caller-supplied ceiling, but
for its three plain instantiations (clampUint8/16/32) that ceiling is just
T's own natural maximum. Split clampUint into a zero-arg-ceiling form that
derives it via int64(^T(0)) — the standard idiom for "largest value this
unsigned type can hold" — and clampUintCeiling, the old explicit-limit body,
kept for clampIv, whose ceiling (15, the game's per-stat IV cap) is
narrower than its storage type's and genuinely can't be derived from T.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… removal

Review round 4, tail item 3.

decoder/stats.go's comment on the removed sc == nil check read "no guard
needed before this used to be a real early-startup window" — two facts
welded into one ungrammatical sentence. git log -p -L traced it to commit
7972303 (round 2, task 4), which seeded statsCollector with a noop at
package init and removed the guard that used to be necessary before that
seeding existed. Reworded as two separate sentences saying so.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… narrowing

Review round 4, tail item 4.

An earlier round narrowed PokemonWebhook's numeric/bool fields from guregu's
null.Int/null.Bool/null.Float to the storage-width-matched null.Value[uint8]
/null.Value[uint16]/null.Value[uint32]/null.Value[bool]/null.Value[float32],
but webhooks.md's payload table was never updated to match. shiny was the
field the maintainer named, but checking every row against the actual struct
(decoder/pokemon_state.go's PokemonWebhook) found the same drift on all 18
narrowed rows, not just that one. Every other webhook payload type
(gym/raid/pokestop/quest/incident/station/...) still uses the classic guregu
types, so this drift was isolated to the pokemon table but wide within it.

Fixed all 18 rows and added null.Value[T] to the "Nullable fields"
conventions section so the table's types trace back to a defined
serialization convention. seen_type's row (null.String) was left as-is: its
Go type is a distinct custom type (NullSeenType) whose JSON behavior matches
null.String exactly, and it predates this narrowing round.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review round 4, tail item 5.

decoder.statsCollector's own package-level initializer already seeds it
with a noop (a later round of this same PR), which runs before any
init() function per the Go spec — including in test binaries. Confirmed
this call was genuinely redundant rather than assuming so from the name:
checked its one non-obvious side effect (marking statsCollectorSet, which
gates InitWriteBehindQueue's boot-ordering panic) and grepped every
decoder _test.go for InitWriteBehindQueue — zero calls, so nothing in the
test binary depends on that flag either.

Removed the call. As a direct consequence, fixed
stats_collector_init_test.go's TestStatsCollectorSeedIsInTheVariableInitializer
comment, which explicitly said the package variable "has already [been]
overwritten by the time any test runs" by this exact call — no longer
true, so reworded to explain why the test reads newSeededStatsCollector()
directly regardless (it's pinning the initializer, not working around
stale shared state).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review round 4, extra item 6 (flagged by a previous task's report but
left in scope since it was outside that task's ten items).

Confirmed dead before removing: grepped every .Xxs/.Xxl read or write in
decoder/ (zero hits beyond the struct declarations), and read
updatePokemonLookup's struct literal directly to confirm it never
populates them. The XXS/XXL size filters (IncludeXxs/IncludeXxl in
api_pokemon_scan_v1.go) read pokemonLookup.Size instead, exactly as
pokemon_lookup_narrow_test.go's own pre-existing comment already
documented ("pinned as-is rather than removed here because removing them
is a separate change" — this is that change).

PokemonLookup is loaded 15-20M times/sec in production profiles (14% of
CPU), so its width is a direct multiplier on scan cost. Removing the two
dead bools drops unsafe.Sizeof(PokemonLookup{}) from 18 to 16 bytes, and
PokemonLookupCacheItem (which embeds it) from 26 to 24 — updated both
pins in TestPokemonLookupSizes. Also updated cachebench's lkPokemon,
which claims to mirror PokemonLookup exactly and had the same two unused
fields.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review round 4, extra item 7 (reported as having no reachable window
today; judged here as worth guarding anyway).

main.go's package-level statsCollector was a bare nil interface until
main() assigned the real collector partway through boot. decode.go
(~40 call sites), routes.go (5), and grpc_server_raw.go (2) all call
methods on it with no nil check; only raw_limiter.go defensively guarded
its own two call sites. Traced main()'s boot order: the HTTP routes and
the gRPC listener both start well after the assignment, and
initRawProcessingLimiter (the one already-guarded goroutine) is also
called after it, so nothing reaches any of those ~47 call sites before
the real collector lands. No reachable window exists today.

Guarded it anyway rather than leaving it: decoder and db already carry
this identical fix from earlier rounds of this same PR, for the same
"no window today, but the guarantee is cheaper to hold than to
re-derive per call site forever" reasoning documented in db/dbDetails.go's
own comment for its equivalent seed. Leaving main as the third unfixed
instance means the next caller added to any of those ~47 sites, or any
future reordering of initRawProcessingLimiter, silently reintroduces a
nil-panic that's already been hit and fixed twice in sibling packages.

Seeded with stats_collector.NewNoopStatsCollector() at declaration,
matching decoder's and db's pattern, and removed the two now-dead
`if statsCollector != nil` guards in raw_limiter.go.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
db/dbDetails.go claimed "no test swaps it" as the reason a plain variable
is safe there, and the test added in the same commit swaps it —
TestSetStatsCollectorRefusesNil restores it through a t.Cleanup. Harmless
in practice, since that package's tests are sequential, but it is
load-bearing guidance that is false as written. The comment now states
the actual invariant: nothing writes it while anything reads it
concurrently, which is what decoder's atomic.Pointer exists for and this
does not need.

InitWriteBehindQueue's new ordering guard is a crash mode that did not
exist before, and golbat's own main() cannot reach it. A fork with its
own main() ordered the other way round can, so the doc comment now says
so and names the fix — swap the two calls, do not delete the check.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Widening ExpireTimestamp and Updated to int64 dropped the `minimum: 0`
huma had been inferring from uint32. That was a side effect of the
widening, not part of the point: the bound is real — all four timestamps
are backed by unsigned columns — and the goal was to make the four agree,
which they can do on the bound as well as the type.

An explicit `minimum:"0"` tag on all four restores it, and gets
first_seen_timestamp and changed a floor they never advertised at all,
since they were already int64 and huma had nothing to infer from.

Schema effect: expire_timestamp and updated keep minimum: 0 across the
widening rather than losing it, and first_seen_timestamp and changed gain
it. TestApiPokemonResultSchemaWidths now requires it on all four.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
It logged once per refused call. Unreachable today — all 13 call sites
pass a compile-time SeenTypeCode* constant — but the branch sits on the
decode path, and the thing that would make it reachable, a computed code
from some future caller, would make it reachable once per sighting. That
is the shape util.DropReporter exists for, and it is what
NullSeenType.Scan's mirror-image warning on the read side already uses.

Aggregated to one line a second through a reporter of its own, with the
refused count in the message. The test now pins the throttle: four
refusals, one line.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@TurtIeSocks TurtIeSocks changed the title perf: shrink cached Pokemon 800→312 bytes (field packing, plain-struct scan history, string interning) perf: shrink cached Pokemon 800→352 bytes (field packing, plain-struct scan history) Aug 17, 2026
@Mygod

Mygod commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

The protobuf-to-plain-struct conversion introduces forward-compatibility data loss when internal data is rewritten by an older binary. Otherwise, the targeted production and dbdebug decoder tests passed.

Review comment:

  • [P2] Retain protobuf unknown fields during scan conversion — decoder/pokemon_scan.go:152-159
    When pokemon_internal_to_db is enabled during a rolling upgrade or rollback, rows may contain fields unknown to this binary. proto.Unmarshal previously retained those bytes in pokemon.internal, but rebuilding fresh PokemonInternal and PokemonScan messages here copies only known fields, so the next encounter overwrites golbat_internal without the newer data. Preserve the unknown bytes or message state when rebuilding.

Arguably doesn't need fixing?

The pre-pokemonScan write boundary marshaled the very grpc.PokemonInternal
it had unmarshaled, so protobuf fields this build has no definition for rode
along in unknownFields for free. pokemonScan is a plain struct with nowhere
to put them, so rebuilding from it copies known fields only — during a
rolling upgrade or after a rollback, the next encounter would quietly
replace a newer node's row with a subset of itself.

Storing the raw bytes per pokemon would fix it and cost 24 bytes on Pokemon,
pushing it from 352 into the 384 size class — roughly 104 MB at the 3.25M
cached in production, to protect a path that is off by default. Merging
unknown fields per element is not possible either: scan-history entries have
no stable identity to match on.

So refuse instead. rewriteGolbatInternal unmarshals the stored bytes and
checks both levels — PokemonInternal itself and every scan_history element,
the latter being the historically likely one — and when either carries
unknown fields it leaves the row exactly as it found it. Zero memory, runs
only with pokemon_internal_to_db enabled, and turns silent loss into
deliberate non-overwrite. Same call SetSeenType makes when handed a code it
cannot render.

The refusal skips RemoveDittoAuxInfo too: that trimming exists to keep the
stored column small, and there is no column write here to keep small.

Undecodable bytes are not unknown fields — populateInternal has already
dropped the history for them — so they stay rewritable rather than stranding
the row forever.

Warnings aggregate through util.DropReporter (every encounter save on an
affected row takes the branch, so the unaggregated form would be a line per
encounter), and golbat_pokemon_internal_rewrite_skipped_total lets an
operator see it happening instead of guessing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jfberry

jfberry commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator

This (protobuf defence) change is an overreach. This is a transitory field, and there is no harm in losing fidelity when the default isn't even to store it.

@jfberry

jfberry commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator

Round-6 verification is done and it's clean: both blockers are genuinely fixed and were verified empirically — the CI job is green with the MariaDB round-trip tests actually executing (not skipped) under both production tag sets, calculateIv now computes comparison, stores, and Iv from the same clamped values for every input, and the Unknown sentinel was traced through every non-test reader of SeenType and is inert at all of them, with COALESCE refusing overwrites at both write boundaries. All ten ranked items, the tidy-ups, and the title/body correction landed. Nothing in the new commits is merge-blocking — the undraft stands from our side. A few small items below, plus one thing that needs an author who predates this PR.

@Mygod — your advice wanted on ProactiveIVSwitch (pre-existing, not this PR's doing)

The review's differential pass sat on top of your boosted-weather guard and found something we'd like your read on before anyone fixes it, since you wrote this and know the intended behavior:

if boostedWeathers&uint8(1)<<weatherUpdate.NewWeather != 0 {
    newWeather = weatherUpdate.NewWeather
}

Go gives & and << equal precedence (left-assoc), so this parses as (boostedWeathers & 1) << NewWeather, not boostedWeathers & (1 << NewWeather). Every boostedWeatherLookup entry is even (bit 0 is weather NONE), so boostedWeathers & 1 is always 0 and the guard is unconditionally false — confirmed empirically, and it's on main, not introduced here. Net effect: newWeather is always 0, so the switch can only ever remove boosts, never apply them — and a currently-boosted pokemon whose new weather still boosts it gets wrongly de-boosted via repopulateIv(0, ...). The one-character fix (boostedWeathers&(uint8(1)<<weatherUpdate.NewWeather) != 0) changes live behavior immediately, which is why we'd rather have your confirmation of intent than just land it.

While you're looking: NewWeather arrives as an unvalidated int32 from an open proto3 enum, and a negative value used as a shift count panics — in a goroutine with no recover in its chain, so one malformed packet from any third-party scanner takes the whole process down. Clamping/validating to 0..7 before the shift (or making it uint8 at construction) closes both the crash and half the precedence question at once. Happy to take both in a follow-up PR if you agree with the intended semantics.

Small asks for this PR

1. The weather fast-path still compares two encodings of "absent". The cheap skip tests narrowUint8(newWeather) == pokemonLookup.Weather, where the lookup encodes NULL weather as -1; the entity-level guard after the lock uses int64OrZero(pokemon.Weather), which encodes NULL as 0. So for a pokemon with encounter values but NULL weather, a no-boost update (newWeather == 0) compares 0 == -1, fails the skip, takes the entity lock — and then the guard's 0 != 0 does nothing. One wasted entity lock per NULL-weather pokemon per weather flip in its cell, which is exactly the lookup-vs-entity disagreement the new comment says was closed. Fix: do the skip comparison in the lookup's own encoding (map "no weather" to -1 on the incoming side, or normalize the lookup value to 0 before comparing).

2. NullSeenType.UnmarshalJSON disagrees with Scan about the degraded state. Scan on an unknown string produces Unknown (inert everywhere — good), but UnmarshalJSON(null) produces code 0 = Unset, which is precisely the code this PR proved destructive (it licenses updateFromWild's rewrite and updateFromNearby's coordinate replacement). No production path unmarshals it today, but the type is exported and is the webhook payload's field type, so any replay tool or future path that round-trips a degraded record converts the safe sentinel into the destructive one. Make the two boundaries agree deliberately — one line. (The read-side behavior itself — degraded rows serving seen_type: null during a mixed-deployment window — is fine as-is; the column is COALESCE-protected.)

3. The warn-throttle globals — fix on the test side if possible. seenTypeScanWarns / seenTypeSetWarns are plain package-level pointers that production reads on row-load/decode paths while tests reassign them unsynchronised — the same shape init_test.go's own comment documents as a former data race for statsCollector. It's quiet today only because nothing loads rows concurrently with the swap. Preference: fix it without touching production code — e.g. have the tests inject their own DropReporter through a test-only seam rather than reassigning the global. If there's no clean way to do that, atomic.Pointer[util.DropReporter] is acceptable, but the production code shouldn't grow complexity to serve a test if it can be avoided.

4. Docs. Scan's comment overclaims the self-heal: wild and lure sightings deliberately don't match Unknown (correctly — matching it would reintroduce the downgrade), so the record heals on encounter or on reload only; reword it, because a reader making the documented behavior "real" by adding case SeenTypeCodeUnknown: to updateFromWild would rebuild exactly the bug the sentinel prevents. Also: webhooks.md's type table still lists seen_type as null.String (the one stale row in a table this PR otherwise corrected), and TestNullSeenTypeJSON's comment names huma as the consumer when the webhook sender's stdlib encoding/json is the real one.

Reviewed and accepted

The IV clamp-at-15 behavior (a hypothetical out-of-range reading becoming a stored 15) was considered and is fine: we don't receive fake IV data in practice, and golbat_field_clamped_total is sufficient signal if that ever changes. No action wanted.

Noted, non-blocking

CI no longer builds the untagged (!go_json) configuration anywhere — the one a bare go test ./... uses, including half of jsonenc; one extra untagged step (even go vet ./...) covers it. The same negative-aliasing pattern just fixed in the pokemon lookup is still live in fortRtree.go (bare int8/int16 casts on quest reward amounts / team / slots — the lookupInt8/16 fix applies verbatim). TestUpdatePokemonLookupSaturatesClampedFields leaks a phantom entry into pokemonFormCount (pair the cleanup with adjustPokemonFormCount(-1)). The statsCollectorSet guard doesn't cover exported InitTypedQueues (queues reading the collector at use time would delete the flag and the ordering requirement together). The publish workflow's build-gate comment overclaims (branch pushes still publish per-branch images; the load-bearing case is fork-PR tokens), and tag-push releases now block on MariaDB service health. clampUint's derived ceiling would silently wrap for a future ~uint64 instantiation (a limit < 0 panic guard is cheap). And the two throttle tests assert an exact count of 1 against a real 1-second wall-clock window — a rare-flake on loaded runners; assert >= 1 and < rows or inject the clock.

@Mygod

Mygod commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Damn why is a bot tagging me and roasting my code. Yes, that should be fixed.

TurtIeSocks and others added 7 commits August 18, 2026 19:01
Reverts f0d4560. @jfberry's ruling: the change is an overreach. golbat_internal
is a transitory field, there is no harm in losing fidelity, and the default is
not even to store it.

That is right. I weighed the memory cost of preserving the unknown bytes and
never asked the prior question — whether the data was worth defending at all.
It is scan history in a cache-like column, behind pokemon_internal_to_db, which
defaults off, and the next encounter rebuilds it. A guard, a metric, a
throttled warning and three tests to protect that is cost with no matching
risk.

Removed: the refusal branch, storedInternalHasUnknownFields, the
internalUnknownFieldSkips DropReporter, golbat_pokemon_internal_rewrite_skipped
_total with its interface method, noop and prometheus wiring and registration,
and the three tests that covered them.

Kept: the rewriteGolbatInternal extraction itself, which the guard was bolted
onto rather than caused. savePokemonRecordAsAtTime is a long orchestration
function, and a named method reads better there than fourteen lines of
protobuf marshaling and Ditto trimming inlined mid-block. It also gives the
write boundary a name to grep for and somewhere to document its contract,
next to the conversion helpers it uses, instead of pointing at a region of
another file. The body is byte-for-byte what was inline before, minus the
guard.

Pokemon is still 352 bytes and PokemonData still 256.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ProactiveIVSwitch's cheap skip compared the incoming weather against
PokemonLookup.Weather, which spells "no weather" as lookupInt8's -1, while
the entity-level guard right after the lock reads the same absence through
int64OrZero as 0. So a pokemon with encounter values and a NULL weather never
matched a no-boost update: 0 == -1 is false, the skip fell through, the entity
lock was taken, and the guard then compared 0 != 0 and did nothing. One wasted
entity lock per NULL-weather pokemon per weather flip in its cell.

Normalise the lookup value into the entity's encoding rather than mapping the
incoming side to -1. The entity guard is the one that decides whether there is
work; the skip exists only to avoid reaching it, so the skip has to speak the
guard's language, not the other way round.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MarshalJSON emits null for any invalid NullSeenType, so a record whose
seen_type degraded to Unknown and one that was never set arrive as the same
three characters. UnmarshalJSON decoded both back to code 0, Unset — the code
this PR proved destructive, since it is what updateFromWild's
`case Unset, Cell, NearbyStop` and updateFromNearby's `case Unset, Cell` act
on. A replay tool or a future read path round-tripping a degraded record
would have converted the safe sentinel into the damaging one.

Nothing in production unmarshals this type today, but it is exported and it
is the webhook payload's field type, so the boundary is reachable by anything
downstream.

The remaining asymmetries with Scan are deliberate and now documented on
UnmarshalJSON: Scan(nil) still yields Unset, because a SQL NULL genuinely
means the column has never held a value and a wild sighting filling it in is
correct; and UnmarshalJSON still errors on an unrecognised string where Scan
degrades, because a failed JSON decode strands no cache entry the way a failed
row load does.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
seenTypeScanWarns and seenTypeSetWarns were the only two pointer-typed
DropReporters in the codebase; the other five (raw_limiter, rtree_evictor,
fort_tracker, stats, ottercache) are values. They were pointers so a test
could swap in a fresh one, which is a write to a package global that
production reads on the row-load and decode paths — the shape init_test.go
already documents as a former data race for statsCollector.

There is no purely test-side seam here. A test asserting on its own throttled
line needs the one-second window to start fresh, and a shared DropReporter
has no way to offer that: its state is two unexported atomics, and the
alternatives all trade the race for something worse — relying on test order
so each reporter's asserting test runs first (breaks under -shuffle and on
the next test that scans an unrecognised value), sleeping out the window (a
second per test and still not exclusive), or merging the two asserting tests
into one so only a virgin reporter is ever needed.

So add Reset to DropReporter and drop the pointers. Both fields are already
atomic, so Reset is race-free against a concurrent Report by construction,
and the package variables are now never written at all. Production comes out
simpler than it went in: two globals lose an indirection and match their five
siblings, and two doc comments lose the paragraph explaining a pointer that
only existed for tests. That is a better trade than atomic.Pointer, which
would have made every read site Load().Report(...) to keep a swap that no
longer needs to happen.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Scan's doc comment overclaimed the self-heal. It said a later wild or lure
sighting fills an unknown seen_type back in; neither does. updateFromWild's
switch lists Unset, Cell and NearbyStop, updateFromNearby's lists Unset and
Cell and returns on default, and updateFromMap only sets a seen type on a new
record — so Unknown reaches no case in any of them. Only the encounter paths
set a seen type without first asking what the current one is. That is correct
behaviour, not a gap: a reader who made the documented claim true by adding
`case SeenTypeCodeUnknown:` to updateFromWild would rebuild the exact
downgrade the sentinel exists to prevent, so the comment now says which paths
heal, which deliberately do not, and what fails if someone changes that.

webhooks.md still typed seen_type as null.String, the one row the PR's earlier
sweep of that table missed. It is a NullSeenType now, with a line on the wire
form since the type is not one of the null.* family the conventions section
covers.

TestNullSeenTypeJSON's comment credited the API response. huma never reaches
NullSeenType.MarshalJSON — ApiPokemonResult's seen_type is a *string built by
Ptr(). The real consumer is the webhook payload, which webhooks/webhook.go
encodes with stdlib encoding/json.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both tests counted log lines against a real one-second wall-clock window and
demanded exactly 1. A runner loaded enough to spread 500 Scans (or four
refused SetSeenType calls) across a window boundary opens a second window and
logs twice — the throttle doing its job, reported as a failure. Rare, and a
rare flake is the worst kind to debug.

Assert what the tests are actually about instead: at least one line, and
fewer than one per event.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every Go step here passes -tags go_json, matching the Dockerfile, and
golangci-lint sets build-tags: [go_json] as well. Between them, no CI job
compiles the !go_json configuration at all — including jsonenc.go and
jsonenc_test.go, which exist only there. Nothing ships untagged, but
`go build ./...` is what a contributor types, so a break in that half would
land unnoticed and greet the next person to clone the repo.

go vet ./... type-checks every package and its tests, which covers the
compile without paying for a second full suite run. It needs no database, so
it sits before the migration step and fails fast.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jfberry

jfberry commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

On golbat_internal, settling this fully — including f0d4560a, not just the earlier defence: we don't need backward compatibility here in either direction. This is transitory operational state, off by default, and the contract we want is the simple one: load it and use it if it parses; if it doesn't, blank is fine; and when we write, we write this binary's view of the world unconditionally. The state evolves with the binary — last-writer-wins is the intended semantic, not an accident to defend against.

So please drop the refuse-to-overwrite guard as well (storedInternalHasUnknownFields and its skip path): 62dd77c5 was the right direction, and f0d4560a re-adds a defence we're deliberately choosing not to take. Beyond the ~100 lines, it costs an extra proto.Unmarshal of the stored bytes on every gated encounter save, and during the very window it protects, it prevents the running binary from persisting its own scan history on rows a newer build touched — which matters more to the running binary than preserving bytes it can't read. Plain rewrite, no guard.

@jfberry

jfberry commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Correction to my previous comment: on verification, 62dd77c5 had already reverted f0d4560a completely — no refuse-to-overwrite guard exists at head (storedInternalHasUnknownFields, the reporter, the metric, and its tests are all gone, zero leftovers), and the net effect of the pair is just the rewriteGolbatInternal extraction, which verified behavior-identical to the previous inline block. So golbat_internal is already exactly where we want it — treat my previous comment as the policy record, not an ask. Apologies for the churn.

Verification of the rest of the post-round-6 commits — all four asks landed correctly: the weather skip and entity guard now agree on "unchanged" for every input (NULL-weather case included); UnmarshalJSON(null) produces the inert sentinel, re-traced through every decode switch and pinned by test; the throttle race is genuinely eliminated; and all three doc corrections check out against the code. The untagged go vet step runs and passes, and the range assertions landed.

One deviation to accept explicitly: the throttle fix value-types the two globals and adds a small Reset() to util.DropReporter — technically production surface, where the ask was test-side-only. But it removes the pointer indirection that existed solely for the test swap, so it meets the actual constraint (don't compromise main code — this simplifies it) and is safer than the atomic.Pointer fallback we'd sanctioned. Fine as landed.

Residual nits, take or leave, none blocking: the SetSeenType throttle test's upper bound (< 4) has much less headroom than its sibling's (< 500) against a badly-preempted runner straddling wall-clock seconds; TestNullSeenTypeUnmarshalNullIsInert marshals via stdlib encoding/json where the file's convention is jsonenc.Marshal; webhooks.md's seen_type prose says null is emitted "when unset" and should add the degraded-value-during-mixed-deployment case; and DropReporter.Reset's two stores aren't atomic as a unit — worth a doc line that callers must quiesce reporters first (only tests call it today).

That closes the review from our side.

TurtIeSocks and others added 4 commits August 19, 2026 10:30
TestNullSeenTypeUnmarshalNullIsInert called encoding/json.Marshal
directly, bypassing the file's established convention of routing
through jsonenc.Marshal so golden-JSON tests track whichever codec
the current build tag actually selects (see jsonenc's package doc).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…nd headroom

Its upper bound was got >= len(refused), pinned at exactly 4 — the
refused-codes list's length. A runner preempted long enough to
straddle a wall-clock second boundary between each of the 4 calls
could open up to 4 separate throttle windows and log 4 warnings,
hitting the bound exactly and flaking a passing run.

Cycle the refused codes 125 times (500 total calls) instead, matching
the order of magnitude its sibling TestScanUnknownSeenTypeWarnIsThrottled
already drives (500 calls, asserting < 500), so the bound is
proportional to what the test actually exercises rather than pinned to
the length of a 4-element slice.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
webhooks.md said null is emitted "when unset". It is also emitted
when this binary reads a seen_type string it does not recognise (a
newer binary's value, seen during a rollback or a lagging replica in
a mixed deployment) and stores the inert SeenTypeCodeUnknown sentinel
instead — that sentinel serializes to null the same as unset.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Only tests call Reset today, and they quiesce production's Report
callers first, but nothing said that was required. Document that
callers must ensure no concurrent Report is in flight, since an
interleaved Report could otherwise observe a cleared count paired
with the old lastLog, or vice versa.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants