perf: shrink cached Pokemon 800→352 bytes (field packing, plain-struct scan history) - #394
perf: shrink cached Pokemon 800→352 bytes (field packing, plain-struct scan history)#394TurtIeSocks wants to merge 74 commits into
Conversation
Design for shrinking the cached Pokemon entity below Go's 512-byte allocation threshold, where GC mark cost jumps 3.1x (measured: 512 B with one pointer marks in 62.6 ms, 520 B in 194.4 ms, across 5M live entries). Chooses narrow null wrapper types (NullUint8 et al, each implementing sql.Scanner and driver.Valuer) over a single validity bitmask. The bitmask is 56 bytes smaller but requires a parallel DB-shaped struct at all six sqlx call sites, and fails at runtime rather than compile time when a NULL column meets a bare uint8. PokemonData 592 -> 224 bytes; Pokemon 800 -> ~400, which crosses from Go's 896 size class to the 416 class for roughly 480 bytes saved per cached pokemon. Also drops four fields that cost memory without earning it: Iv (the column is GENERATED ALWAYS AS ... VIRTUAL), Capture1/2/3 (no callers, absent from both the select columns and the upsert), changedFields (const-folded dead in production builds), and narrows SeenType from a 24-byte string header to a 2-byte enum. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The spec claimed `iv` is a GENERATED ALWAYS AS ... VIRTUAL column and that pokemonBatchUpsertQuery writing `:iv` against it was a pre-existing bug to resolve. Both are wrong. The schema comment at decoder/pokemon.go:113 is stale. sql/11_ivchanges.up.sql drops the generated column and adds a plain nullable float(5,2) in its place, so the column is real and writable and the upsert is correct as written. Iv could still be dropped and recomputed from the three IV fields, but that means changing the upsert, the select columns, and four call sites — one of them on the public API response path — to save 8 bytes against ~100 bytes of headroom. Narrowed to NullFloat32 and left alone instead. PokemonData 224 -> 232 bytes, Pokemon ~402 -> ~410, both still well under the 512-byte threshold. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
sql/7_add_height_size.up.sql renames the original size double(18,14) to height and adds a new size tinyint unsigned, so NullUint8 for Size is a direct column match rather than a judgement call about observed range. That is the third stale claim found in the schema comment at decoder/pokemon.go:88-140, after the iv generated-column and the four-value seen_type enum. Added a section telling implementers to verify column types against the migrations rather than the comment. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Eight tasks from the approved design. Task 0 is the pprof measurement that decides whether the work is worth doing at all; tasks 1-6 are the implementation; task 7 records the measured outcome against the estimate. Each task ends with an independently testable deliverable. The size assertion test lands first and its expected values are updated by every subsequent task, so each task's memory effect is explicit in its own diff. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
unsafe.Sizeof works directly in a table entry; the wrapper functions were indirection a reviewer would rightly flag.
Instruments struct size assertions that later tasks will update as the Pokemon type is packed. Pins current reality: PokemonData at 592 bytes, Pokemon at 800 bytes (above the 512-byte GC threshold). Documents the gcSizeThreshold constant explaining why staying under 512 bytes matters. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Task 2's review found NullFloat32.MarshalJSON formats at bitSize 32 where guregu/null.Float formats at 64, so weight emits 6.7 rather than 6.699999809265137. Both documents asserted byte-identical JSON, which is now knowingly false for that one type. Ruled: keep bitSize 32. The values are protobuf float fields promoted through float64() at decoder/pokemon_decode.go:749,751, so the extra digits were promotion noise. The divergence gets its own test so it stops being accidental. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ack to NullUint64.Scan - NullFloat32.MarshalJSON now returns an error for NaN/Inf instead of emitting invalid JSON tokens. This prevents client-supplied malformed proto floats from corrupting API payloads. - Updated NullFloat32 doc comment to explicitly document the bitSize 32 divergence from guregu/null (bitSize 64) and explain why it's correct. - NullUint64.Scan now has a ParseUint fallback for []byte/string inputs to handle unsigned values above MaxInt64 (unlikely but future-proof). - Updated NullUint64 doc comment to clarify the actual schema precondition (signed bigint columns) and that unsigned values are tolerated. - Extended TestUint64FullRange to test the []byte path. - Added TestFloat32JSONIsNarrower to document the deliberate JSON divergence. - Added TestFloat32JSONNaNInf to verify error handling for special float values. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PokemonData 592 -> 280 bytes (the design doc estimated 232; the "pointer-carrying, last" field group's own 8-byte alignment requirement reintroduces 7 bytes of padding a fully byte-count-monotonic order would have avoided — see entity_sizes_test.go and pokemon.go's struct doc comment for the measured breakdown). Pokemon 800 -> 456 bytes; still carries changedFields and internal, both left for a later task. Nullable numerics move from guregu/null's 16-byte wrappers to nulltypes equivalents sized to the actual columns, and fields are reordered by descending alignment. Setters keep their null.X signatures and clamp out-of-range values to the column boundary, counted by golbat_field_clamped_total. Direct field assignments in pokemon_decode.go and weather_iv.go move to setter calls (or, for AtkIv/DefIv/StaIv, direct clamp+assign inside calculateIv, which remains their sole mutator by design) so they cannot skip the clamp. Drops Capture1/2/3, which had setters but no callers and appeared in neither pokemonSelectColumns nor pokemonBatchUpsertQuery. The webhook payload keeps its capture_* fields at 0, which is what consumers have always received. The API response's float fields (weight/height/iv) widen through a shortest-round-trip string rather than a naive float64 cast, so widening a narrowed float32 back to *float64 doesn't expose float32 rounding noise at float64 precision (float64(float32(3.14)) is 3.140000104904175, not 3.14) - this keeps the golden JSON test's wire format byte-identical. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Task 3 measured PokemonData at 280 bytes, not the 232 the plan estimated. The estimate was arithmetically impossible: the declared fields carry 273 bytes of payload and the struct aligns to 8, so every ordering rounds to 280 with 7 bytes of mandatory trailing padding. Field order remains load-bearing — a careless reordering still adds padding — but 280 is the floor for this field set. Pokemon came out at 456, inside Go's 480-byte size class and under the 512 threshold. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Same correction as the spec: 273 bytes of payload align to 280, so the plan's Step 8 constant and commit-message template were both wrong. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review fixes for the PokemonData narrowing commit:
- The struct's doc comment and TestPokemonEntitySizes's wantPokemonData
comment both claimed a fully-monotonic field order would have avoided
the 7 bytes of padding before the pointer group. That's arithmetically
false: the field payload sums to 273 bytes, Go's 8-byte struct
alignment rounds any ordering of this field set up to 280, and the
current order already hits that minimum - the 7 bytes just move
elsewhere (e.g. trailing padding) under any other ordering, they don't
disappear. Corrected in both places; the design doc's 232/264 estimates
were arithmetic errors, not a target this order missed.
- The size-class remark ("[257, 288] size class") was attached to
wantPokemonData, but PokemonData is never independently heap-allocated
(embedded in Pokemon, copied by value into []PokemonData batches), so
no size class applies to it. Moved to wantPokemon's comment, where 456
landing in the 480-byte class under the 512 threshold is the actually
load-bearing fact.
- Added decoder/pokemon_clamp_test.go: IncFieldClamped and the four clamp
helpers had no coverage. A small counting fake wrapping the noop
collector (mirroring the swap/restore pattern already used in
station_battle_test.go) pins that out-of-range Set* calls land on the
column boundary and count exactly once, in-range values count zero,
and clampFloat32 never clamps (weight/height/iv are documented as
always in range).
- Documented why SetHeight/SetWeight compare the clamped float32 values
exactly instead of with the old null.Float tolerance: both sides are
already narrowed to float32 by the time they're compared, so the
float64-promotion jitter the tolerance existed for is collapsed before
the comparison runs.
Verified with -race (beyond the task's specified test command) that the
counting-fake swap pattern doesn't introduce a new race: the one race
`go test -race ./decoder/` reports (station_battle_test.go vs the stats
aggregation worker) reproduces identically with pokemon_clamp_test.go
removed from the package, confirming it predates this change. Flagged
separately for a dedicated fix rather than addressed here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review round 2 fixes: - The clamp-path test added in the previous fix propagated the exact race decoder/init_test.go's comment warned about: reassigning the package-level statsCollector from a test races the background stats aggregation worker and StartWorkerBacklogReporter's ticker, both of which read it continuously for the life of the test binary. Fixed at the root instead of narrowing the test: statsCollector is now atomic.Pointer[stats_collector.StatsCollector], read through a new getStatsCollector() accessor. Converted all ~80 call sites (18 non-test files, mechanical) and all four test-side swaps (the new pokemon_clamp_test.go plus three pre-existing ones in station_battle_test.go) to a shared setStatsCollectorForTest helper. `go test -race ./decoder/` is now clean across repeated runs, including the specific test that used to fail (TestCreateStationWebhooksEmitsFutureBattle) - this also resolves the pre-existing race filed separately as a background task, which is now withdrawn. - The previous fix round corrected the padding-arithmetic mistake in pokemon.go and entity_sizes_test.go but left two copies of the same false claim standing in the task report itself (the "Measured sizes" narrative and a Concerns bullet), plus a self-audit line claiming they were already fixed. Edited both in place to match the corrected arithmetic (273-byte payload rounds to 280 under 8-byte alignment under any field ordering; this order already hits that floor). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Covers the failure mode the narrow-wrapper approach was chosen to avoid: a
NULL column landing in a narrowed field. Also pins the two precision cases
that must not regress, the full 64-bit cell_id and float64 lat/lon.
Skips when GOLBAT_TEST_DSN is unset, since CI defines no database service.
Two bugs surfaced and fixed while proving this against a live database, both
in the test's own setup rather than in nulltypes:
- shiny has a non-NULL schema DEFAULT ('0'), so omitting it from the insert
(as done for every other nullable column) inserted 0/Valid instead of NULL,
defeating that one check. Now bound explicitly to NULL.
- defer db.Close() ran before t.Cleanup's delete (cleanups fire after a
function's own defers), so the sentinel-row delete silently failed on
every run. Close is now registered via t.Cleanup, ordered to run last.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
seen_type is an eight-value enum held in a 24-byte null.String with a heap pointer. NullSeenType stores the code and converts at the database and JSON boundaries, so both wire formats are unchanged. An unrecognised enum value is now an error rather than a silently stored string, since the alternative is corrupting scan statistics when the game adds a seen type ahead of the migrations. PokemonData 280 -> 256 bytes, Pokemon 456 -> 416 bytes (measured, not the design doc's estimate). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 24-byte []string slice header was unconditionally present on every
cached Pokemon even though dbDebugEnabled is a build-tag const, so every
append into it was already const-folded away in production. Replaced it
with `debug pokemonDebugState`, a field whose type is defined once per
build tag (mirroring the existing dbDebugEnabled/dbDebugLog split):
db_debug.go gives it a real []string accumulator under -tags dbdebug,
db_debug_off.go gives it a zero-sized stub otherwise. The debug build's
save-time log output is unchanged — dbDebugLog still gets one aggregated
"changed=[...]" line per save, not per-field logging.
Measured: unsafe.Sizeof(Pokemon{}) goes from 416 to 392 in production.
This does NOT change the allocator's size class: 392 still rounds up to
416, confirmed empirically via runtime.MemStats.TotalAlloc across 200k
live *Pokemon (416.0 bytes/object, unchanged). The win is a dead field
gone from the layout and one fewer pointer word in the GC scan bitmap,
not a reduction in allocated memory - the entity_sizes_test.go comments
spell this out so nobody "fixes" the number later assuming it moved.
Adds TestPokemonUnderGCThreshold, the acceptance test for the whole
packing effort: Pokemon must stay under 512 bytes, above which measured
GC mark cost jumps 3.1x. Also removes the now-stale nolint:unused off
gcSizeThreshold, which this test is the first consumer of.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Added Results section to the design doc capturing what was actually measured (256 B PokemonData, 392 B Pokemon in production) and clarifying where the design's predictions were wrong: - The 232 B estimate for PokemonData was arithmetically impossible (minimum achievable 280 B from field payload + alignment). - The Task 6 field deletion produced no allocator savings (both 416 B and 392 B land in the same size class). The measurable outcome: allocator gives back 480 bytes per cached pokemon (896 class -> 416 class), totaling 2.4 GB at 5M entities, matching the design's magnitude estimate. CPU impact remains unverified (required production profile never captured). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five cleanups from the whole-branch review (verdict: ready to merge, no Critical/Important findings): - Rewrite PokemonData's load-bearing layout comment, which described the pre-task-5 280-byte/273-payload layout. Measured the real numbers via unsafe.Offsetof/Sizeof: 256 bytes, 251-byte payload, 25-byte 1-byte group, 96-byte pointer group, 5 bytes of padding at offset 155->160. Also fixes the comment's stale pointer to the schema comment further down the file, which now documents known divergences rather than containing stale claims. - Drop the dangling "REMINDER! Keep hasChangesPokemon updated" comment; that function no longer exists anywhere in the codebase. - NullSeenType.Value() now errors instead of returning "" when Code is past the end of seenTypeStrings, closing the path by which a bad code could silently write an empty string into the seen_type ENUM column. Unreachable today (every construction validates), but Code and SeenTypeFrom are exported, so the invariant was held by discipline alone. - TestPokemonFullRowRoundTrip now asserts Weight and adds/asserts Iv, pinning NullFloat32's float64-widening Value() against both column precisions it writes into (weight is double(18,14), iv is float(5,2)). Unverified by execution — GOLBAT_TEST_DSN was intentionally left unset. - Fix a test comment in nulltypes_test.go that cited the wrong hex value for its decimal literal (0xc556d6894e95b000, not 0xC5A0...); the decimal literal and assertion were already correct. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Absolutely will need to be benchmarked; and against go 1.27 which changes allocation to be more efficient anyway. |
|
Incidentally, since the null package already supports generics you may not need the newly defined nulltypes but rather just right-size in the current package The embedded protobuf is absolutely ripe for change, it's a smell that it's in there in protobuf form - I did once replace it with a conventional structure and move to protobuf on demand (almost nowhere --> it is only marshalled when writing to disk, and the default is that it isn't). The pokemonChanged value could be renamed to a generic name to encapsulate the change list, and rolled out to the other objects for a near free improvement. |
|
Finally, pull the SeenTypeCodeWild = 0 fix into this PR - and the re-widening of the webhook to float64 and reconversion seem like they are wasted steps for some misattributed compatibility requirement |
The maintainer (jfberry) pointed out that github.com/guregu/null/v6 already ships a generic null.Value[T] (embedding stdlib sql.Null[T]) that does the same job as the hand-rolled decoder/nulltypes package: identical sizes for every narrowed type, Scan range-rejection out of the box, and float32 JSON marshalling at native 32-bit precision with no custom MarshalJSON needed. Confirmed all of this empirically before touching any code. Deletes decoder/nulltypes entirely (702 lines) and converts every field, clamp helper, and constructor call across 11 files to null.Value[T]. PokemonData/Pokemon still measure 256/392 bytes (TestPokemonEntitySizes, unchanged) and the golden API JSON string in api_pokemon_response_test.go is untouched. CellId moves from a NullUint64 storing a bit-reinterpreted S2 cell id to null.Value[int64], matching cell_id's actual signed-bigint column type. null.Value[uint64] can't Scan a negative int64 (it round-trips the driver value through a string and ParseUint rejects the leading "-"), which is exactly the case cell_id needs since real S2 cell ids are frequently negative. Giving CellId the correct signed type turns two former workarounds into straight-line code: SetCellId no longer needs its uint64(v.Int64) bit-reinterpret cast, and the API response builder's widenPtr[uint64, int64] call collapses to a plain .Ptr(). One read site (pokemon_state.go's MatchStatsGeofenceWithCell call) picked up an explicit uint64(...) cast instead, matching the pattern gym/pokestop/station already use for their own CellId fields. SpawnId stays null.Value[uint64]: go-sql-driver/mysql returns a native uint64 for the unsigned bigint column, and pokemon_state.go's existing strconv.FormatUint(pokemon.SpawnId.ValueOrZero(), 16) call requires a uint64 argument, so it was both the type the driver already produces and the type every downstream reader wants. Verified via go build/test under both go_json and go_json dbdebug, -race, golangci-lint, and TestPokemonNullColumnRoundTrip / TestPokemonFullRowRoundTrip against a live MariaDB dev instance (the latter now also exercises a negative CellId, -16, in place of the old hex bit-pattern fixture that no longer type-checks against int64). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review finding: sql.Null[uint64].Value() delegates to
driver.DefaultParameterConverter.ConvertValue, which explicitly refuses
any value with the high bit set ("uint64 values with high bit set are
not supported"). The nulltypes.NullUint64.Value() this replaced did a
bit-reinterpret (return int64(n.V), nil) that always succeeded, so the
type swap silently dropped a real capability: null.Value[uint64] can
Scan the full unsigned range of spawn_id (bigint unsigned) but cannot
always Value() it back. Unreachable today only because both
SetSpawnId call sites route through strconv.ParseInt(s, 16, 64), which
can't produce a sign-bit value — an upstream accident, not a proof of
equivalence.
Switches SpawnId to null.Value[int64], matching CellId's existing
type and reasoning: int64 has no such restriction, and real spawn ids
are ~48-bit, far below where either type's range would matter.
SetSpawnId now stores v.Int64 directly with no cast, same shape as
SetCellId. Downstream: pokemon_state.go's
strconv.FormatUint(..., 16) call picked up an explicit uint64(...)
cast (matching the pattern already used for CellId at pokemon_state.go
:282), and three call sites building *int64 API fields
(api_pokemon_response.go, api_pokemon_scan_v2.go, api_pokemon_scan_v3.go)
simplified from widenPtr[uint64, int64](...) to a plain .Ptr(), since
SpawnId.Ptr() now already returns *int64.
Also corrects a gap the previous commit's report should have caught:
TestPokemonFullRowRoundTrip never actually set or asserted SpawnId
against the live database - the value it cited as proof
(SpawnId: null.ValueFrom(uint64(7777))) lives in the unrelated
JSON-only golden test in api_pokemon_response_test.go, which never
touches the database. Added a real SpawnId round-trip assertion here
so the type this commit picks is actually exercised against MariaDB,
not just proven to compile.
Sizes unchanged (PokemonData=256, Pokemon=392): int64 and uint64 are
both 8-byte payloads, so null.Value[int64] and null.Value[uint64] are
both 16 bytes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
SeenTypeCodeWild = iota made 0 - the Go zero value - the enum's most common real value. Any code reading .Code without first checking .Valid would silently treat a NULL seen_type as wild; one such site existed in the write-behind delay logic and was caught during review. Insert SeenTypeCodeUnset at 0 and shift the eight real codes up by one. seenTypeStrings gets a leading "" entry so String() reports "" for both Unset and any out-of-range code via the same lookup; seenTypeCodes excludes that entry so ParseSeenType can never produce Unset from a real string. Value() now rejects any code whose String() is empty, which additionally closes off Unset - writing "" to the seen_type ENUM column is accepted silently by MariaDB. Searched every use of a SeenTypeCode value: all go through named constants, String()/Value()/MarshalJSON(), or ParseSeenType - never a raw code number persisted, transmitted, or compared - so the shift is safe. Sizes (256/392) and the API golden JSON are unaffected, as expected: this changes which integer represents each seen type, not any string or column width. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
nullFloatFromFloat32 widened the stored null.Value[float32] Weight/ Height back to guregu/null.Float at the webhook payload boundary, to keep webhook bytes identical to before the struct-packing PR. The maintainer judged that requirement misattributed: webhook consumers are this project's call, not a wire contract owed byte-identity. PokemonWebhook's Weight/Height now hold null.Value[float32] directly and nullFloatFromFloat32 is deleted, its only caller gone. This changes webhook output: weight and height now emit 6.7 where they previously emitted 6.699999809265137 - the short form the API endpoint already emits, for the same reason - both originate as protobuf float fields, and the extra digits were an artifact of a float64() promotion that never carried information. Narrowed the integer fields (Gender, Cp, Form, Costume, the three IVs, PokemonLevel, Move1/2, Size, Weather, DisplayPokemonId/Form, LastModifiedTime) the same way, deciding to treat this as one consistent policy rather than a case-by-case call: the widening existed for one reason (byte-identical webhooks) that no longer applies to any field, narrowing an int is a JSON no-op (guregu/null's generic Value[T] marshals via encoding/json, which renders 15 as "15" regardless of int width - verified against the package source), and a struct where the float fields mirror storage width but the int fields still widen for no live reason is an inconsistency with no upside. nullIntFromUint survives for its two other callers (a setter-feeding reverse conversion in pokemon_decode.go, and the stats collector's UpdateVerifiedTtl, which is guregu/null throughout). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pokemon embedded `internal grpc.PokemonInternal` purely to hold Ditto-detection scan history in memory. That carried 64 bytes of protobuf machinery (MessageState, sizeCache, unknownFields, slice header) on every cached pokemon, plus 48 bytes of the same overhead on every history entry, for a message that is only ever marshalled when writing the golbat_internal column — which is gated behind pokemon_internal_to_db, off by default. The field becomes `scanHistory []*pokemonScan`: a plain 44-byte struct with the same twelve int32/bool fields as the proto. Elements stay behind pointers because the Ditto code takes a pointer to a history entry, holds it across other work and mutates through it; a value slice would have `&s[i]` invalidated by any reallocating append. The protobuf remains the wire format and is untouched. Conversion happens at exactly two boundaries: populateInternal unmarshals the stored bytes into a temporary grpc.PokemonInternal, and savePokemonRecordAsAtTime builds one to marshal. Bytes written before this change decode identically, and bytes written after it are byte-identical to what the old code emitted — both pinned by a golden payload in decoder/pokemon_scan_test.go. Everything else is a type swap: locateScan, locateAllScans, checkScans, setDittoAttributes, resetDittoAttributes, confirmDitto, detectDitto, addEncounterPokemon and recomputeCpIfNeeded keep their logic verbatim, and the five helper methods move from grpc/helper.go (hand-written, now dead) to the new type unchanged. pokemonScan grows a String() so the Ditto debug logs and error messages keep the proto-text content they had, including "<nil>". Pokemon: 392 -> 352 bytes, and unlike the previous task this one moves the allocator. 352 is itself a Go size class, so the bytes handed out per cached pokemon go 416 -> 352 (measured, n=200000: 352.0 exactly, against 416.0 for a 392-byte control). History entries go 96 -> 48. PokemonData is untouched at 256. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…dary TestPokemonEntitySizes' doc comment still described the pre-task-7 struct: 392 bytes, mid-class, "24 bytes of free space ... so a modest field addition is free in allocator terms". At 352 that is exactly inverted. 352 is itself a Go size class, so the next byte added — a single bool — costs 32 per cached pokemon, roughly 160 MB at 5M, about half of what removing the embedded protobuf just won. Someone hitting the failed assertion would have read the old comment, concluded the field was free, and bumped the constant. Also state in pokemon_scan.go that a field added to grpc.PokemonScan must be added to pokemonScan and to BOTH converters, and back that with a test rather than a comment: TestPokemonScanCoversEveryProtoField compares the exported field sets of the two structs and cross-checks the count against the wire descriptor. The golden-bytes fixture cannot catch this — a fixture only knows about the fields it was built with, so a thirteenth proto field would decode into the temporary at the read boundary and never be written back. Verified the new test fires: an extra field on pokemonScan fails it by name, and a missing one fails the build. The rest is small. scanHistoryFromProto uses the nil-safe getters on repeated elements. The round-trip test indexes its fixture slice instead of taking the address of a range variable, which is correct under Go 1.22+ loop semantics but reads like the classic aliasing bug. The design doc's results sections said 392/416 and "120 bytes of headroom" as though they were still current. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Renames Pokemon's pokemonDebugState to the entity-neutral
debugChangeAccumulator and rolls it out to the eight other entities that
still carried a raw `changedFields []string` (Pokestop, Gym, Station,
Spawnpoint, Incident, Tappable, Route, Player), replacing an
unconditional 24-byte slice header with a build-tag-split type that
costs zero bytes in production builds.
Station, Spawnpoint, Incident, and Tappable cross a Go allocator size
class boundary from this (Spawnpoint 136->104, dropping from the
144-byte class to 112 — the one flagged as mattering most, since it's
cached in the millions). Pokestop, Gym, Route, and Player shrink the
same 24 bytes but land in the same allocator class as before, so they
save GC scan-bitmap pressure but nothing at allocation time; measured
both empirically (TotalAlloc delta / n) rather than assumed from the
static size-class table, which undercounts for pointer-containing
("scan") structs at this size range on this Go toolchain.
Tappable and Player had `changedFields` as their last struct field, so
renaming in place would have hit the zero-sized-last-field padding trap
(dodged the same way Pokemon's original conversion did); both got the
field reordered ahead of `newRecord` with a comment explaining why.
Adds TestClassMovedEntitySizes pinning the four entities whose class
actually moved, skipping the four that didn't per the same reasoning
TestPokemonEntitySizes already established.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The comments explaining jsonenc said -tags go_json was what makes
huma_api.go select goccy for API responses ("the codec huma_api.go
uses to serve every API response" — under a build-tag heading). That
gets the causality backwards: huma_api.go's newHumaConfig imports
goccy directly with no build constraint, so every huma-registered
route — everything the golden tests here pin — is marshaled through
goccy unconditionally, tag or no tag. What -tags go_json actually
gates is gin's own internal JSON codec (github.com/gin-gonic/gin/
codec/json), used by the raw c.JSON() calls outside huma (routes.go's
PokemonScan and GetHealth).
No behavioral gap results — the Dockerfile and Makefile both default
to building with the tag, so huma's unconditional goccy and gin's
tag-selected goccy never disagree in a real build — but the comments
described a mechanism that doesn't exist. Rewrote jsonenc.go's package
doc as the one authoritative explanation of what the tag does and
doesn't gate, and pointed every test file's shorter comment at it
instead of repeating (and this time getting wrong) the same claim six
times. jsonenc's tag-gated design is unchanged and still correct: it
makes each test track whichever codec the build it's compiled into
actually selected, which is the real reason for it.
Comment-only change, confirmed via diff — no marshal calls, no golden
strings, no test logic touched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review round 4, tail items 1-2 (combined: both edit the same clamp/narrow doc-comment block in decoder/pokemon.go, and the shared `saturate` comment references facts from each — splitting them would leave one commit's comment referring to an identifier the other hadn't introduced yet). narrowUint32 had no production caller: unlike narrowUint8/16, which compare raw proto values against stored Form/Weather/Costume/Gender columns throughout pokemon_decode.go and weather_iv.go, nothing does the same for either uint32 column (expire_timestamp, updated). Its only callers were its own test cases. Removed the function and the three cases, and updated the doc comments that used to enumerate it alongside narrowUint8/16. clampUint's `limit` parameter was always the caller-supplied ceiling, but for its three plain instantiations (clampUint8/16/32) that ceiling is just T's own natural maximum. Split clampUint into a zero-arg-ceiling form that derives it via int64(^T(0)) — the standard idiom for "largest value this unsigned type can hold" — and clampUintCeiling, the old explicit-limit body, kept for clampIv, whose ceiling (15, the game's per-stat IV cap) is narrower than its storage type's and genuinely can't be derived from T. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… removal Review round 4, tail item 3. decoder/stats.go's comment on the removed sc == nil check read "no guard needed before this used to be a real early-startup window" — two facts welded into one ungrammatical sentence. git log -p -L traced it to commit 7972303 (round 2, task 4), which seeded statsCollector with a noop at package init and removed the guard that used to be necessary before that seeding existed. Reworded as two separate sentences saying so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… narrowing Review round 4, tail item 4. An earlier round narrowed PokemonWebhook's numeric/bool fields from guregu's null.Int/null.Bool/null.Float to the storage-width-matched null.Value[uint8] /null.Value[uint16]/null.Value[uint32]/null.Value[bool]/null.Value[float32], but webhooks.md's payload table was never updated to match. shiny was the field the maintainer named, but checking every row against the actual struct (decoder/pokemon_state.go's PokemonWebhook) found the same drift on all 18 narrowed rows, not just that one. Every other webhook payload type (gym/raid/pokestop/quest/incident/station/...) still uses the classic guregu types, so this drift was isolated to the pokemon table but wide within it. Fixed all 18 rows and added null.Value[T] to the "Nullable fields" conventions section so the table's types trace back to a defined serialization convention. seen_type's row (null.String) was left as-is: its Go type is a distinct custom type (NullSeenType) whose JSON behavior matches null.String exactly, and it predates this narrowing round. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review round 4, tail item 5. decoder.statsCollector's own package-level initializer already seeds it with a noop (a later round of this same PR), which runs before any init() function per the Go spec — including in test binaries. Confirmed this call was genuinely redundant rather than assuming so from the name: checked its one non-obvious side effect (marking statsCollectorSet, which gates InitWriteBehindQueue's boot-ordering panic) and grepped every decoder _test.go for InitWriteBehindQueue — zero calls, so nothing in the test binary depends on that flag either. Removed the call. As a direct consequence, fixed stats_collector_init_test.go's TestStatsCollectorSeedIsInTheVariableInitializer comment, which explicitly said the package variable "has already [been] overwritten by the time any test runs" by this exact call — no longer true, so reworded to explain why the test reads newSeededStatsCollector() directly regardless (it's pinning the initializer, not working around stale shared state). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review round 4, extra item 6 (flagged by a previous task's report but
left in scope since it was outside that task's ten items).
Confirmed dead before removing: grepped every .Xxs/.Xxl read or write in
decoder/ (zero hits beyond the struct declarations), and read
updatePokemonLookup's struct literal directly to confirm it never
populates them. The XXS/XXL size filters (IncludeXxs/IncludeXxl in
api_pokemon_scan_v1.go) read pokemonLookup.Size instead, exactly as
pokemon_lookup_narrow_test.go's own pre-existing comment already
documented ("pinned as-is rather than removed here because removing them
is a separate change" — this is that change).
PokemonLookup is loaded 15-20M times/sec in production profiles (14% of
CPU), so its width is a direct multiplier on scan cost. Removing the two
dead bools drops unsafe.Sizeof(PokemonLookup{}) from 18 to 16 bytes, and
PokemonLookupCacheItem (which embeds it) from 26 to 24 — updated both
pins in TestPokemonLookupSizes. Also updated cachebench's lkPokemon,
which claims to mirror PokemonLookup exactly and had the same two unused
fields.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review round 4, extra item 7 (reported as having no reachable window today; judged here as worth guarding anyway). main.go's package-level statsCollector was a bare nil interface until main() assigned the real collector partway through boot. decode.go (~40 call sites), routes.go (5), and grpc_server_raw.go (2) all call methods on it with no nil check; only raw_limiter.go defensively guarded its own two call sites. Traced main()'s boot order: the HTTP routes and the gRPC listener both start well after the assignment, and initRawProcessingLimiter (the one already-guarded goroutine) is also called after it, so nothing reaches any of those ~47 call sites before the real collector lands. No reachable window exists today. Guarded it anyway rather than leaving it: decoder and db already carry this identical fix from earlier rounds of this same PR, for the same "no window today, but the guarantee is cheaper to hold than to re-derive per call site forever" reasoning documented in db/dbDetails.go's own comment for its equivalent seed. Leaving main as the third unfixed instance means the next caller added to any of those ~47 sites, or any future reordering of initRawProcessingLimiter, silently reintroduces a nil-panic that's already been hit and fixed twice in sibling packages. Seeded with stats_collector.NewNoopStatsCollector() at declaration, matching decoder's and db's pattern, and removed the two now-dead `if statsCollector != nil` guards in raw_limiter.go. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
db/dbDetails.go claimed "no test swaps it" as the reason a plain variable is safe there, and the test added in the same commit swaps it — TestSetStatsCollectorRefusesNil restores it through a t.Cleanup. Harmless in practice, since that package's tests are sequential, but it is load-bearing guidance that is false as written. The comment now states the actual invariant: nothing writes it while anything reads it concurrently, which is what decoder's atomic.Pointer exists for and this does not need. InitWriteBehindQueue's new ordering guard is a crash mode that did not exist before, and golbat's own main() cannot reach it. A fork with its own main() ordered the other way round can, so the doc comment now says so and names the fix — swap the two calls, do not delete the check. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Widening ExpireTimestamp and Updated to int64 dropped the `minimum: 0` huma had been inferring from uint32. That was a side effect of the widening, not part of the point: the bound is real — all four timestamps are backed by unsigned columns — and the goal was to make the four agree, which they can do on the bound as well as the type. An explicit `minimum:"0"` tag on all four restores it, and gets first_seen_timestamp and changed a floor they never advertised at all, since they were already int64 and huma had nothing to infer from. Schema effect: expire_timestamp and updated keep minimum: 0 across the widening rather than losing it, and first_seen_timestamp and changed gain it. TestApiPokemonResultSchemaWidths now requires it on all four. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
It logged once per refused call. Unreachable today — all 13 call sites pass a compile-time SeenTypeCode* constant — but the branch sits on the decode path, and the thing that would make it reachable, a computed code from some future caller, would make it reachable once per sighting. That is the shape util.DropReporter exists for, and it is what NullSeenType.Scan's mirror-image warning on the read side already uses. Aggregated to one line a second through a reporter of its own, with the refused count in the message. The test now pins the throttle: four refusals, one line. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
The protobuf-to-plain-struct conversion introduces forward-compatibility data loss when internal data is rewritten by an older binary. Otherwise, the targeted production and dbdebug decoder tests passed. Review comment:
Arguably doesn't need fixing? |
The pre-pokemonScan write boundary marshaled the very grpc.PokemonInternal it had unmarshaled, so protobuf fields this build has no definition for rode along in unknownFields for free. pokemonScan is a plain struct with nowhere to put them, so rebuilding from it copies known fields only — during a rolling upgrade or after a rollback, the next encounter would quietly replace a newer node's row with a subset of itself. Storing the raw bytes per pokemon would fix it and cost 24 bytes on Pokemon, pushing it from 352 into the 384 size class — roughly 104 MB at the 3.25M cached in production, to protect a path that is off by default. Merging unknown fields per element is not possible either: scan-history entries have no stable identity to match on. So refuse instead. rewriteGolbatInternal unmarshals the stored bytes and checks both levels — PokemonInternal itself and every scan_history element, the latter being the historically likely one — and when either carries unknown fields it leaves the row exactly as it found it. Zero memory, runs only with pokemon_internal_to_db enabled, and turns silent loss into deliberate non-overwrite. Same call SetSeenType makes when handed a code it cannot render. The refusal skips RemoveDittoAuxInfo too: that trimming exists to keep the stored column small, and there is no column write here to keep small. Undecodable bytes are not unknown fields — populateInternal has already dropped the history for them — so they stay rewritable rather than stranding the row forever. Warnings aggregate through util.DropReporter (every encounter save on an affected row takes the branch, so the unaggregated form would be a line per encounter), and golbat_pokemon_internal_rewrite_skipped_total lets an operator see it happening instead of guessing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
This (protobuf defence) change is an overreach. This is a transitory field, and there is no harm in losing fidelity when the default isn't even to store it. |
|
Round-6 verification is done and it's clean: both blockers are genuinely fixed and were verified empirically — the CI job is green with the MariaDB round-trip tests actually executing (not skipped) under both production tag sets, @Mygod — your advice wanted on
|
|
Damn why is a bot tagging me and roasting my code. Yes, that should be fixed. |
Reverts f0d4560. @jfberry's ruling: the change is an overreach. golbat_internal is a transitory field, there is no harm in losing fidelity, and the default is not even to store it. That is right. I weighed the memory cost of preserving the unknown bytes and never asked the prior question — whether the data was worth defending at all. It is scan history in a cache-like column, behind pokemon_internal_to_db, which defaults off, and the next encounter rebuilds it. A guard, a metric, a throttled warning and three tests to protect that is cost with no matching risk. Removed: the refusal branch, storedInternalHasUnknownFields, the internalUnknownFieldSkips DropReporter, golbat_pokemon_internal_rewrite_skipped _total with its interface method, noop and prometheus wiring and registration, and the three tests that covered them. Kept: the rewriteGolbatInternal extraction itself, which the guard was bolted onto rather than caused. savePokemonRecordAsAtTime is a long orchestration function, and a named method reads better there than fourteen lines of protobuf marshaling and Ditto trimming inlined mid-block. It also gives the write boundary a name to grep for and somewhere to document its contract, next to the conversion helpers it uses, instead of pointing at a region of another file. The body is byte-for-byte what was inline before, minus the guard. Pokemon is still 352 bytes and PokemonData still 256. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ProactiveIVSwitch's cheap skip compared the incoming weather against PokemonLookup.Weather, which spells "no weather" as lookupInt8's -1, while the entity-level guard right after the lock reads the same absence through int64OrZero as 0. So a pokemon with encounter values and a NULL weather never matched a no-boost update: 0 == -1 is false, the skip fell through, the entity lock was taken, and the guard then compared 0 != 0 and did nothing. One wasted entity lock per NULL-weather pokemon per weather flip in its cell. Normalise the lookup value into the entity's encoding rather than mapping the incoming side to -1. The entity guard is the one that decides whether there is work; the skip exists only to avoid reaching it, so the skip has to speak the guard's language, not the other way round. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MarshalJSON emits null for any invalid NullSeenType, so a record whose seen_type degraded to Unknown and one that was never set arrive as the same three characters. UnmarshalJSON decoded both back to code 0, Unset — the code this PR proved destructive, since it is what updateFromWild's `case Unset, Cell, NearbyStop` and updateFromNearby's `case Unset, Cell` act on. A replay tool or a future read path round-tripping a degraded record would have converted the safe sentinel into the damaging one. Nothing in production unmarshals this type today, but it is exported and it is the webhook payload's field type, so the boundary is reachable by anything downstream. The remaining asymmetries with Scan are deliberate and now documented on UnmarshalJSON: Scan(nil) still yields Unset, because a SQL NULL genuinely means the column has never held a value and a wild sighting filling it in is correct; and UnmarshalJSON still errors on an unrecognised string where Scan degrades, because a failed JSON decode strands no cache entry the way a failed row load does. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
seenTypeScanWarns and seenTypeSetWarns were the only two pointer-typed DropReporters in the codebase; the other five (raw_limiter, rtree_evictor, fort_tracker, stats, ottercache) are values. They were pointers so a test could swap in a fresh one, which is a write to a package global that production reads on the row-load and decode paths — the shape init_test.go already documents as a former data race for statsCollector. There is no purely test-side seam here. A test asserting on its own throttled line needs the one-second window to start fresh, and a shared DropReporter has no way to offer that: its state is two unexported atomics, and the alternatives all trade the race for something worse — relying on test order so each reporter's asserting test runs first (breaks under -shuffle and on the next test that scans an unrecognised value), sleeping out the window (a second per test and still not exclusive), or merging the two asserting tests into one so only a virgin reporter is ever needed. So add Reset to DropReporter and drop the pointers. Both fields are already atomic, so Reset is race-free against a concurrent Report by construction, and the package variables are now never written at all. Production comes out simpler than it went in: two globals lose an indirection and match their five siblings, and two doc comments lose the paragraph explaining a pointer that only existed for tests. That is a better trade than atomic.Pointer, which would have made every read site Load().Report(...) to keep a swap that no longer needs to happen. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Scan's doc comment overclaimed the self-heal. It said a later wild or lure sighting fills an unknown seen_type back in; neither does. updateFromWild's switch lists Unset, Cell and NearbyStop, updateFromNearby's lists Unset and Cell and returns on default, and updateFromMap only sets a seen type on a new record — so Unknown reaches no case in any of them. Only the encounter paths set a seen type without first asking what the current one is. That is correct behaviour, not a gap: a reader who made the documented claim true by adding `case SeenTypeCodeUnknown:` to updateFromWild would rebuild the exact downgrade the sentinel exists to prevent, so the comment now says which paths heal, which deliberately do not, and what fails if someone changes that. webhooks.md still typed seen_type as null.String, the one row the PR's earlier sweep of that table missed. It is a NullSeenType now, with a line on the wire form since the type is not one of the null.* family the conventions section covers. TestNullSeenTypeJSON's comment credited the API response. huma never reaches NullSeenType.MarshalJSON — ApiPokemonResult's seen_type is a *string built by Ptr(). The real consumer is the webhook payload, which webhooks/webhook.go encodes with stdlib encoding/json. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both tests counted log lines against a real one-second wall-clock window and demanded exactly 1. A runner loaded enough to spread 500 Scans (or four refused SetSeenType calls) across a window boundary opens a second window and logs twice — the throttle doing its job, reported as a failure. Rare, and a rare flake is the worst kind to debug. Assert what the tests are actually about instead: at least one line, and fewer than one per event. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every Go step here passes -tags go_json, matching the Dockerfile, and golangci-lint sets build-tags: [go_json] as well. Between them, no CI job compiles the !go_json configuration at all — including jsonenc.go and jsonenc_test.go, which exist only there. Nothing ships untagged, but `go build ./...` is what a contributor types, so a break in that half would land unnoticed and greet the next person to clone the repo. go vet ./... type-checks every package and its tests, which covers the compile without paying for a second full suite run. It needs no database, so it sits before the migration step and fails fast. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
On So please drop the refuse-to-overwrite guard as well ( |
|
Correction to my previous comment: on verification, Verification of the rest of the post-round-6 commits — all four asks landed correctly: the weather skip and entity guard now agree on "unchanged" for every input (NULL-weather case included); One deviation to accept explicitly: the throttle fix value-types the two globals and adds a small Residual nits, take or leave, none blocking: the SetSeenType throttle test's upper bound ( That closes the review from our side. |
TestNullSeenTypeUnmarshalNullIsInert called encoding/json.Marshal directly, bypassing the file's established convention of routing through jsonenc.Marshal so golden-JSON tests track whichever codec the current build tag actually selects (see jsonenc's package doc). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…nd headroom Its upper bound was got >= len(refused), pinned at exactly 4 — the refused-codes list's length. A runner preempted long enough to straddle a wall-clock second boundary between each of the 4 calls could open up to 4 separate throttle windows and log 4 warnings, hitting the bound exactly and flaking a passing run. Cycle the refused codes 125 times (500 total calls) instead, matching the order of magnitude its sibling TestScanUnknownSeenTypeWarnIsThrottled already drives (500 calls, asserting < 500), so the bound is proportional to what the test actually exercises rather than pinned to the length of a 4-element slice. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
webhooks.md said null is emitted "when unset". It is also emitted when this binary reads a seen_type string it does not recognise (a newer binary's value, seen during a rollback or a lagging replica in a mixed deployment) and stores the inert SeenTypeCodeUnknown sentinel instead — that sentinel serializes to null the same as unset. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Only tests call Reset today, and they quiesce production's Report callers first, but nothing said that was required. Document that callers must ensure no concurrent Report is in flight, since an interleaved Report could otherwise observe a cleared count paired with the old lastLog, or vice versa. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Shrinks the cached
Pokemonstruct from 800 bytes to 352, moving it from Go's 896-byte allocator size class to the 352-byte class — 544 fewer bytes per cached pokemon.PokemonLookup, the per-candidate scan struct, also drops 18 → 16 bytes. Measured on production instance under live traffic:String interning was split out to #395 at @jfberry's request, so it is no longer part of this change.
Why 512 bytes
Go's GC treats objects above 512 bytes materially worse. Measured across 5M live entries with forced GC, mean of 5 runs:
Pokemon)Dropping pointer count from 12 to 1 buys 3%. Adding 8 bytes across the 512 boundary costs 3.1x.
TestPokemonUnderGCThresholdguards the line.Approach
Nullable fields use
null.Value[T].guregu/null's non-generic types are 16 bytes each because they embedsql.NullInt64, and most nullable pokemon columns aretinyintorsmallint.null.Value[T]embeds stdlibsql.Null[T]and measures 2/4/8/16/8/2 bytes for uint8/uint16/uint32/uint64/float32/bool.Scanrange-rejects andencoding/jsonrendersfloat32at 32-bit precision, so both behaviours an earlier draft hand-rolled come from the library.cell_idandspawn_idarenull.Value[int64].sql.Null[uint64]cannot scan a negativeint64(cell_idis a signedbigintwhose real values are frequently negative), and itsValue()rejects anyuint64with the high bit set.Scan history is a plain Go struct.
Pokemonembeddedgrpc.PokemonInternal, a generated protobuf message used purely as in-memory Ditto state, carryingMessageState/SizeCache/UnknownFieldsat 64 bytes per pokemon and 88 per scan — and only marshalled behind four ANDed conditions with the controlling flag defaulting off. Now[]*pokemonScan, 24-byte header and 44 bytes per entry, with protobuf built on demand at the two DB boundaries. Elements stay behind pointers deliberately: the Ditto code holds a pointer to a history entry across other work and mutates through it, so a value slice would have&s[i]invalidated byappend.Fields ordered by descending alignment, and three deleted:
Capture1/2/3(setters with no callers, in neither the select columns nor the upsert),changedFields(const-folded dead in production builds, but its 24-byte header and one GC-scanned word stayed), andseen_typeas a string (an eight-value enum in a 24-bytenull.Stringplus a heap pointer, now a one-byte code).The debug change accumulator rolled out to every entity. Four crossed an allocator class —
Station,Spawnpoint,Incident,Tappable.Pokestop,Gym,RouteandPlayershed 24 bytes but stayed in the same class, so they save nothing at allocation time.What the profile shows
The first version of this description estimated the CPU saving at "low single-digit percent," sourced from
docs/decode-performance-findings.md. That 5.4% figure is from theprotobenchharness, not production, and was wrong by roughly 5x. @jfberry's "absolutely will need to be benchmarked" was the right call.Measured on
main: GC mark 24.22% of CPU, live heap 6.66 GB against ~14 GB RSS — so about half the process was collector headroom, withGOGCunset. Pokemon were 2.71 GB of that, 40.8%, at ~3.25M cached.On this branch, per-entity bytes match the predicted size-class ratios to three decimals:
Scan entries beat prediction because they also lost their protobuf machinery, not just a size class.
The RSS win outran the struct arithmetic by 1.6x. The model said 1.84 GB of live heap saved, doubled for
GOGC=100headroom, so ~3.7 GB of RSS. pm2 shows 5.75 GB. The difference is second-order and the model did not include it: fewer and more uniform objects mean less GC metadata and less span fragmentation, andHeapReleasedwent from 8 MB to 647 MB — the runtime is now returning pages where before it essentially was not.Caveats on the measurement
The two captures were taken at different points in the day.
mainat 19:17 (evening peak, 3.25M pokemon), this branch at 07:40 (morning trough, 2.74M). That gap is diurnal, not cache warmth. Adjusting for 0.51M more pokemon at 352 B with the GOGC doubling puts the branch near 9.6 GB at peak — still ~5.4 GB and 36% below baseline, but a capture at ~19:00 would settle it properly.The GC figure is two samples, 18.56% and 19.58%, against a baseline that was a single 30s sample. Read it as ~19% ± 0.5 and ~5 points saved, not 5.7.
The capture is ~25 commits behind. Every production number here was taken on this branch at
bca1669. Four review rounds have landed since, all of them correctness fixes rather than size changes, soPokemonis still 352 — but the figures have not been re-observed.Nothing has been measured on go 1.27, whose allocator changes could move all of this. Raised by @jfberry and still open.
Wire changes
API responses:
weight,heightandivrender with fewer digits —6.7rather than6.699999809265137. They originate as protobuffloatfields promoted throughfloat64(), so the extra digits were an artifact of that promotion. Verified byte-identical across 500,018 values through both the stdlib and goccy encoders.Webhook payloads: same fields, same change.
The OpenAPI schema changed on 19 fields, and separately on four timestamps — 16 integers
format: int64→int32, 3 floatsdouble→float. Worth calling out separately: narrowing to unsigned Go types also made huma emitminimum: 0on those 16 fields. That is a validation constraint rather than a format label, so a strict client or validating proxy would begin rejecting negatives. Accurate to the domain — none of those fields can legitimately be negative — but it is new contract surface beyond the format change, and it was not part of the original ask.TestApiPokemonResultSchemaWidthsnow pins the shape so a future width change cannot drift it silently.A later round then took the four timestamp fields the other way.
expire_timestampandupdatedhad becomeformat: int32, which overflows generated clients in 2038 — and inconsistently, sincefirst_seen_timestampstayedint64. All four are nowformat: int64withminimum: 0. Note the second half of that:first_seen_timestampandchangedgain a floor they never advertised, because huma had nothing to infer one from atint64. Accurate to the domain, but new constraint surface.seen_typeenum strings and thegolbat_internalprotobuf are unchanged.Review feedback
Round 1 —
nullalready supports generics (thedecoder/nulltypespackage an earlier draft added is deleted, −702 lines); pull theSeenTypeCodeWild = 0fix in (code 0 is now an explicitUnsetsentinel); the webhook re-widening is wasted (gone); the embedded protobuf is ripe for change (done, stacked here); rename the change list and roll it out (done).Round 2 — four asks, all addressed:
SetSeenTypetakes aSeenTypeCode, not a string. The constants are typed, so a typo now fails to compile instead of silently no-opping, and the decode path loses a map lookup.widenPtr/widenFloatPtrdeleted across 19 call sites by narrowing the response types.NullSeenType.Scandegrades instead of failing. An unrecognised enum value now warns and yields invalid rather than erroring, because an erroringScanstranded anewRecordcache entry that re-failed its DB load on every sighting, silently dropping all processing for that pokemon for the duration of a mixed deployment.Value()stays strict — writing an out-of-enum string to a MariaDB ENUM is silently stored as''.On #3, since it was introduced by this PR
Several sites compared a clamp-saturated stored value against a raw proto value. With a costume of 300, the stored value saturates at 255 and
255 != 300on every sighting, forever. The branch that guards nulls out weight, height, size, moves, cp, shiny, ditto and pvp — so every encounter's enrichment was undone by the next sighting, fleet-wide.An earlier review on this branch flagged it as Minor and "not reachable with today's proto values," and it shipped on that basis. That was wrong twice over:
pogoenums are openint32, so unknown and negative wire values pass straight through, and Golbat ingests raw protos from third-party scanners — a malformed packet is enough.Fixed at 7 sites across 5 functions (3 of which the ask did not name) with non-counting
narrowUint8/16/32helpers on comparison sides, the countingclampUint*delegating to them sogolbat_field_clamped_totalstill fires exactly once per real store. Regression tests cover each site individually — reverting one fails only its own test.Round 4 — four asks plus a ranked list, all addressed. Two of the four turned out deeper than stated:
seen_typewas degrading toUnset, which is an active state.updateFromWilddowngraded the record,updateFromNearbyreplaced precise coordinates with cell centres, and the save wrote the damage back over a newer binary's value. Pre-PRmainstored the string opaquely and every switch fell through harmlessly, so the degrade shipped worse than what it replaced — in exactly the mixed-deployment case it was built for. Now a distinctunknowncode, inert at all 17 switch sites, withCOALESCEon both write paths so it refuses to overwrite.calculateIvcomputedIvfrom the raw IV sum while storing clamped values.ivisfloat(5,2) unsigned, so a sum above 450 produces a value MariaDB rejects underSTRICT_TRANS_TABLES— failing the entire multi-row batch upsert. Narrowing the inputs as specified did not close it, because the clamp used the column's 255 rather than the game's 15. Clamping at 15 does, and subsumes it: sum ≤ 45 meansIv≤ 100.Chasing that second one surfaced a third bug nobody had connected:
PokemonLookup'sint8fields use-1for "absent", andint8(255)is-1. A clamped value entered the scan index as the sentinel for never encountered, silently changing which DNF filters matched. Every lookup field now saturates at a sentinel-safe ceiling instead of truncating — covering the whole ≥128 band, not just the exact ceiling — andFormgets its own helper because its-1is the wildcard-form key rather than an absence marker.Atk/Def/Sta/Ivwere reachable from rows already in the database, since a write-side clamp cannot heal stored data.CI also now runs under the production build tag. It was running
go test ./...without-tags go_json, so every golden-JSON proof in this PR was verified against stdlib while production serves through goccy. Nothing failed once corrected — goccy matches byte-for-byte across every fixture and adversarial case tested — but the proofs were unenforced against the shipping codec. A MariaDB service container now makes the round-trip tests actually execute rather than skip.Rounds 5 and 6 — @Mygod flagged that the protobuf-to-plain-struct conversion drops unknown fields when an older binary rewrites
golbat_internal. A guard was built to refuse the overwrite; @jfberry ruled it an overreach — the field is transitory, the flag defaults off, and losing fidelity there is harmless — so it was reverted. Four smaller asks landed instead: the proactive-IV skip now compares weather in one encoding rather than testing the lookup's-1against the entity's0;NullSeenType.UnmarshalJSONyields the inertUnknownsentinel rather than the destructiveUnset; the warn throttles are reset by tests instead of swapped, closing a latent race; and three stale doc claims are corrected. CI also now vets the untagged build, which nothing else covered.Testing
pokemonBatchUpsertQuery: a row with every nullable column NULL, and a fully-populated row covering the full 64-bitcell_id,spawn_id, and float64 lat/lon precision.golbat_internal: a hand-written byte literal in the oldgrpc.PokemonInternalshape decodes through the new path, and the new write path produces bytes byte-identical to it.TestPokemonScanCoversEveryProtoFieldreflects over both structs and diffs exported field-name sets, so adding a proto field without mapping it fails loudly.TestPokemonEntitySizespins the sizes;TestPokemonUnderGCThresholdguards the 512 line.-race,golangci-lint, and the full suite.statsCollectorbecame anatomic.Pointerseeded with a noop at package init. A test needed to swap it, anddecoder/init_test.goalready documented that doing so races the stats aggregation worker, which reads the global on a ticker for the life of the process. This also retires a pre-existing race instation_battle_test.go.Follow-ups
ProactiveIVSwitch's boosted-weather guard never fires — pre-existing onmain, confirmed by @Mygod.boostedWeathers&uint8(1)<<w != 0parses as(boostedWeathers & 1) << w, since Go gives&and<<equal precedence. EveryboostedWeatherLookupentry is even, so the guard is unconditionally false andnewWeatheris always 0 — the switch can only remove boosts, never apply them, and a currently-boosted pokemon whose new weather still boosts it gets wrongly de-boosted. Separately,NewWeatherarrives as an unvalidatedint32from an open proto3 enum, and a negative shift count panics in a goroutine with norecoverin its chain, so one malformed packet takes the process down. Both belong in their own PR, since the fix changes live behaviour immediately.The same negative-aliasing pattern is still live in
fortRtree.go— bareint8/int16casts on quest reward amounts, team and slots. ThelookupInt8/lookupInt16helpers added here apply verbatim.clampUint's derived ceiling would silently wrap for a future~uint64instantiation; alimit < 0panic guard is cheap.The
statsCollectorSetordering guard doesn't cover the exportedInitTypedQueues.TestUpdatePokemonLookupSaturatesClampedFieldsleaks a phantom entry intopokemonFormCount.The publish workflow's build-gate comment overclaims (branch pushes still publish per-branch images; the load-bearing case is fork-PR tokens), and tag-push releases now block on MariaDB service health.
Bit-packing the numeric fields. Deliberately not done here. From 312 bytes, packing the small numerics into shared words reaches roughly 241 (the 256 class); adding a database row shim so
PvpandGolbatInternalcan leave the struct reaches roughly 193 (the 208 class) — another ~112 bytes per pokemon. It has to be a build tag rather than a config flag, because Go fixes struct offsets at compile time and a runtime branch keeps the struct at the larger size, saving nothing. The cost is a build matrix of 4+ combinations and converting ~250 direct field reads to accessors in both layouts, so the readable build stops being readable. Worth its own PR against this measured baseline.PvpandGolbatInternalwere assessed and left alone. Two measured negative results, both now pinned inentity_sizes_test.go:Pvpis not dead between its write and its zeroing (the queue snapshot, the direct-write fallback and the webhook builder all read it), and narrowing it alone buys exactly zero allocated bytes, because 296 still lands in the 320 class and the next class down is 288.The intern table wants an alert threshold, not just a dashboard line. The length cap bounds entry size, not table count; a caller sending a million distinct well-formed usernames still grows it.
golbat_intern_table_sizeis the instrument.A clamped value now lands exactly on
PokemonLookup's missing-value sentinel —int8(255) = -1, andint16(65535) = -1forCpandForm. That aliases "clamped" with "unknown" in scan filters, which the pre-narrowingint8(300) = 44did not. Narrow, latent, introduced here.Off-heap/arena storage for
PokemonDatadeserves another look. It was ruled out earlier using ~5% for GC's share of CPU; at a measured 24.22% that arithmetic does not hold.Pokestop(1152 B) andGym(968 B) are both far over the 512-byte threshold.Design:
docs/superpowers/specs/2026-08-16-pokemon-struct-packing-design.md🤖 Generated with Claude Code