Fix geofence queries: bind the fence, and test containment in Go - #400
Fix geofence queries: bind the fence, and test containment in Go#400TurtIeSocks wants to merge 2 commits into
Conversation
The three geofence queries built their statement by concatenating the
fence's JSON into it:
"AND ST_CONTAINS(ST_GeomFromGeoJSON('"+string(bytes)+"', 2, 0), ...)"
The fence comes from a request body, and geojson.Feature round-trips its
whole properties map through MarshalJSON, so a property value containing
a single quote closes the string literal it lands in. That is SQL
injection on /api/quest-status, /api/pokestop-positions and
/api/clear-quests. All three require authentication, so it is not
reachable anonymously.
Move the predicate into the FenceContainsPredicate constant, which binds
the fence with a placeholder, and add FenceQueryArgs to assemble the
bounding-box corners and the fence JSON in the order every one of these
queries binds them. The statement text no longer varies with the fence,
so there is no literal for its contents to escape.
No change to which rows match: the same predicates run in the same
order, with the fence passed as data instead of as text.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
I don't think this actually moves the fence build out of the command (as per my message on the other issue). |
Asking MariaDB to evaluate ST_CONTAINS for every candidate row is what
made large geofences time out. Measured on MariaDB 11.8 with 1,000,000
pokestops and a 2000-vertex fence whose bounding box held 628,560
candidates:
current query, fence inlined 55.0 s
SET @Fence, then ST_CONTAINS(@Fence) 57.6 s
the same via a CTE 57.8 s
bounding-box scan, all rows read 0.9 s
the same 628,560 points in Go 3.3 s
Hoisting the GeoJSON parse out of the row loop changes nothing, which is
what identifies per-row containment rather than repeated parsing as the
cost. So the fix is to stop asking SQL for containment: select candidates
by bounding box, which the ix_coords index already serves, and test them
against a compiled fence while the rows stream past.
geo.CompileFence is the same matcher MatchGeofences already uses for
stats and webhook area attribution, and is differentially tested against
orb. Only matching rows are retained, so nothing materialises the full
candidate set.
Fences that are not polygons keep the SQL predicate, so exotic geometries
behave exactly as before.
One deliberate behaviour change: CompileFence counts the polygon boundary
as inside, where ST_CONTAINS follows OGC and treats boundary points as
outside. A fort exactly on a fence edge is now attributed the same way
here as it already is in stats and webhooks.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Split out of #398 so it can land on its own. Two separate things in here. First, the fence was being concatenated into the SQL text rather than bound. A geofence body round-trips its whole Second, the timeout, with the numbers in the description. Short version is that hoisting the GeoJSON parse out of the row loop changes nothing, so the cost is per row @jfberry does this shape look right to you? Happy to split the two commits into separate PRs if you'd rather take the injection fix on its own first. |
Problem
The three geofence queries built their statement by concatenating the fence's JSON into it:
The fence arrives as a request body, and
geojson.Featureround-trips its entirepropertiesmap throughMarshalJSON. A property value is free-form text, so anything a caller puts there lands verbatim inside that quoted literal, including a quote of its own./api/quest-status,/api/pokestop-positionsand/api/clear-questsare all affected, along with the ginNormaliseFenceRequestpath. All three require authentication, so this is not reachable anonymously.Approach
FenceContainsPredicateis now a constant carrying a bind placeholder, andFenceQueryArgsassembles the bounding-box corners and the fence JSON in the order every one of these queries binds them. The statement text no longer varies with the fence, so its contents have no literal to escape.Which rows match does not change. The same predicates run in the same order against the same values; only the transport differs, with the fence passed as data rather than as text. Each query keeps its own comparison operators, including the
>/<in quest-status against the>=/<=in the other two.The timeout, measured
I had assumed repeated GeoJSON parsing was the cost. That was wrong, and the measurement says so plainly. MariaDB 11.8, 1,000,000 pokestops with
ix_coords(lat, lon), a 2000-vertex fence (83 KB of GeoJSON) whose bounding box holds 628,560 candidates:SET @fence = ST_GeomFromGeoJSON(...), thenST_CONTAINS(@fence, ...)WITH ... AS MATERIALIZEDHoisting the parse out of the row loop changes nothing, and that is the result that matters: it rules out repeated parsing and leaves per-row
ST_CONTAINSagainst a many-vertex polygon as the cost. Neither variant suggested in #398 moves the number, and one of them does not parse on MariaDB at all, sinceAS MATERIALIZEDis PostgreSQL syntax.So containment moves to Go. Candidates come from the bounding box, which the existing index already serves, and each one is tested against
geo.CompileFenceas the rows stream past, retaining only matches. That is the same matcherMatchGeofencesalready uses for stats and webhook area attribution, and it is differentially tested against orb.Two consequences worth stating. Fences that are not polygons keep the SQL predicate, so exotic geometries are untouched. And
CompileFencecounts the polygon boundary as inside whileST_CONTAINSfollows OGC and treats it as outside, so a fort exactly on a fence edge is now attributed the same way here as it already is everywhere else.These are synthetic numbers on uniform random data in a container, not a production table, so treat the ratio as the finding and not the absolute times.
One note for anyone revisiting the
SET @fenceform regardless: user variables are connection-scoped and the handle is pooled (SetMaxOpenConnsinmain.go). These call sites useSelect,GetandSelectContext, which each take an arbitrary connection, so theSETand theSELECTcan land on different ones.@fencewould then be NULL,ST_Contains(NULL, ...)would be NULL, and every row would filter out: zero results, no error.Testing
Six tests in
db/geofence_query_test.go. Three for the injection fix: the predicate keeps its placeholder and carries no literal to break out of, a hostilepropertiesvalue survives only as a bound argument, and the bounding-box corner order the callers depend on. Three for the containment move: non-polygon fences fall back to SQL while polygons and multipolygons do not, the matcher's lat/lon argument order is right (the fences used have differing lat and lon spans, so a swap is visible), and boundary points count as inside.go build -tags go_json ./...andgolangci-lint runare clean, and thedb,decoderand root suites all pass.🤖 Generated with Claude Code