fix(metrics): stop reclassified connections from polluting HTTP metrics - #323
Merged
Merged
Conversation
A connection the parser gave up on kept being reported as an HTTP request for
the rest of its life.
trackParseFail sets conn.protocolOverride = protocolReclassified (0xFF) once a
connection has failed to parse parseFailThreshold times in a row, which is the
agent concluding eBPF misidentified it. 0xFF matches none of the 14 protocol
cases, so every later event fell into default:, which called
observe(r.Protocol, "unknown", "", "", 0, ...)
passing r.Protocol — the original, wrong protocol — rather than the override.
observe() maps HTTP2 to HTTP, so a Postgres connection misdetected as HTTP/2
incremented container_http_requests_total on every subsequent event, with a
literal "unknown" status and zero duration.
Measured on dev before the change: 1.24M external "HTTP requests" per hour,
99.7% of them status="unknown", with 97% originating from three private IPs
that are actually Postgres and an egress proxy. Genuine external HTTP traffic
was outnumbered roughly 400:1 — GitHub's 2,951 requests against 1.2M of noise —
which is why status codes looked unusable in aggregate.
Reclassified connections now emit nothing. Giving up on a connection has to mean
giving up on reporting it. Genuinely unhandled protocols still reach default:
and are counted under their own protocol, which is correct there because it is
not the sentinel.
This also explains why traces were unaffected (1 bad row in 31,900 spans while
metrics were 99.7% wrong): default: only calls observe() and never creates a
span, so the pollution was confined to the metrics pipeline.
No test: the containers package is excluded from CI and cannot be built on
macOS. Verification is the deployed metric — container_http_requests_total
{status="unknown"} should collapse from 1.24M/hr to near zero.
There was a problem hiding this comment.
Code Review
This pull request introduces a protocolReclassified case in onL7RequestWithResult to suppress reporting stats for connections where the protocol was misidentified, preventing skewed metrics. The reviewer points out that when a connection is reclassified, associated parsers are not cleaned up, which could lead to memory leaks on long-lived connections, and provides a code suggestion to clean them up in trackParseFail.
Review flagged that reclassification stops parsing but does not free the parsers. Confirmed: googleHTTP2Parsers entries are only deleted once the connection is gone or the pid dies, and conn.http2Parser / postgresParser / mysqlParser live as long as the connection does. That is dead weight from the moment we reclassify — the connection will never be parsed again, yet the HTTP/2 parser keeps its HPACK decoders and partial-frame buffers, and the SQL parsers keep their prepared-statement state. It also lands on exactly the wrong connections: the ones reaching reclassification are the long-lived, high-volume Postgres and egress-proxy connections, so 'until the connection closes' can be hours. Predates this branch — the switch already skipped parsing for reclassified connections — but reclassification is the correct point to release, and this PR is what makes it unambiguous that nothing will use them again.
blue4209211
approved these changes
Sep 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
A connection the parser gave up on kept being reported as an HTTP request for the rest of its life.
trackParseFailsetsconn.protocolOverride = protocolReclassified(0xFF) once a connection has failed to parseparseFailThresholdtimes consecutively — the agent concluding eBPF misidentified it.0xFFmatches none of the 14 protocol cases, so every later event fell intodefault::Two problems in that one call:
r.Protocol, not the override — the original, wrong protocol.observe()maps HTTP2 → HTTP, so a Postgres connection misdetected as HTTP/2 incrementedcontainer_http_requests_totalon every event, with a literal"unknown"status and zero duration.Measured on dev
status="unknown"Real external HTTP traffic was outnumbered roughly 400:1 — GitHub's 2,951 requests against 1.2M of noise. That is why status codes looked unusable in aggregate, and why external HTTP request counts were meaningless.
Why traces were unaffected
Trace data quality was near-perfect over the same window — 1 bad row in 31,900 spans — because
default:only callsobserve()and never creates a span. The pollution was confined to metrics. That discrepancy is what led to the cause.Fix
Reclassified connections emit nothing. Genuinely unhandled protocols still reach
default:and are counted under their own protocol, which is correct there precisely because it is not the sentinel.Testing
No unit test. The
containerspackage is excluded from CI (go test $(go list ./... | grep -v '/containers$')) and cannot be built on macOS, so there is nowhere for one to run.gofmtandGOOS=linux gopls checkclean; full CI suite passes.Verification is the deployed metric:
container_http_requests_total{status="unknown"}should collapse from 1.24M/hr to near zero, and external HTTP counts should become dominated by real traffic.Note on scope
This is a reporting bug, not a detection bug. The reclassification machinery was working correctly — it identified these connections as misdetected. It just did not stop them being counted.