Skip to content

fix(metrics): stop reclassified connections from polluting HTTP metrics - #323

Merged
blue4209211 merged 2 commits into
mainfrom
fix/reclassified-connections-pollute-metrics
Sep 6, 2026
Merged

fix(metrics): stop reclassified connections from polluting HTTP metrics#323
blue4209211 merged 2 commits into
mainfrom
fix/reclassified-connections-pollute-metrics

Conversation

@mayankpande88

Copy link
Copy Markdown
Contributor

The bug

A connection the parser gave up on kept being reported as an HTTP request for the rest of its life.

trackParseFail sets conn.protocolOverride = protocolReclassified (0xFF) once a connection has failed to parse parseFailThreshold times consecutively — the agent concluding eBPF misidentified it. 0xFF matches none of the 14 protocol cases, so every later event fell into default::

observe(r.Protocol, "unknown", "", "", 0, ...)

Two problems in that one call:

  1. It still emits. Reclassification means we established the protocol is not what we thought. Continuing to report is worse than reporting nothing.
  2. It passes r.Protocol, not the override — the original, wrong protocol. observe() maps HTTP2 → HTTP, so a Postgres connection misdetected as HTTP/2 incremented container_http_requests_total on every event, with a literal "unknown" status and zero duration.

Measured on dev

External "HTTP requests" 1.24M/hour
With status="unknown" 99.7%
From 3 private IPs (Postgres + egress proxy) 97%

Real external HTTP traffic was outnumbered roughly 400:1 — GitHub's 2,951 requests against 1.2M of noise. That is why status codes looked unusable in aggregate, and why external HTTP request counts were meaningless.

Why traces were unaffected

Trace data quality was near-perfect over the same window — 1 bad row in 31,900 spans — because default: only calls observe() and never creates a span. The pollution was confined to metrics. That discrepancy is what led to the cause.

Fix

Reclassified connections emit nothing. Genuinely unhandled protocols still reach default: and are counted under their own protocol, which is correct there precisely because it is not the sentinel.

Testing

No unit test. The containers package is excluded from CI (go test $(go list ./... | grep -v '/containers$')) and cannot be built on macOS, so there is nowhere for one to run. gofmt and GOOS=linux gopls check clean; full CI suite passes.

Verification is the deployed metric: container_http_requests_total{status="unknown"} should collapse from 1.24M/hr to near zero, and external HTTP counts should become dominated by real traffic.

Note on scope

This is a reporting bug, not a detection bug. The reclassification machinery was working correctly — it identified these connections as misdetected. It just did not stop them being counted.

A connection the parser gave up on kept being reported as an HTTP request for
the rest of its life.

trackParseFail sets conn.protocolOverride = protocolReclassified (0xFF) once a
connection has failed to parse parseFailThreshold times in a row, which is the
agent concluding eBPF misidentified it. 0xFF matches none of the 14 protocol
cases, so every later event fell into default:, which called

    observe(r.Protocol, "unknown", "", "", 0, ...)

passing r.Protocol — the original, wrong protocol — rather than the override.
observe() maps HTTP2 to HTTP, so a Postgres connection misdetected as HTTP/2
incremented container_http_requests_total on every subsequent event, with a
literal "unknown" status and zero duration.

Measured on dev before the change: 1.24M external "HTTP requests" per hour,
99.7% of them status="unknown", with 97% originating from three private IPs
that are actually Postgres and an egress proxy. Genuine external HTTP traffic
was outnumbered roughly 400:1 — GitHub's 2,951 requests against 1.2M of noise —
which is why status codes looked unusable in aggregate.

Reclassified connections now emit nothing. Giving up on a connection has to mean
giving up on reporting it. Genuinely unhandled protocols still reach default:
and are counted under their own protocol, which is correct there because it is
not the sentinel.

This also explains why traces were unaffected (1 bad row in 31,900 spans while
metrics were 99.7% wrong): default: only calls observe() and never creates a
span, so the pollution was confined to the metrics pipeline.

No test: the containers package is excluded from CI and cannot be built on
macOS. Verification is the deployed metric — container_http_requests_total
{status="unknown"} should collapse from 1.24M/hr to near zero.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a protocolReclassified case in onL7RequestWithResult to suppress reporting stats for connections where the protocol was misidentified, preventing skewed metrics. The reviewer points out that when a connection is reclassified, associated parsers are not cleaned up, which could lead to memory leaks on long-lived connections, and provides a code suggestion to clean them up in trackParseFail.

Comment thread containers/container.go
Review flagged that reclassification stops parsing but does not free the
parsers. Confirmed: googleHTTP2Parsers entries are only deleted once the
connection is gone or the pid dies, and conn.http2Parser / postgresParser /
mysqlParser live as long as the connection does.

That is dead weight from the moment we reclassify — the connection will never be
parsed again, yet the HTTP/2 parser keeps its HPACK decoders and partial-frame
buffers, and the SQL parsers keep their prepared-statement state. It also lands
on exactly the wrong connections: the ones reaching reclassification are the
long-lived, high-volume Postgres and egress-proxy connections, so 'until the
connection closes' can be hours.

Predates this branch — the switch already skipped parsing for reclassified
connections — but reclassification is the correct point to release, and this PR
is what makes it unambiguous that nothing will use them again.
@blue4209211
blue4209211 merged commit f636d64 into main Sep 6, 2026
7 checks passed
@blue4209211
blue4209211 deleted the fix/reclassified-connections-pollute-metrics branch September 6, 2026 07:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants