Skip to content

feat(metrics): split L7 event counter by HTTP/2 frame direction - #318

Merged
blue4209211 merged 2 commits into
mainfrom
diag/http2-frame-direction
Sep 5, 2026
Merged

feat(metrics): split L7 event counter by HTTP/2 frame direction#318
blue4209211 merged 2 commits into
mainfrom
diag/http2-frame-direction

Conversation

@mayankpande88

Copy link
Copy Markdown
Contributor

What #317 established

The pipeline counters localised the failure. External HTTP/2, 10 minutes on dev at rc.5:

Stage external internal (control)
L7 events 1,155,643 844,250
stream_created 53 8,052
response_status 0 5,478
end_stream 0 5,159
completed 0 5,158
hpack_error 451 20,824

Ruled out and needing no further attention: parser cap drops were 0, stale fd reuse was 3 externally in 10 minutes. Both candidates #317 was originally written for are dead.

Two things stand out:

  1. ~22,000 external events per stream created, against ~105 internally — a 200× difference.
  2. response_status is exactly zero externally. Zero is the load-bearing detail: :status decodes from the HPACK static table (index 8), which survives a degraded decoder, and the internal control produces 5,478 of them on the same code path. Server-side HEADERS are never decoded externally at all.

What this PR separates

Two explanations remain for that ratio, and frame direction distinguishes them in one deploy:

Observation Meaning
server-frame events scarce responses never reach the parser
server-frame events plentiful the bytes are not HTTP/2 — the eBPF port-based detection (is_likely_http2_port, 443/8443, plus a frame-shape check) is over-matching TLS traffic

The second would recast node_agent_hpack_decode_errors_total (6.28M/12h) as a symptom of feeding non-HTTP/2 bytes to an HPACK decoder, rather than a cause — which would also explain why fixing genuine HPACK/truncation defects in #316 reduced the error count without restoring any decoding.

Adds a direction label (client/server, - where inapplicable) to node_agent_l7_events_total.

Status

Measurement only, no behaviour change. gofmt, GOOS=linux gopls check, full CI suite pass.

Stated plainly: four diagnoses have already been wrong here (mid-stream join, attach ordering, payload truncation, parser lifecycle). This PR does not assert a cause — it is the single cheapest measurement that discriminates between the two remaining ones, and either outcome is informative.

The pipeline counters from #317 localised the failure but not its cause.
External HTTP/2, 10 minutes on dev:

  L7 events        1,155,643        internal control:   844,250
  stream_created          53                              8,052
  response_status          0                              5,478
  completed                0                              5,158

That is ~22,000 external events per stream created, against ~105 internally —
a 200x difference — and response_status is exactly zero. Zero matters because
:status decodes from the HPACK static table (index 8), which survives a
degraded decoder; the internal control produces 5,478 of them on the same code
path. Server-side HEADERS are therefore never decoded externally at all.

Also ruled out by that run, so they need no further attention: parser cap drops
were 0, and stale fd reuse was 3 events externally in 10 minutes.

Two explanations remain for the ratio, and direction separates them cleanly:

  server-frame events scarce    -> responses never reach the parser
  server-frame events plentiful -> the bytes are not HTTP/2 and the eBPF
                                   port-based detection (is_likely_http2_port,
                                   443/8443, plus a frame-shape check) is
                                   over-matching TLS traffic

The second would also recast node_agent_hpack_decode_errors_total as a symptom
of feeding non-HTTP/2 bytes to an HPACK decoder rather than a cause.

Adds a direction label ("client"/"server", "-" where inapplicable) to
node_agent_l7_events_total. Measurement only.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new 'direction' label to the L7EventsTotal Prometheus metric to distinguish between client and server frames in HTTP/2 traffic, while using a default '-' value for other protocols. This is achieved by adding a helper function frameDirection in containers/container.go and updating the metric definition in containers/llm_metrics.go. I have no feedback to provide as the changes are clean and well-documented.

@blue4209211
blue4209211 merged commit 71fcb4c into main Sep 5, 2026
8 checks passed
@blue4209211
blue4209211 deleted the diag/http2-frame-direction branch September 5, 2026 06:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants