Skip to content

feat(metrics): split L7 event counter by HTTP/2 frame direction #318

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
blue4209211 merged 2 commits into from
Sep 5, 2026
Merged

feat(metrics): split L7 event counter by HTTP/2 frame direction #318

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
75ae506
feat(metrics): split L7 event counter by HTTP/2 frame direction
mayankpande88 Sep 4, 2026
2cacf1e
Merge branch 'main' into diag/http2-frame-direction
mayankpande88 Sep 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion containers/container.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -902,6 +902,18 @@ func (c *Container) onL7Request(pid uint32, fd uint64, timestamp uint64, r *l7.R
return ip2fqdn
}

// frameDirection labels an HTTP/2 event by which side's frames it carries.
// Other protocols report "-" rather than inventing a direction for them.
func frameDirection(m l7.Method) string {
switch m {
case l7.MethodHttp2ClientFrames:
return "client"
case l7.MethodHttp2ServerFrames:
return "server"
}
return "-"
}

// protocolLabel renders a protocol for use as a metric label. Protocol.String()
// falls back to "UNKNOWN:<n>" for unrecognised values, which would be unbounded
// cardinality on a label, so those collapse to a single bucket.
Expand Down Expand Up @@ -1065,7 +1077,7 @@ func (c *Container) onL7RequestWithResult(pid uint32, fd uint64, timestamp uint6
destClass = "external"
}
proto := protocolLabel(r.Protocol)
L7EventsTotal.WithLabelValues(proto, destClass).Inc()
L7EventsTotal.WithLabelValues(proto, destClass, frameDirection(r.Method)).Inc()
if r.PayloadSize > uint64(len(r.Payload)) {
L7PayloadTruncatedTotal.WithLabelValues(proto, destClass).Inc()
}
Expand Down
12 changes: 10 additions & 2 deletions containers/llm_metrics.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,12 +136,20 @@ var (
// rate(node_agent_l7_payload_truncated_total{protocol="http2",destination="external"}[5m])
// against the same labels on node_agent_l7_events_total to see what share of
// external HTTP/2 traffic is arriving incomplete.
// direction is "client"/"server" for HTTP/2 (which frames the event carries)
// and "-" for protocols where the distinction does not apply.
//
// External HTTP/2 delivers ~22,000 events per stream created, against ~105
// internally. Splitting by direction separates the two explanations for
// that: if server-frame events are scarce, responses never reach the parser;
// if they are plentiful, the bytes being fed to it are not HTTP/2 at all and
// the port-based detection heuristic is over-matching.
L7EventsTotal = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "node_agent_l7_events_total",
Help: "L7 events processed, by protocol and destination class",
Help: "L7 events processed, by protocol, destination class and frame direction",
},
[]string{"protocol", "destination"},
[]string{"protocol", "destination", "direction"},
)

L7PayloadTruncatedTotal = prometheus.NewCounterVec(
Expand Down
Loading