Skip to content

fix(lambda): repair the metrics catalog and resolve each metric's statistic - #33

Merged
agustincelentano merged 2 commits into
mainfrom
feat/metric-statistics-and-stream-age
Aug 3, 2026
Merged

fix(lambda): repair the metrics catalog and resolve each metric's statistic#33
agustincelentano merged 2 commits into
mainfrom
feat/metric-statistics-and-stream-age

Conversation

@agustincelentano

Copy link
Copy Markdown
Contributor

Fixes the performance screen for Lambda scopes, which showed container metrics —
CPU, memory, throughput — that a Lambda cannot answer, and got them all wrong
where it did answer.

Why the screen showed the wrong graphs

The telemetry workflows hand their stdout to the platform as the response body,
and assume_role wrote its progress there. The catalog arrived behind two lines
of text:

   Assuming role: arn:aws:iam::...:role/..._lambda_role
   Role assumed successfully
{ "results": [ ... ] }

The platform could not parse that, so GET /metric answered
400 "Error fetching data from external provider". Left without a catalog, the UI
falls back to the container metric set, and each of those returned 400 as well.
That fallback was the visible symptom, not the cause.

instance/build_context printed four diagnostic lines the same way and broke
instance:data.

Only _ar_log changes, including the branch that delegates to the pretty logger.
utils/log keeps writing to stdout: the deployment workflows show that output as
progress, and silencing it there would leave deploys mute.

Why the numbers were wrong

fetch_metric read METRIC_STATISTIC, and nothing ever set it — neither
build_context nor the workflow — so every metric fell back to Sum. Three of
them are counts and were fine; the other two reported something their name does
not describe. Measured on a live function:

Metric Was Is
Duration 1076 ms (sum of durations) 2 ms (average)
ConcurrentExecutions 205 (sum of samples) 1 (maximum)

Duration as a sum grows with traffic even when the function gets faster, so a
rising graph read as degradation when nothing had degraded.

The statistic now comes from the metric name, the way scopes/k8s/metric/metric
resolves type and unit: it belongs to the metric, not to the request.

What else changes

A failed or empty query returns the response with an empty series instead of
exiting non-zero, so a graph with no data stays a graph with no data instead of
surfacing a provider error.

Adds stream.iterator_age, the one metric that says whether the consumer of a
DynamoDB stream is falling behind. The others describe the function's health, and
it can be healthy while the stream piles up against the 24 hour retention. The
prefix puts it in its own section, following how scheduled_task namespaces its
cronjob.* metrics.

Verified

Against a live function, with the agent pointed at this branch:

  • GET /metric returns the six metrics with their titles and units
  • Duration returns ~2 ms over 60 datapoints
  • stream.iterator_age returns 2583 ms after writing to the linked table, and an
    empty series when the function is only serving HTTP — CloudWatch does not
    publish it unless an event source mapping invoked the function
  • an unknown metric name returns unit: unknown without failing
  • the performance screen shows the six graphs and no longer shows the container set

🤖 Generated with Claude Code

…terator age

fetch_metric read METRIC_STATISTIC, and nothing ever set it: build_context does
not export it and the workflow does not declare it, so every metric fell back to
Sum. Three of them are counts and were fine; the other two reported numbers that
do not mean what their name says. Measured on a live function: Duration read
1076 ms where the latency was 5.25 ms, and ConcurrentExecutions read 205 where
the peak was 1.

The statistic now comes from the metric name, the way the containers scope
resolves type and unit in scopes/k8s/metric/metric — it belongs to the metric,
not to the request.

A failed or empty query now returns the response with an empty series instead of
exiting non-zero. Reaching the platform as an error turned a graph with no data
into "Error fetching data from external provider".

Adds stream.iterator_age, which is the one metric that says whether the consumer
of a DynamoDB stream is falling behind. The others describe the function's
health, and it can be healthy while the stream piles up behind it.
The telemetry workflows hand their stdout to the platform as the response body,
and assume_role wrote its progress there. The catalog arrived behind two lines of
text:

       Assuming role: arn:aws:iam::...:role/..._lambda_role
       Role assumed successfully
    { "results": [ ... ] }

The platform could not parse that, so GET /metric answered 400 "Error fetching
data from external provider", the UI was left without a catalog and fell back to
the container metrics — CPU, memory, throughput — none of which a Lambda scope
can answer. Every one of those returned 400 too.

Only _ar_log changes, including the branch that delegates to the pretty logger.
utils/log keeps writing to stdout because the deployment workflows show that
output as progress, and silencing it there would leave deploys mute.

instance/build_context printed four diagnostic lines to stdout and broke
instance:data the same way.
@agustincelentano
agustincelentano merged commit a4642cc into main Aug 3, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants