fix(lambda): repair the metrics catalog and resolve each metric's statistic - #33
Merged
Merged
Conversation
…terator age fetch_metric read METRIC_STATISTIC, and nothing ever set it: build_context does not export it and the workflow does not declare it, so every metric fell back to Sum. Three of them are counts and were fine; the other two reported numbers that do not mean what their name says. Measured on a live function: Duration read 1076 ms where the latency was 5.25 ms, and ConcurrentExecutions read 205 where the peak was 1. The statistic now comes from the metric name, the way the containers scope resolves type and unit in scopes/k8s/metric/metric — it belongs to the metric, not to the request. A failed or empty query now returns the response with an empty series instead of exiting non-zero. Reaching the platform as an error turned a graph with no data into "Error fetching data from external provider". Adds stream.iterator_age, which is the one metric that says whether the consumer of a DynamoDB stream is falling behind. The others describe the function's health, and it can be healthy while the stream piles up behind it.
The telemetry workflows hand their stdout to the platform as the response body,
and assume_role wrote its progress there. The catalog arrived behind two lines of
text:
Assuming role: arn:aws:iam::...:role/..._lambda_role
Role assumed successfully
{ "results": [ ... ] }
The platform could not parse that, so GET /metric answered 400 "Error fetching
data from external provider", the UI was left without a catalog and fell back to
the container metrics — CPU, memory, throughput — none of which a Lambda scope
can answer. Every one of those returned 400 too.
Only _ar_log changes, including the branch that delegates to the pretty logger.
utils/log keeps writing to stdout because the deployment workflows show that
output as progress, and silencing it there would leave deploys mute.
instance/build_context printed four diagnostic lines to stdout and broke
instance:data the same way.
fedemaleh
approved these changes
Aug 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the performance screen for Lambda scopes, which showed container metrics —
CPU, memory, throughput — that a Lambda cannot answer, and got them all wrong
where it did answer.
Why the screen showed the wrong graphs
The telemetry workflows hand their stdout to the platform as the response body,
and
assume_rolewrote its progress there. The catalog arrived behind two linesof text:
The platform could not parse that, so
GET /metricanswered400 "Error fetching data from external provider". Left without a catalog, the UIfalls back to the container metric set, and each of those returned 400 as well.
That fallback was the visible symptom, not the cause.
instance/build_contextprinted four diagnostic lines the same way and brokeinstance:data.Only
_ar_logchanges, including the branch that delegates to the pretty logger.utils/logkeeps writing to stdout: the deployment workflows show that output asprogress, and silencing it there would leave deploys mute.
Why the numbers were wrong
fetch_metricreadMETRIC_STATISTIC, and nothing ever set it — neitherbuild_contextnor the workflow — so every metric fell back toSum. Three ofthem are counts and were fine; the other two reported something their name does
not describe. Measured on a live function:
DurationConcurrentExecutionsDurationas a sum grows with traffic even when the function gets faster, so arising graph read as degradation when nothing had degraded.
The statistic now comes from the metric name, the way
scopes/k8s/metric/metricresolves type and unit: it belongs to the metric, not to the request.
What else changes
A failed or empty query returns the response with an empty series instead of
exiting non-zero, so a graph with no data stays a graph with no data instead of
surfacing a provider error.
Adds
stream.iterator_age, the one metric that says whether the consumer of aDynamoDB stream is falling behind. The others describe the function's health, and
it can be healthy while the stream piles up against the 24 hour retention. The
prefix puts it in its own section, following how
scheduled_tasknamespaces itscronjob.*metrics.Verified
Against a live function, with the agent pointed at this branch:
GET /metricreturns the six metrics with their titles and unitsDurationreturns ~2 ms over 60 datapointsstream.iterator_agereturns 2583 ms after writing to the linked table, and anempty series when the function is only serving HTTP — CloudWatch does not
publish it unless an event source mapping invoked the function
unit: unknownwithout failing🤖 Generated with Claude Code