fix(metrics): correct task duration q95 and ship Grafana dashboard - #26
Merged
Conversation
Internal processors poll on a fixed ticker, so most ticks process zero tasks. The deferred prometheus timer recorded a duration for every tick, seeding task_processing_duration_seconds with empty-query DB latency and keeping histogram_quantile reporting a stale q95 for idle task types (e.g. audit.write showed a duration with no tasks). Record duration only when a poll processed >=1 task; errored polls are excluded too. Also rebalance the histogram buckets: dense in the 1ms-1s working range (a ~50ms p95 previously fell into the wide [10ms,100ms] bucket and drifted under interpolation) and sparse above 5s, where internal processors time out at 30s and finer minute-scale buckets would only add permanently-empty series.
Wrap every histogram_quantile in rate() over the _bucket series: without it the quantile was computed from counters accumulated since process start, so idle task types kept showing a stale q95. Set the top-level dashboard id to null so importing over an existing UID no longer fails with 'cannot change the ID of a dashboard'. Add grafana/ at the repo root with an import-friendly copy of the metrics dashboard and a README, and link it from the main README so the ready-made dashboard is discoverable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes two independent causes of a misleading Task Processing Duration q95 —
one in the exporter, one in the dashboard query — and makes the Grafana
dashboard discoverable and importable.
at least one task, so empty ticker polls no longer seed the histogram
with empty-query DB latency and leave idle task types showing a stale q95
p95 previously fell into the wide [10ms,100ms] bucket and drifted under
interpolation), sparse above 5s where processors time out at 30s
so quantiles reflect the selected window instead of counters accumulated
since process start
a dashboard" on import over an existing UID
dashboard plus a README, linked from the main README