Skip to content

Add a metrics endpoint - #123

Open
beetlebugorg wants to merge 16 commits into
mainfrom
feat/metrics-endpoint
Open

Add a metrics endpoint#123
beetlebugorg wants to merge 16 commits into
mainfrom
feat/metrics-endpoint

Conversation

@beetlebugorg

@beetlebugorg beetlebugorg commented Sep 1, 2026

Copy link
Copy Markdown
Owner

What

A dims-metrics handler serves Prometheus text 0.0.4 at a location that sets it. DimsMetricsEnabled is the switch and defaults to Off.

src/metrics.c holds a shared memory block that every worker adds to with 64-bit atomics, sized once in post config from AP_MPMQ_HARD_LIMIT_DAEMONS. src/metrics_report.c walks it and writes the exposition. dims_handler registers a pool cleanup per request, so every return path records once.

The endpoint reports 27 families:

Group Families
Request path dims_requests_total{endpoint,outcome}, dims_responses_total{endpoint,code}, dims_requests_in_flight{endpoint}, dims_request_duration_seconds{endpoint}
Source dims_source_fetch_duration_seconds, dims_source_bytes, dims_source_bytes_total, dims_source_format_total{format}, dims_origin_responses_total{code}
Processing dims_imagemagick_duration_seconds, dims_source_frames, dims_output_bytes, dims_output_bytes_total, dims_output_format_total{format}
ImageMagick and memory dims_imagemagick_resource_bytes{resource}, dims_imagemagick_resource_max_bytes{resource}, dims_imagemagick_resource_limit_bytes{resource}, dims_process_resident_bytes, dims_process_resident_max_bytes, dims_process_virtual_bytes, dims_workers
httpd dims_httpd_workers{state}, dims_httpd_processes, dims_httpd_processes_limit, dims_httpd_threads_per_process, dims_httpd_connections{state}, dims_httpd_generation
Build dims_start_time_seconds, dims_build_info

Counters and histograms come from the block. The ImageMagick resources and the process memory are per worker: each child writes its own slot when a request ends, and the endpoint sums the live slots and reports the widest one. The httpd families read the scoreboard at scrape time.

dims_stats_rec goes away, and dims_status_handler sums the requests table for the four numbers it prints.

The container serves images on 8000 and metrics on 8001, with DIMS_METRICS_ENABLED off.

Why

/dims-status/ prints four counters in prose. A Prometheus scrape of that page returns no series. The request record already holds the duration, the source size, the origin status, the frame count, and the output size, so the endpoint reports them without new measurement.

OTLP is push only. Its specification does not define a mode where a server exposes metrics for a collector to fetch. Scraping is the supported path into OpenTelemetry, and the compatibility specification states that the collector adds the resource attributes from the scrape target.

Folding dims_stats_rec into the block fixes two defects. dims_init set success_count to 1 at startup, so the endpoint reported one success before it served a request. The four increments sat in dims_send_image. A failure on a server with no error image returns before that function, so those requests went uncounted.

Port 8001 exists because Require ip reads the ingress controller address rather than the caller address. A second listener moves the decision to the network.

Verify

make -C test test
make -C test test-update
make -C test sanitize

305 cases pass, every golden file reproduces, and 114 sanitizer cases report no finding. test/http/test_metrics.c covers the format, each metric group, and the directive. test/unit/test_metrics.c covers the bucket search and the index maps. test/http/test_status.c asserts the two endpoints agree.

Against the built image:

docker run -d -p 8001:8001 -e DIMS_METRICS_ENABLED=on mod-dims:ci
curl http://127.0.0.1:8001/metrics

I measured 404 with the image defaults and 200 with the variable on. Port 8000 returns 404 for /metrics in both cases.

Breaking

/dims-status/ reports zero successes on a server before its first request, where it reported one. It also counts a failed request that returns no image, which it skipped before.

dims_metrics_init creates it in post config, before the first fork, and
sizes it from AP_MPMQ_HARD_LIMIT_DAEMONS. dims_metrics_child_init claims a
process slot.

The block holds the counters, the histograms, and one slot per child. The
index maps turn a status, a response code, and an image format into a slot,
and hold every value outside their list in the last one.

Nothing reads the block yet.
The bucket search, the histogram totals, and the four index maps. A bound
belongs to its own bucket, and a value above the last bound belongs to +Inf.

stub_httpd.c gains ap_mpm_query, which sizes the process slots.
DimsMetricsEnabled is the switch and defaults to off. A location that sets
the dims-metrics handler serves Prometheus text 0.0.4 when the directive is
on, and httpd returns 404 when it is off.

The writer prints a family per counter and per histogram, with cumulative
_bucket lines, a _sum, and a _count. Every value is zero until the next
commits record them.

A bucket bound goes through snprintf. The httpd formatter prints .005 for
0.005 and rounds 1048576 to 1048580, and a dashboard matches le as a string.

src/metrics_report.c holds the writer, so the in-process cases link the
block and the index maps without the httpd response API.
dims_requests_total, dims_responses_total, dims_request_duration_seconds,
and dims_requests_in_flight.

dims_handler sets d->endpoint and calls dims_metrics_request_begin, which
increments the gauge and registers a pool cleanup. The cleanup decrements
the gauge and records the outcome, the response code, and the duration, so
every return path in the handler records exactly once.
dims_source_fetch_duration_seconds, dims_source_bytes,
dims_source_bytes_total, dims_origin_responses_total, and
dims_source_format_total.

The request record already holds the duration, the size, and the origin
status, so the same pool cleanup records them. dims_fetch_remote_image
keeps the format the wand read as a metrics slot, because the string
belongs to ImageMagick.

download_time counts milliseconds, so a fetch inside one millisecond
reports zero. It belongs in the first bucket rather than nowhere.
dims_imagemagick_duration_seconds, dims_source_frames,
dims_output_format_total, dims_output_bytes_total, and dims_output_bytes.

dims_send_image keeps the format and the blob length on the request record,
and dims_process_image keeps the frame count. The pool cleanup reports them
with the rest.
Each child writes MagickGetResource for the four limited resources, and its
resident and virtual size from /proc/self/statm, into the slot it claimed.
The endpoint sums the live slots and reports the widest one, so two series
cover a pool of any size.

MagickGetResourceLimit reads the ceiling a worker runs under, so a
dashboard divides use by limit.

/proc/self/statm is Linux only. A platform without it reports no memory
gauges rather than zero.
dims_httpd_workers by scoreboard state, the process count and its ceiling,
the thread count, connections by state, and the MPM generation.

The scoreboard is the server's own record, so the endpoint reads it at
scrape time rather than through the metrics block.

Ready workers against the ceiling times the thread count is the headroom an
operator reads before raising DIMS_SERVER_LIMIT.
dims_stats_rec and its shared memory block go away. dims_status_handler
sums the requests table for the four numbers it prints, so the two
endpoints report the same counters.

dims_init set success_count to 1 at startup, so /dims-status/ reported one
success before it served a request. Summing the table reports zero.

The four apr_atomic_inc32 calls in mod_dims.c sat in dims_send_image, which
a failure with no error image never reaches. The pool cleanup records every
request whatever path it takes.
Port 8000 serves images. Port 8001 serves /metrics and /dims-status/, so a
Service exposes 8000 and a scrape targets 8001.

DIMS_METRICS_ENABLED defaults to off, so the image publishes no metrics
until an operator turns it on.

Behind an ingress controller httpd reads the controller address rather than
the caller address, so Require ip cannot separate a scrape from a caller. A
second listener moves the decision to the network.

The Server image job scrapes 8001 and checks for a module counter and an
httpd counter.
A page for /metrics and a page for DimsMetricsEnabled, plus rows in the
endpoints table, the defaults table, and the sidebar.

dims_build_info and dims_start_time_seconds report the build and the
restart time. DimsStatusVerbose off drops the build gauge, the same way it
drops the version lines from /dims-status/.

A label value escapes a backslash, a double quote, and a newline. The
library versions come from ImageMagick and libcurl.

dims3.md and dims4.md state that the client segment is a legacy of those two
endpoints and that /dims5/ dropped it.
dims_source_fetch_total and dims_source_fetch_errors_total by libcurl code.
dims_netguard_refusals_total by reason, and dims_allowlist_checks_total by
mode and result. dims_signature_checks_total by endpoint and result, and
dims_eurl_decrypt_total. dims_operations_total, dims_operation_failures_total,
and dims_operation_duration_seconds per operation.
dims_imagemagick_exceptions_total by kind and severity.
dims_overlay_cache_lookups_total, dims_overlay_cache_evictions_total,
dims_error_images_total, and dims_error_image_failures_total.

An operation counts one call. A multi-frame source runs the command loop
once per frame, and strip and format run without a command, so the trigger
label separates the two.

apr_time_now costs 15.7 ns on this host, measured over 20 million calls.
Two per operation call adds about 31 ns against a transform measured in
milliseconds.

The allowlist reports at two sites. handler.c refuses a /dims3/ request
before the fetch, and the guard runs its own check at fetch time under
DimsAllowlistSigned.
Fourteen families and their labels join the endpoint page.
test/endurance/observe holds a Prometheus and a Grafana that join the soak
network and scrape the server on port 8001. up.sh starts them and prints the
dashboard address. Grafana listens on every interface, so another machine
reaches it.

The dashboard has six sections: the overview, the request path, the source
fetch, the image processing, the guard and the signature, and the resources.

run.sh sets DIMS_METRICS_ENABLED=on, so a run publishes its metrics.
The module answers a conditional request with 304 through
ap_meets_conditions. The response code list held no slot for it, so a
caching client counted under "other".

The dashboard splits its error ratio in two. A 4xx is the request the caller
sent, and a 5xx is the module or its source failing. One number covering
both reads as a fault when a caller sends a bad signature.
ImageMagick loads its coders with dlopen and unloads them before the
process ends. Valgrind discards the symbols for unloaded code, so every
frame inside ImageMagick printed as ???.

--keep-debuginfo=yes holds them. A leak now names the function and the
line: AcquireSemaphoreInfo through NewMagickWand, and GetMagicInfo through
MagickReadImageBlob.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant