fix(prometheus): keep looking for a Prometheus the agent could not find at boot

[mayankpande88]

Summary

Prometheus service discovery ran once, at startup, and that one result decided for the life of the process whether prometheus_* actions, the enrichers, right-sizing and the service map were registered at all.

So an agent that started before its Prometheus stayed blind to it until someone restarted the pod. That happens more easily than it sounds: a fresh cluster where the agent and kube-prometheus-stack are installed together (the Service does not exist yet when we look), or any cluster where monitoring is added afterwards. The only trace was a single WARN in the boot log — from the outside it just looked like the product had no metrics: empty charts, and no right-sizing recommendations on workloads that plainly needed them.

Discovery is no longer a one-shot decision:

• The client is always constructed, with or without a URL, so its actions are always registered. Calls before a URL is known fail with prometheus: base URL not configured rather than unknown action.
• Its base URL is settable at runtime (RWMutex-guarded, read fresh per request), so a URL found later needs no re-registration and no restart.
• When no URL is configured at boot, a watcher re-runs discovery every 5 minutes and points the client at the first Prometheus that appears.

service_map, alert_rules and continuous_rightsizing lose their promClient != nil gates accordingly — each already reports the missing URL itself.

The install script is unaffected: it still passes --set globalConfig.prometheus_url=..., and an explicitly configured URL short-circuits all of this.

Type of change

• Bug fix (non-breaking)

Chart version

• No version bump needed (runner code only)

Test plan

make validate passes in runner/; full suite green under -race; golangci-lint run clean on the changed files (the one repo-wide gosec G704 in pkg/observability/signoz/client.go is pre-existing and untouched).

Tests added:

• TestWatchUntilFound_PicksUpAServiceAddedLater — nothing to find, then a Prometheus Service appears; the watcher reports it.
• TestWatchUntilFound_DoesNotInheritTheNegativeCache — guards the subtle part. Verified by regression: rewriting the watcher to reuse one Discoverer makes it fail, because FindFirst caches misses for an hour and would freeze the retry after the first attempt for the rest of the process's life.
• TestSetURL_MakesAnUnconfiguredClientWork — a client built with no URL rejects queries, then serves them after SetURL, with the trailing slash trimmed.
• TestSetURL_IsSafeWhileQueriesRun — concurrent SetURL and Query under -race, since the watcher writes the URL from its own goroutine while handlers read it.

How to verify on a cluster

1. Install the agent on a cluster with no Prometheus and no globalConfig.prometheus_url. The runner logs prometheus not configured: ... retry_interval=5m0s and the account's Prometheus health badge reads disconnected.
2. Ask it for metrics — the action is registered and answers prometheus: base URL not configured rather than failing as an unknown action.
3. helm install kube-prometheus-stack into that cluster and wait up to 5 minutes.
4. The runner logs prometheus discovered after startup ... url=http://...:9090 with no pod restart, the health badge flips to connected, metrics queries answer, and right-sizing recommendations start appearing on the next scan.

Related issues

None open for this. Found while verifying a customer evaluation report of "metrics unavailable" against the code.

[gemini-code-assist]

Code Review

This pull request introduces dynamic Prometheus discovery after startup, allowing the agent to register Prometheus actions unconditionally and update the client's URL in the background once a Prometheus service is discovered. Feedback on these changes points out that other components initialized at startup, such as the Telemetry Poster and Grafana Proxy, still rely on the static configuration URL and will not receive the dynamically discovered URL, which could lead to incomplete telemetry and failing proxied queries.

[mayankpande88]

Good catch on both — that was a real gap I introduced, since the boot-time path used to write the discovered URL back into cfg and the watcher deliberately does not (other goroutines read cfg concurrently).

Fixed in 57a779f:

1. Telemetry — now reports promClient.URL(). It is evaluated inside the per-tick Datasources closure, so it tracks discovery without touching cfg.
2. Grafana proxy — rather than refactor New's signature, Proxy gains an optional PrometheusURLFn func() string consulted per request; main wires it to promClient.URL. The static PrometheusURL field still serves every other caller and every existing test. Covered by TestProxy_HandlePrometheus_UsesTheURLLookupWhenSet, which 503s before discovery and forwards after, and pins the trailing-slash trim so the base and path don't double up.

I checked the other two cfg.PrometheusURL reads: line 217 is boot-time discovery itself and line 270 is the client constructor, both before the watcher can run, so neither needs the same treatment.

[github-actions]

📦 Image Tags Updated

I've automatically updated the image tags in `charts/nudgebee-agent/values.yaml` to the latest versions from GHCR for the `main` branch.

The image tags are now synchronized with the latest builds and ready for release.