Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc-server/docs/installation/agent/_category_.json
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"label": "K8s Agent", "position": 2}
{"label": "Cluster Collector", "position": 2}
2 changes: 1 addition & 1 deletion doc-server/docs/installation/agent/connect/_category_.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,6 @@
"position": 2,
"link": {
"type": "generated-index",
"description": "Point the agent at the metrics, logs, traces, and alerts you already collect."
"description": "Point the Cluster Collector at the metrics, logs, traces, and alerts you already collect."
}
}
30 changes: 15 additions & 15 deletions doc-server/docs/installation/agent/connect/alertmanager.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,17 @@ sidebar_position: 1

# Alert Forwarding (Alertmanager)

NudgeBee investigates the alerts you already have. To get them, your Alertmanager has to POST them to the agent. The agent Helm chart cannot set this up, because the configuration lives in your Alertmanager.
NudgeBee investigates the alerts you already have. To get them, your Alertmanager has to POST them to the collector. The collector Helm chart cannot set this up, because the configuration lives in your Alertmanager.

If you skip it, nothing breaks visibly. Metrics are pulled, so a bad Prometheus URL shows up right away. Alerts are pushed, so when no receiver targets the agent, all the pods stay healthy, no error is logged, and NudgeBee just never raises an alert-driven event. If your cluster shows metrics and workloads but no alerts, start here.
If you skip it, nothing breaks visibly. Metrics are pulled, so a bad Prometheus URL shows up right away. Alerts are pushed, so when no receiver targets the collector, all the pods stay healthy, no error is logged, and NudgeBee just never raises an alert-driven event. If your cluster shows metrics and workloads but no alerts, start here.

There are three independent checks:

| Check | What it proves | Where to diagnose |
|---|---|---|
| **Alertmanager Connected** in Agent Health | The runner can reach the configured Alertmanager `/-/healthy` endpoint. | Agent configuration, service discovery, authentication, and NetworkPolicy. |
| **Alertmanager Connected** in Agent Health | The runner can reach the configured Alertmanager `/-/healthy` endpoint. | Collector configuration, service discovery, authentication, and NetworkPolicy. |
| NudgeBee receiver appears in the loaded Alertmanager route tree | Alertmanager accepted the routing configuration. | The generated Alertmanager config and route ordering. |
| A firing alert appears in NudgeBee | Alertmanager matched the route and delivered the webhook to the correct agent/account. | Alertmanager delivery logs, receiver URL, network path, and agent logs. |
| A firing alert appears in NudgeBee | Alertmanager matched the route and delivered the webhook to the correct collector/account. | Alertmanager delivery logs, receiver URL, network path, and collector logs. |

A green Agent Health status proves only the first check. It does not prove that Alertmanager is configured to send alerts to NudgeBee.

Expand Down Expand Up @@ -97,7 +97,7 @@ helm upgrade --install nudgebee-prometheus prometheus-community/kube-prometheus-
-f https://raw.githubusercontent.com/nudgebee/k8s-agent/main/kube-prometheus-stack-values.yaml
```

One catch: a values file cannot template, so the URL in it is hardcoded to `nudgebee-agent-runner.nudgebee-agent.svc`. It resolves only if your agent release is named `nudgebee-agent` in a namespace of the same name. With any other name, download the file, replace that URL with the one `helm install` printed, and install from your copy. When the URL does not resolve, Alertmanager logs the failed sends and fires `AlertmanagerFailedToSendAlerts`, but NudgeBee has no way to tell you it is missing alerts.
One catch: a values file cannot template, so the URL in it is hardcoded to `nudgebee-agent-runner.nudgebee-agent.svc`. It resolves only if your collector release is named `nudgebee-agent` in a namespace of the same name. With any other name, download the file, replace that URL with the one `helm install` printed, and install from your copy. When the URL does not resolve, Alertmanager logs the failed sends and fires `AlertmanagerFailedToSendAlerts`, but NudgeBee has no way to tell you it is missing alerts.

If you already run kube-prometheus-stack and did not install it from that file, add the route and receiver under `alertmanager.config` in your own values:

Expand Down Expand Up @@ -198,7 +198,7 @@ kubectl create secret generic metrics-datasource-secret \
-n nudgebee-agent
```

If your backend uses basic auth or OAuth2 instead, VMAlert takes `datasource.basicAuth` or `datasource.oauth2` in place of the bearer token below. None of this involves the NudgeBee agent, which only receives what VMAlertmanager forwards.
If your backend uses basic auth or OAuth2 instead, VMAlert takes `datasource.basicAuth` or `datasource.oauth2` in place of the bearer token below. None of this involves the NudgeBee Cluster Collector, which only receives what VMAlertmanager forwards.

### 2. Install

Expand All @@ -210,7 +210,7 @@ helm upgrade --install vma vm/victoria-metrics-k8s-stack --version 0.57.0 -f vm-

### 3. `vm-operator.yaml`

Point `datasource.url` at the query endpoint the agent already uses (`globalConfig.prometheus_url`). Everything else the VictoriaMetrics stack can install is turned off here, so this release only evaluates rules and routes alerts.
Point `datasource.url` at the query endpoint the collector already uses (`globalConfig.prometheus_url`). Everything else the VictoriaMetrics stack can install is turned off here, so this release only evaluates rules and routes alerts.

```yaml
victoria-metrics-operator:
Expand Down Expand Up @@ -359,17 +359,17 @@ To keep the token out of the URL, send it as a header instead. Both work:
credentials: '<token>'
```

**If one Alertmanager serves several clusters**, split the traffic rather than sending everything to one destination. Add a route per cluster matching on the external label your Prometheus or Ruler sets (`cluster`, `prometheus`, or whatever you configured), and give each route its own receiver — either the in-cluster agent for that cluster, or the same public webhook with a different `&cluster=` query label so NudgeBee can tell the events apart.
**If one Alertmanager serves several clusters**, split the traffic rather than sending everything to one destination. Add a route per cluster matching on the external label your Prometheus or Ruler sets (`cluster`, `prometheus`, or whatever you configured), and give each route its own receiver — either the in-cluster collector for that cluster, or the same public webhook with a different `&cluster=` query label so NudgeBee can tell the events apart.

This matters most when the receiver is an in-cluster agent: the agent stamps every alert it accepts with its own cluster name, so alerts from cluster B arriving at cluster A's agent are attributed to cluster A and name resources that do not exist there.
This matters most when the receiver is an in-cluster collector: the collector stamps every alert it accepts with its own cluster name, so alerts from cluster B arriving at cluster A's collector are attributed to cluster A and name resources that do not exist there.

---

## Using an AlertmanagerConfig CR

If your platform manages Alertmanager entirely through CRs, you can route to NudgeBee that way — but not by simply creating an `AlertmanagerConfig` in the agent's namespace. That is the one arrangement that quietly does the wrong thing.
If your platform manages Alertmanager entirely through CRs, you can route to NudgeBee that way — but not by simply creating an `AlertmanagerConfig` in the collector's namespace. That is the one arrangement that quietly does the wrong thing.

The operator injects a `namespace=<the CR's own namespace>` matcher into every route it generates from an `AlertmanagerConfig`. A CR in the agent's namespace therefore forwards only alerts that originated in that namespace. NudgeBee receives a trickle, which reads as "mostly working" rather than as a broken config.
The operator injects a `namespace=<the CR's own namespace>` matcher into every route it generates from an `AlertmanagerConfig`. A CR in the collector's namespace therefore forwards only alerts that originated in that namespace. NudgeBee receives a trickle, which reads as "mostly working" rather than as a broken config.

What controls this is `spec.alertmanagerConfigMatcherStrategy.type` on the `Alertmanager` resource:

Expand Down Expand Up @@ -431,7 +431,7 @@ Do not reach for `None` to fix this. It drops the namespace restriction for ever

## Troubleshooting: Why is NudgeBee Not Receiving Alerts? {#verify}

If your cluster shows healthy metrics and active workloads in the Console but NudgeBee never generates alert-driven events or incident investigations, Alertmanager webhooks are not reaching the agent.
If your cluster shows healthy metrics and active workloads in the Console but NudgeBee never generates alert-driven events or incident investigations, Alertmanager webhooks are not reaching the collector.

Follow this systematic diagnostic checklist to locate and fix the blockage.

Expand Down Expand Up @@ -546,13 +546,13 @@ Look for errors like:

#### Correcting the Webhook URL

The receiver URL must match your agent release name and namespace:
The receiver URL must match your collector release name and namespace:

```
http://<release-name>-runner.<agent-namespace>.svc.cluster.local/api/alerts
```

- If Alertmanager runs in a different namespace (e.g. `monitoring`) than the agent (`nudgebee-agent`), always supply the full `.svc.cluster.local` domain.
- If Alertmanager runs in a different namespace (e.g. `monitoring`) than the collector (`nudgebee-agent`), always supply the full `.svc.cluster.local` domain.
- The runner Service listens on port **80** and routes to container port 5000. Do not append `:5000` to the Service URL.

---
Expand Down Expand Up @@ -599,7 +599,7 @@ spec:

### Step 6: Test Runner Webhook Intake Directly

You can test the agent runner's `/api/alerts` endpoint independently of Alertmanager to confirm it processes payloads and generates findings:
You can test the collector runner's `/api/alerts` endpoint independently of Alertmanager to confirm it processes payloads and generates findings:

1. Port-forward the runner Service:
```bash
Expand Down
2 changes: 1 addition & 1 deletion doc-server/docs/installation/agent/connect/grafana.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ NudgeBee can integrate with Grafana to surface dashboards directly in the NudgeB

Setting `url` is what enables the integration; there is no `enabled` flag.

3. **Apply configuration and upgrade the agent**
3. **Apply configuration and upgrade the collector**
- **Using shell script**:
```bash
sh installation.sh -a <NUDGEBEE_AUTH_KEY> -f values.yaml
Expand Down
32 changes: 16 additions & 16 deletions doc-server/docs/installation/agent/connect/logging/elk.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,15 @@ sidebar_position: 2

# Elasticsearch & OpenSearch (ELK)

Integrate your existing Elasticsearch or OpenSearch cluster with the NudgeBee Kubernetes Agent for centralized log querying and automated incident analysis.
Integrate your existing Elasticsearch or OpenSearch cluster with the NudgeBee Cluster Collector for centralized log querying and automated incident analysis.

The agent supports:
The collector supports:
- **Elasticsearch**: Elasticsearch 7.x, 8.x, and Elastic Cloud.
- **OpenSearch**: Amazon OpenSearch Service, self-hosted OpenSearch, and Logz.io (via OpenSearch PPL or standard Elasticsearch Query DSL).

---

## 1. NudgeBee Agent Configuration
## 1. NudgeBee Cluster Collector Configuration

Elasticsearch is **opt-in**. You must set `runner.es.enabled: true` **and** supply `runner.es.url`. Setting only the URL leaves Elasticsearch disabled so that accidental configuration does not override other providers.

Expand All @@ -21,7 +21,7 @@ runner:
es:
# Explicit opt-in required (defaults to false)
enabled: true
# Cluster endpoint reachable from the agent runner pod
# Cluster endpoint reachable from the collector runner pod
url: "https://elasticsearch-es-internal-http.monitoring.svc:9200"

# Auth Option A: API Key (takes precedence over username/password)
Expand All @@ -47,16 +47,16 @@ runner:
| :--- | :--- | :--- | :--- |
| `runner.es.enabled` | boolean | `false` | Enables Elasticsearch integration. Must be explicitly set to `true`. |
| `runner.es.url` | string | `""` | Base URL for Elasticsearch or OpenSearch (e.g., `http://elasticsearch:9200` or `https://...`). |
| `runner.es.apiKey` | string | `""` | Elasticsearch API key. When present, the agent adds `Authorization: ApiKey <apiKey>`. |
| `runner.es.apiKey` | string | `""` | Elasticsearch API key. When present, the collector adds `Authorization: ApiKey <apiKey>`. |
| `runner.es.username` | string | `""` | Basic auth username. Used only if `apiKey` is empty. |
| `runner.es.password` | string | `""` | Basic auth password. |
| `runner.es.headers` | string | `""` | Semicolon-separated extra headers (`Key: Value; Key2: Value2`) passed on every request. |
| `runner.es.sslVerify` | boolean | `false` | When connecting via HTTPS, whether to enforce TLS certificate verification. Defaults to `false`. |

:::tip Provider Precedence
When multiple logging integrations are defined, the agent applies strict priority ordering:
When multiple logging integrations are defined, the collector applies strict priority ordering:
**Pinot → Elasticsearch → SigNoz → Loki**.
If `runner.es.enabled: true` is set alongside `runner.loki.url`, the agent queries **Elasticsearch**. See the [Logging Overview](./index.md#provider-precedence) for details.
If `runner.es.enabled: true` is set alongside `runner.loki.url`, the collector queries **Elasticsearch**. See the [Logging Overview](./index.md#provider-precedence) for details.
:::

---
Expand Down Expand Up @@ -139,15 +139,15 @@ runner:

## 3. OpenSearch & PPL Support

The NudgeBee Agent natively supports OpenSearch clusters:
The NudgeBee Cluster Collector natively supports OpenSearch clusters:
- **Elasticsearch Query DSL**: Standard `POST /{index}/_search` queries work across both Elasticsearch and OpenSearch.
- **Piped Processing Language (PPL)**: The agent transparently invokes `POST /_plugins/_ppl` when queries utilize OpenSearch PPL expressions.
- **Piped Processing Language (PPL)**: The collector transparently invokes `POST /_plugins/_ppl` when queries utilize OpenSearch PPL expressions.

---

## 4. Health Probing & Diagnostics

The Kubernetes Agent periodically validates connection health by probing:
The Cluster Collector periodically validates connection health by probing:
```http
GET <ES_URL>/_cluster/health
```
Expand Down Expand Up @@ -175,7 +175,7 @@ A healthy response returns JSON with `"status": "green"` or `"status": "yellow"`

## 5. Troubleshooting Common Issues {#troubleshooting}

### Scenario 1: `HTTP 401 Unauthorized` in Health Check or Agent Logs
### Scenario 1: `HTTP 401 Unauthorized` in Health Check or Collector Logs

* **Symptom**: Agent Health badge shows `Logs: Disconnected` with error `HTTP 401: Unauthorized`.
* **Root Cause**: Invalid API key or username/password credentials.
Expand All @@ -196,14 +196,14 @@ A healthy response returns JSON with `"status": "green"` or `"status": "yellow"`
### Scenario 3: `x509: certificate signed by unknown authority`

* **Symptom**: Health check or query fails during TLS handshake with an internal self-signed certificate.
* **Root Cause**: The agent pod does not trust the cluster's internal Certificate Authority.
* **Root Cause**: The collector pod does not trust the cluster's internal Certificate Authority.
* **Resolution**:
- **Quick Fix**: Set `runner.es.sslVerify: false` in your Helm values to disable strict TLS verification.
- **Production Fix**: Mount your internal CA certificate into the runner deployment or add it to the cluster's trusted CA bundle.

### Scenario 4: Empty Log Query Results (`index_not_found_exception`)

* **Symptom**: Queries succeed but return 0 hits, or agent logs report `index_not_found_exception [no such index]`.
* **Symptom**: Queries succeed but return 0 hits, or collector logs report `index_not_found_exception [no such index]`.
* **Root Cause**: The queried index pattern does not match the actual index names or data streams in Elasticsearch.
* **Resolution**:
1. List existing indices from the cluster:
Expand All @@ -212,17 +212,17 @@ A healthy response returns JSON with `"status": "green"` or `"status": "yellow"`
curl -k -fsS -u "elastic:<password>" \
https://elasticsearch-es-internal-http.monitoring.svc:9200/_cat/indices?v
```
2. Check data stream aliases if using modern Elastic Agent or Fleet:
2. Check data stream aliases if using modern Elastic Collector or Fleet:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Elastic Agent is the official name of the product by Elastic (often used with Fleet). Renaming it to "Elastic Collector" is incorrect and may confuse users. It should remain "Elastic Agent".

Suggested change
2. Check data stream aliases if using modern Elastic Collector or Fleet:
2. Check data stream aliases if using modern Elastic Agent or Fleet:
References
  1. When writing documentation, always use the verbatim labels, feature flags, or UI strings as they appear in the product interface, even if they deviate from standard spelling or capitalization rules, to prevent confusing users who are trying to locate those elements.

```bash
kubectl run es-data-streams --rm -i --restart=Never --image=curlimages/curl -- \
curl -k -fsS -u "elastic:<password>" \
https://elasticsearch-es-internal-http.monitoring.svc:9200/_data_stream
```
3. If your logs reside in non-standard index names (e.g., `app-logs-*`), configure the index pattern appropriately or set `ELASTICSEARCH_LOG_INDEX` in runner environment variables.

### Scenario 5: Elasticsearch Configured but Agent Shows Loki / SigNoz
### Scenario 5: Elasticsearch Configured but Collector Shows Loki / SigNoz

* **Symptom**: You added `runner.es.url`, but the agent still reports logs via Loki or SigNoz.
* **Symptom**: You added `runner.es.url`, but the collector still reports logs via Loki or SigNoz.
* **Root Cause**: `runner.es.enabled` was not set to `true`. Unlike Loki or SigNoz (which turn on whenever their URL is non-empty), Elasticsearch is an explicit opt-in.
* **Resolution**:
Ensure both `runner.es.enabled: true` and `runner.es.url` are set:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ sidebar_position: 1
---
# Logging

NudgeBee integrates with your existing logging backends to provide instant log context during automated root cause analysis (RCA) and incident triage. Instead of streaming raw log streams out of your cluster to a third-party service, the NudgeBee Agent runner acts as an authenticated in-cluster query proxy, fetching only the targeted log slices needed to investigate specific alert windows.
NudgeBee integrates with your existing logging backends to provide instant log context during automated root cause analysis (RCA) and incident triage. Instead of streaming raw log streams out of your cluster to a third-party service, the NudgeBee Cluster Collector runner acts as an authenticated in-cluster query proxy, fetching only the targeted log slices needed to investigate specific alert windows.

## Supported Logging Providers

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ sidebar_position: 4

Last9 exposes Loki apis and can be configured using Loki integrations.

## NudgeBee Agent Configuration
## NudgeBee Cluster Collector Configuration

```yaml
runner:
Expand Down
4 changes: 2 additions & 2 deletions doc-server/docs/installation/agent/connect/logging/logz.io.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,6 @@ Both `enabled: true` and `url` are required — a URL on its own leaves Elastics

Get the token from Logz.io under **Settings → Manage tokens → API tokens**. Use the region-specific host if your account is not in the default region (for example `https://api-eu.logz.io`).

:::note Requires agent chart 0.1.22 or newer
`runner.es.headers` was added in chart 0.1.22. On older agents the value is ignored and queries to Logz.io fail authentication.
:::note Requires collector chart 0.1.22 or newer
`runner.es.headers` was added in chart 0.1.22. On older collectors the value is ignored and queries to Logz.io fail authentication.
:::
Loading