Improve CrowdSec hardening and reverse proxy troubleshooting

src/pages/manage/reverse-proxy/access-logs.mdx

@@ -9,10 +9,10 @@ NetBird logs every request and connection that passes through your reverse proxy
 
 ## Viewing access logs
 
-Access logs are available in the NetBird dashboard under **Activity** > **Proxy Events**. This view displays a table of all HTTP requests and L4 connections that have passed through your reverse proxy services, with filters to narrow down results by time range, status, or other fields.
+Access logs are available in the NetBird dashboard under **Reverse Proxy** > **Access Logs**. This view displays a table of all HTTP requests and L4 connections that have passed through your reverse proxy services, with filters to narrow down results by time range, status, or other fields.
 
 <p>
-    <img src="/docs-static/img/manage/reverse-proxy/access-logs/access-logs-table.png" alt="Proxy Events table showing reverse proxy access log entries" className="imagewrapper"/>
+    <img src="/docs-static/img/manage/reverse-proxy/access-logs/proxy-events-table.png" alt="Proxy Events table showing reverse proxy access log entries" className="imagewrapper"/>
 </p>
 
 You can also retrieve access logs programmatically using the API:
@@ -37,9 +37,9 @@ Every log entry (HTTP and L4) shares a common set of fields. Some fields are onl
 | **Bytes Downloaded** | Bytes sent from backend to client | Yes | Yes |
 | **Source IP** | The client's IP address | Yes | Yes |
 | **Location** | Country, city, and subdivision based on source IP geolocation | Yes | Yes |
-| **Auth Method** | Authentication method used (SSO, password, PIN, header, or none) | Yes | N/A |
-| **User** | The authenticated user's ID (if SSO was used) | Yes | N/A |
-| **Reason** | Reason for denial, if applicable | Yes | Yes |
+| **Auth Method** | Raw `auth_method_used` value: `oidc` (shown as SSO in the dashboard), `password`, `pin`, or `header`. For denied requests, carries the restriction code instead (e.g. `ip_restricted`, `crowdsec_ban`). Omitted from the API response when empty | Yes | Restriction code on denials |
+| **User** | The authenticated user's ID, set when `oidc` authentication was used. Omitted from the API response when empty | Yes | N/A |
+| **Reason** | Exactly one of two values: `Authentication failed` when authentication or an access restriction rejected the request, or `Request failed` when an authenticated request returned `4xx`/`5xx`. Set for HTTP entries only and omitted from the API response when empty. Never a specific denial code: see the note under [Deny reasons](#deny-reasons) | Yes | Omitted |
 
 ## Understanding log entries
 
@@ -48,18 +48,18 @@ Every log entry (HTTP and L4) shares a common set of fields. Some fields are onl
 HTTP log entries fall into three categories based on the status code:
 
 - **Allowed requests**: successful requests show a `2xx` status code along with the authentication method used to access the service.
-- **Denied requests**: failed authentication or access restriction blocks show `401` or `403` status codes with a reason explaining why the request was denied (e.g., invalid password, missing SSO session, IP restricted, country restricted).
+- **Denied requests**: failed authentication or access restriction blocks show `401` or `403` status codes with `reason` set to `Authentication failed`. The specific cause (invalid password, missing SSO session, IP restricted, country restricted, CrowdSec verdict) is carried in `auth_method_used`, not in `reason`.
 - **Errors**: backend errors or proxy issues show `5xx` status codes. These typically indicate that the target service is unreachable or returned an error.
 
 ### L4 log entries
 
 L4 entries are logged when the connection closes and record the total bytes transferred in each direction and the connection duration. L4 entries do not have HTTP status codes.
 
-Denied L4 connections (blocked by access restrictions) are logged immediately with a deny reason. Since L4 services do not support authentication, denials come from access restrictions only.
+Denied L4 connections (blocked by access restrictions) are logged immediately. L4 entries carry no `reason` value, so the restriction code identifies the denial. Since L4 services do not support authentication, denials come from access restrictions only.
 
 ### Deny reasons
 
-The following deny reasons can appear for both HTTP and L4 services:
+The following deny reasons identify why a connection was rejected. Note that for HTTP services these values are not carried in the entry's `reason` field: see the note below the table.
 
 | Reason | Description |
 |--------|-------------|
@@ -73,7 +73,28 @@ The following deny reasons can appear for both HTTP and L4 services:
 
 All CrowdSec decision types (ban, captcha, throttle) result in a connection denial in enforce mode. The proxy does not serve captcha challenges or apply rate limiting: the decision type is recorded for informational purposes only.
 
-When CrowdSec is in **observe** mode, the verdict appears in the log metadata but the deny reason field is empty (the connection is allowed). In the dashboard, these entries render with an observe-mode badge on the reason cell and show the underlying decision type (ban, captcha, throttle, unavailable) on hover. This lets you audit what CrowdSec would block without affecting traffic. For a self-test workflow, see [Testing the integration](/selfhosted/maintenance/crowdsec#testing-the-integration).
+<Note>
+For HTTP services, the deny code from the table above is recorded in the `auth_method_used` field, and the entry's `reason` field carries a synthesized generic value rather than the specific code. This applies to every access restriction, not only CrowdSec:
+
+```json
+{ "status_code": 403, "reason": "Authentication failed", "auth_method_used": "ip_restricted" }
+```
+
+```json
+{
+  "status_code": 403,
+  "reason": "Authentication failed",
+  "auth_method_used": "crowdsec_ban",
+  "metadata": { "crowdsec_verdict": "crowdsec_ban" }
+}
+```
+
+When reading entries through `GET /api/events/proxy`, match on `auth_method_used` (and `metadata.crowdsec_verdict` for CrowdSec specifically) rather than `reason`.
+
+Observe-mode entries carry the normal status code and record both `crowdsec_mode` and `crowdsec_verdict` in `metadata`. Because the connection is allowed, CrowdSec itself contributes no `reason`, but the field can still be populated by a later stage of the request such as an authentication or backend failure. Treat `metadata.crowdsec_mode` as the signal that an entry is an observe-mode verdict, not the absence of `reason`.
+</Note>
+
+When CrowdSec is in **observe** mode, the verdict appears in the log metadata and CrowdSec adds no deny reason of its own (the connection is allowed). In the dashboard, these entries render with an observe-mode badge on the reason cell and show the underlying decision type (ban, captcha, throttle, unavailable) on hover. This lets you audit what CrowdSec would block without affecting traffic. For a self-test workflow, see [Testing the integration](/selfhosted/maintenance/crowdsec#testing-the-integration).
 
 ## Use cases
 

src/pages/selfhosted/maintenance/crowdsec-dashboard.mdx

@@ -96,15 +96,16 @@ services:
       - '--experimental.plugins.bouncer.version=v1.6.0'
 ```
 
-If your generated Compose project network is named `netbird_netbird`, make sure Traefik uses that Docker network:
+The generated Compose file already sets `--providers.docker.network` in the Traefik command block, normally to `netbird`, matching the network key in the Compose file. Leave it as generated and do not add a second copy of the flag.
 
-```yaml
-services:
-  traefik:
-    command:
-      - '--providers.docker.network=netbird_netbird'
+Only change it if Traefik logs warnings about a missing Docker network. Compose prefixes the network name with the project name, which defaults to the install directory name, so the actual name varies by deployment. Check it before editing:
+
+```bash
+docker network ls | grep netbird
 ```
 
+An install in `/root` produces `root_netbird`; one in `/opt/netbird` produces `netbird_netbird`.
+
 Under `services.traefik.depends_on`, make Traefik wait until CrowdSec is healthy:
 
 ```yaml
@@ -299,10 +300,105 @@ curl -ks -A 'Mozilla/5.0 NetBirdDashboardCheck' -o /dev/null -w '%{http_code}\n'
 
 The response should be `200`.
 
+## Verifying the middleware stays attached
+
+This protection fails open. If the middleware is detached for any reason, most commonly a label edit applied with `docker compose restart` instead of `docker compose up -d`, which does not recreate the container, requests stop being inspected and nothing reports an error. The dashboard loads, authentication works, and all containers report healthy. The only symptom is that a known malicious probe returns `404` instead of `403`.
+
+After any change to the labels, recreate the affected containers and re-run the probe:
+
+```bash
+docker compose up -d dashboard netbird-server
+curl -ks -o /dev/null -w '%{http_code}\n' "https://$NETBIRD_DOMAIN/.env"
+```
+
+A `403` confirms the middleware is in the request path. A `404` means it is not.
+
+To surface this automatically, monitor the same probe from outside the deployment and treat `403` as the healthy response. The middleware is attached to the two routers separately, so probe both. A check on `/.env` alone exercises only the dashboard router and would still return `403` if the middleware were detached from the API router. Point an uptime monitor or a cron job at each of:
+
+```text
+https://<your-domain>/.env
+https://<your-domain>/api/.env
+```
+
+Alert when either response is anything other than `403`. This is an inverted check, alerting on the absence of a block rather than on an outage, but it is the only signal that confirms the protection layer is still in the request path.
+
+<Warning>
+  Do not enable the Traefik API (`--api.insecure=true`) for this purpose. The
+  API serves the full dynamic configuration without authentication, including
+  middleware plugin settings such as `crowdsecLapiKey`. Even with port `8080`
+  unpublished, every container on the Docker network could then read the
+  CrowdSec bouncer key.
+</Warning>
+
+<Warning>
+  This is detection only. Nothing here restores enforcement automatically: the
+  probe tells you protection has stopped, and reattaching the middleware
+  remains a manual step.
+</Warning>
+
+## Recovering from a dashboard lockout
+
+If AppSec or an IP decision blocks you from the dashboard, the block also covers `/api` and `/oauth2`, so it cannot be lifted from the UI. All recovery is performed over SSH on the host.
+
+First identify which of the two mechanisms is blocking you, because they are handled differently:
+
+```bash
+docker compose exec crowdsec cscli decisions list
+```
+
+If your address is listed, an **IP decision** is blocking you. If it is not, the block is an **AppSec** match on the request itself: AppSec inspects requests and does not create decisions, so `cscli decisions delete` has no effect in that case. Confirm with `cscli metrics show appsec` and `cscli alerts list`, where AppSec matches appear with `kind` set to `waf`.
+
+Allowlisting the address resolves both cases. `cscli allowlists add` fails with `allowlist '<name>' not found` if the list does not already exist, so create it first if you have not set one up:
+
+```bash
+docker compose exec crowdsec cscli allowlists create netbird-admins -d "Addresses that should never be blocked"
+docker compose exec crowdsec cscli allowlists add netbird-admins <your-ip> -d "admin"
+```
+
+Adding an address that currently carries a decision expires that decision immediately. Allowlisted addresses are also exempted from AppSec blocking, though that takes effect on the Traefik bouncer's next stream refresh rather than instantly, so allow up to a minute. See [Allowlisting addresses](/selfhosted/maintenance/crowdsec#allowlisting-addresses) for details.
+
+If an IP decision is present and you want it gone without allowlisting the address permanently, delete it directly:
+
+```bash
+docker compose exec crowdsec cscli decisions delete --ip <your-ip>
+```
+
+The Traefik bouncer runs in `stream` mode and refreshes periodically, so allow up to a minute for access to return.
+
+If the dashboard is still unreachable and you need immediate access, detach the middleware. It is attached to **two** routers and both must be commented out:
+
+```yaml
+services:
+  dashboard:
+    labels:
+      # - traefik.http.routers.netbird-dashboard.middlewares=netbird-dashboard-crowdsec@docker
+
+  netbird-server:
+    labels:
+      # - traefik.http.routers.netbird-dashboard-api.middlewares=netbird-dashboard-crowdsec@docker
+```
+
+```bash
+docker compose up -d dashboard netbird-server
+```
+
+<Warning>
+  Commenting out only the `dashboard` label leaves the dashboard reachable while
+  `/api` and `/oauth2` continue to return `403`. The interface loads but cannot
+  authenticate or fetch data, which is easily mistaken for a different fault.
+  Detach both labels, or neither.
+</Warning>
+
+Restore both labels once the underlying issue is resolved, recreate the containers, and confirm with the `/.env` probe above.
+
 ## Troubleshooting
 
 If all dashboard requests return `403` immediately after startup, Traefik may have started before CrowdSec LAPI and AppSec were ready. Confirm that the CrowdSec health check is present and that Traefik uses `depends_on.condition: service_healthy`.
 
+If CrowdSec is stopped or unreachable while `crowdsecAppsecUnreachableBlock=true` is set, all dashboard requests are denied by design.
+
+Separately, a reverse proxy service in `enforce` mode denies all connections while its bouncer has not completed its initial sync, for example when the proxy starts with the LAPI unavailable. See [Enforcement modes](/selfhosted/maintenance/crowdsec#enforcement-modes).
+
 If Traefik logs warnings about a missing Docker network, check the actual network name:
 
 ```bash