Skip to content

feat: degrade gracefully when no Kubernetes cluster is present - #2

Merged
mattDev0 merged 1 commit into
mainfrom
feat/graceful-k8s-degradation
Aug 22, 2026
Merged

feat: degrade gracefully when no Kubernetes cluster is present#2
mattDev0 merged 1 commit into
mainfrom
feat/graceful-k8s-degradation

Conversation

@mattDev0

Copy link
Copy Markdown
Owner

Implements Option 3 of adr_01, which is now marked Accepted.

The problem

Production moved to Docker Compose on 2026-07-14, but the Kubernetes half of the dashboard was never adjusted to match. To stop the agent blocking at startup, scripts/deploy.sh fabricated a kubeconfig pointing at https://localhost:8443.

That hid the problem rather than solving it, because Client::try_default() only parses configuration — it never contacts a cluster. So the agent reported:

{"status": "healthy", "k8s": true}

…against a cluster that does not exist. Downstream, the deployments table, pod-health panel and log streaming all queried nothing, and the SLO tiles displayed 100% availability and a full error budget for it. scripts/health-check.sh even carried a comment acknowledging /health returns 503 "when Kubernetes is not configured".

Approach

Keep Compose as the runtime, keep the manifests as a supported alternative target, and remove the dishonest parts rather than the capability.

Agent

  • Verify before advertising. A real API call now confirms the server answers before Kubernetes is reported as available.
  • Bounded startup. try_connect() replaces an unbounded backoff loop that could never let the process start without a cluster. Three attempts, then continue without one.
  • AppState.kube_client becomes Option<Client>. Cluster-backed routes return 503, not 500 — an absent dependency is not a defect.
  • /health returns 200 with k8s: false instead of 503. Running without a cluster is a supported configuration. This also fixes a real bug: the old 503 made the orchestrator raise AgentUnreachableException for a perfectly healthy agent.
  • The deployment monitor only starts when a cluster is available.

Frontend

  • Kubernetes panels are replaced by an explicit "no cluster connected" state.
  • The pod / availability-SLI / error-budget tiles render instead of 0 Pods and 100.0%.
  • Cluster fetches are skipped entirely when the agent reports no cluster, so no pointless 503s.

Infrastructure

  • Placeholder kubeconfig generation removed from deploy.sh (23 lines) along with its compose mount.
  • Under infrastructure/k8s/ the service account supplies credentials, so the clustered path is unchanged — deploying the manifests restores every cluster feature automatically.

Validation

apps/agent          cargo test    5 passed (2 new)
apps/agent          cargo clippy -- -D warnings    clean
apps/orchestrator   mvn test      20 passed
apps/frontend       eslint        0 errors, 1 pre-existing warning
apps/frontend       vite build    OK
docker-compose*.yml YAML parse    OK

New tests:

  • test_cluster_routes_report_unavailable_without_kubernetes/deployments and /pods/health return 503 with no cluster.
  • test_health_is_ok_and_honest_without_kubernetes/health returns 200 with k8s: false.

Docs

  • adr_01 moves from Proposed to Accepted, with the decision and the reasoning against Options 1 and 2 recorded. The three options are retained.
  • .agents/rules/engineering-rules.md no longer claims the repos share a cluster, ingress and cloud.
  • README.md: corrected the claims that the agent "runs as a Kubernetes pod" and queries K3s, and marked cluster-dependent features accordingly.

Deployment note

No secrets or deployment credentials change. On merge the agent will log Starting without Kubernetes... and report k8s: false — that is the expected, correct output for this environment, not a regression.

Production moved to Docker Compose on 2026-07-14, but the Kubernetes
half of the dashboard was never adjusted. scripts/deploy.sh fabricated a
kubeconfig pointing at https://localhost:8443 so the agent would start,
and because Client::try_default() only parses configuration and never
contacts a cluster, /health reported "k8s": true against a cluster that
did not exist. The deployments table, pod-health panel and log streaming
queried nothing, and the SLO tiles showed 100% availability for it.

Implements Option 3 of adr_01: keep Compose as the runtime, keep the
manifests as a supported alternative target, and stop pretending.

Agent:
- Verify the API server actually answers before advertising cluster
  support, with bounded retries instead of an unbounded backoff loop
  that would never let the process start without a cluster.
- AppState.kube_client becomes Option<Client>; cluster-backed routes
  return 503 rather than 500, distinguishing an absent dependency from
  a defect.
- /health returns 200 with k8s=false instead of 503. Running without a
  cluster is a supported configuration, not a fault, and the previous
  503 also made the orchestrator raise AgentUnreachableException for a
  perfectly healthy agent.
- The deployment monitor only starts when a cluster is available.

Frontend:
- Kubernetes panels are replaced by an explicit "no cluster connected"
  state, and the pod/SLI/error-budget tiles render "-" rather than
  0 Pods and a 100% availability SLI.
- Cluster fetches are skipped entirely when the agent reports no cluster.

Infrastructure:
- Remove the placeholder kubeconfig generation and its compose mount.
  Under infrastructure/k8s the service account supplies credentials, so
  the clustered path is unchanged.

Adds tests covering 503 on cluster routes and an honest 200 from /health
when no cluster is configured.
@mattDev0
mattDev0 merged commit 5e4fbd1 into main Aug 22, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant