feat: degrade gracefully when no Kubernetes cluster is present - #2
Merged
Conversation
Production moved to Docker Compose on 2026-07-14, but the Kubernetes half of the dashboard was never adjusted. scripts/deploy.sh fabricated a kubeconfig pointing at https://localhost:8443 so the agent would start, and because Client::try_default() only parses configuration and never contacts a cluster, /health reported "k8s": true against a cluster that did not exist. The deployments table, pod-health panel and log streaming queried nothing, and the SLO tiles showed 100% availability for it. Implements Option 3 of adr_01: keep Compose as the runtime, keep the manifests as a supported alternative target, and stop pretending. Agent: - Verify the API server actually answers before advertising cluster support, with bounded retries instead of an unbounded backoff loop that would never let the process start without a cluster. - AppState.kube_client becomes Option<Client>; cluster-backed routes return 503 rather than 500, distinguishing an absent dependency from a defect. - /health returns 200 with k8s=false instead of 503. Running without a cluster is a supported configuration, not a fault, and the previous 503 also made the orchestrator raise AgentUnreachableException for a perfectly healthy agent. - The deployment monitor only starts when a cluster is available. Frontend: - Kubernetes panels are replaced by an explicit "no cluster connected" state, and the pod/SLI/error-budget tiles render "-" rather than 0 Pods and a 100% availability SLI. - Cluster fetches are skipped entirely when the agent reports no cluster. Infrastructure: - Remove the placeholder kubeconfig generation and its compose mount. Under infrastructure/k8s the service account supplies credentials, so the clustered path is unchanged. Adds tests covering 503 on cluster routes and an honest 200 from /health when no cluster is configured.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements Option 3 of
adr_01, which is now marked Accepted.The problem
Production moved to Docker Compose on 2026-07-14, but the Kubernetes half of the dashboard was never adjusted to match. To stop the agent blocking at startup,
scripts/deploy.shfabricated a kubeconfig pointing athttps://localhost:8443.That hid the problem rather than solving it, because
Client::try_default()only parses configuration — it never contacts a cluster. So the agent reported:{"status": "healthy", "k8s": true}…against a cluster that does not exist. Downstream, the deployments table, pod-health panel and log streaming all queried nothing, and the SLO tiles displayed 100% availability and a full error budget for it.
scripts/health-check.sheven carried a comment acknowledging/healthreturns 503 "when Kubernetes is not configured".Approach
Keep Compose as the runtime, keep the manifests as a supported alternative target, and remove the dishonest parts rather than the capability.
Agent
try_connect()replaces an unbounded backoff loop that could never let the process start without a cluster. Three attempts, then continue without one.AppState.kube_clientbecomesOption<Client>. Cluster-backed routes return 503, not 500 — an absent dependency is not a defect./healthreturns 200 withk8s: falseinstead of 503. Running without a cluster is a supported configuration. This also fixes a real bug: the old 503 made the orchestrator raiseAgentUnreachableExceptionfor a perfectly healthy agent.Frontend
—instead of0 Podsand100.0%.Infrastructure
deploy.sh(23 lines) along with its compose mount.infrastructure/k8s/the service account supplies credentials, so the clustered path is unchanged — deploying the manifests restores every cluster feature automatically.Validation
New tests:
test_cluster_routes_report_unavailable_without_kubernetes—/deploymentsand/pods/healthreturn 503 with no cluster.test_health_is_ok_and_honest_without_kubernetes—/healthreturns 200 withk8s: false.Docs
adr_01moves from Proposed to Accepted, with the decision and the reasoning against Options 1 and 2 recorded. The three options are retained..agents/rules/engineering-rules.mdno longer claims the repos share a cluster, ingress and cloud.README.md: corrected the claims that the agent "runs as a Kubernetes pod" and queries K3s, and marked cluster-dependent features accordingly.Deployment note
No secrets or deployment credentials change. On merge the agent will log
Starting without Kubernetes...and reportk8s: false— that is the expected, correct output for this environment, not a regression.