harden(k3s): node-config bundle — secrets-encryption (secretbox) + CIS hardening - #89
Open
arunanshub wants to merge 5 commits into
Open
harden(k3s): node-config bundle — secrets-encryption (secretbox) + CIS hardening#89arunanshub wants to merge 5 commits into
arunanshub wants to merge 5 commits into
Conversation
etcd snapshots ship to Cloudflare R2 in plaintext today, so every Secret in etcd (Sealed-Secrets master key, hcloud token, Cloudflare tunnel token, SMTP/Grafana creds) is readable offline from a snapshot. Enable AES-CBC secrets encryption at rest to close that exposure in every future snapshot. This is the last outstanding item from the 2026-06-19 optimization sweep. New operator-supervised one-shot `ansible/playbooks/k3s-secrets-encryption.yml` encoding the doc-cited HA "enable on existing cluster" flow as five phased plays: precondition+enable -> flag rollout + rolling restart -> start-stage/ hash-match gate + rotate-keys -> post-rotate rolling restart -> verify Enabled. Safety: - Destructive CLI stages (enable, rotate-keys) are status-gated so an interrupted run resumes rather than double-applies. - Hash-match assert guards rotate-keys (k3s: mismatched-hash rotation can permanently corrupt the cluster). - Every rolling play is serial:1 + max_fail_percentage:0 (etcd quorum-safe); Ready-wait uses default([]) since the operator kubeconfig points at cp-1. - Requires -e snapshot_confirmed=true to force a pre-change etcd snapshot. Version gate: modern enable-on-existing-cluster flow needs v1.33.10+/v1.34.6+/v1.35.3+k3s1; cluster runs v1.36.2+k3s1 -> supported. Doc: https://docs.k3s.io/cli/secrets-encrypt Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DSsgEHGrf6oKAQoerbB5Lt
Set secrets-encryption-provider: secretbox (XSalsa20-Poly1305, AEAD) instead of the k3s default aescbc (AES-CBC, unauthenticated). Chosen before the first run: the Play 3 rotate-keys step lands all data on a secretbox key (the documented aescbc->secretbox migration), so we never pay a later rotate+reencrypt to switch providers. Free now, expensive after go-live. Provider support: k3s >= v1.32.4+k3s1; cluster v1.36.2+k3s1. Doc: https://docs.k3s.io/security/secrets-encryption Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DSsgEHGrf6oKAQoerbB5Lt
…pher) Fold k3s node-config hardening into the secrets-encryption PR. One drop-in, one rolling restart, delivered via a new ansible one-shot: - kube-apiserver: enable-admission-plugins=NodeRestriction (not in the default set; caps each kubelet to its own Node + bound pods — meaningful here since all 3 nodes are cp_worker so every kubelet holds node creds). - kubelet: streaming-connection-idle-timeout=5m + strong AEAD tls-cipher-suites (both real kubelet CLI flags per the k3s hardening guide). Safety: adds a LOCAL-apiserver /readyz gate per node, because a bad kube-apiserver-arg crash-loops the in-process apiserver while the kubelet still reports Ready via the LB — a plain node-Ready gate would mask it and march on. serial:1 + max_fail_percentage:0 halts before the next node. Doc: https://docs.k3s.io/security/hardening-guide Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DSsgEHGrf6oKAQoerbB5Lt
SafeDep Report SummaryNo dependency changes detected. Nothing to scan. This report is generated by SafeDep Github App |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



All k3s node-config tuning/hardening from the optimization sweep, bundled. Each is a separate operator-run ansible one-shot (you hold ansible/k8s access), delivered via the repo's established drop-in pattern (
config.yaml.d/*.yaml,serial:1,max_fail_percentage:0, node-Ready gate). Nothing auto-applies.1. Secrets encryption at rest —
ansible/playbooks/k3s-secrets-encryption.ymletcd snapshots ship to R2 (
infra/cloudflare_storage.tf) in plaintext today — every Secret (Sealed-Secrets master key, hcloud/Cloudflare tokens, SMTP/Grafana creds) is readable offline. This closes it. Last outstanding item from the 2026-06-19 sweep (Phase 4).enable→ flag rollout + rolling restart → start-stage/hash-match gate +rotate-keys→ post-rotate rolling restart → verifyEnabled/reencrypt_finished.secretbox(XSalsa20-Poly1305, AEAD) over the defaultaescbc(unauthenticated CBC). Set before first run sorotate-keyslands all data on a secretbox key — the documented aescbc→secretbox migration — at zero extra cost. Switching post-go-live would be a full key cycle.rotate-keys(k3s warns mismatched-hash rotation can permanently corrupt the cluster);-e snapshot_confirmed=trueforces a pre-change etcd snapshot.2. CIS hardening flags —
ansible/playbooks/k3s-cis-hardening.ymlOne drop-in, one rolling restart:
kube-apiserver --enable-admission-plugins=NodeRestriction— not in the default set; caps each kubelet to its own Node + bound pods. Meaningful here since all 3 nodes arecp_worker, so every kubelet holds node creds. Additive to apiserver defaults; hccm/Cilium/kured use their own SAs.kubelet --streaming-connection-idle-timeout=5m+ strong AEAD--tls-cipher-suites— both real kubelet CLI flags per the k3s hardening guide (nomax-parallel-image-pulls-style crash-loop)./readyzgate per node — a badkube-apiserver-argcrash-loops the in-process apiserver while the kubelet still reports Ready via the LB, so a plain node-Ready gate would mask it.serial:1halts before the next node.How to run (per playbook)
For secrets-encryption:
k3s etcd-snapshot save --name pre-secrets-encryptionfirst, thenjust ansible-converge k3s-secrets-encryption -e snapshot_confirmed=true.Deliberately NOT here
--disable local-storage,GOMEMLIMITgovernor) each carry a caveat (default-SC semantics / GC-thrash) — see the PR thread, opt-in.policyAuditMode → enforce, CoreDNS ArgoCD durability — non-k3s-node-config; separate PR.🤖 Generated with Claude Code
https://claude.ai/code/session_01DSsgEHGrf6oKAQoerbB5Lt