Skip to content

harden(k3s): node-config bundle — secrets-encryption (secretbox) + CIS hardening - #89

Open
arunanshub wants to merge 5 commits into
masterfrom
claude/cluster-optimization-stack-2ulghp
Open

harden(k3s): node-config bundle — secrets-encryption (secretbox) + CIS hardening#89
arunanshub wants to merge 5 commits into
masterfrom
claude/cluster-optimization-stack-2ulghp

Conversation

@arunanshub

@arunanshub arunanshub commented Jul 14, 2026

Copy link
Copy Markdown
Owner

All k3s node-config tuning/hardening from the optimization sweep, bundled. Each is a separate operator-run ansible one-shot (you hold ansible/k8s access), delivered via the repo's established drop-in pattern (config.yaml.d/*.yaml, serial:1, max_fail_percentage:0, node-Ready gate). Nothing auto-applies.

1. Secrets encryption at rest — ansible/playbooks/k3s-secrets-encryption.yml

etcd snapshots ship to R2 (infra/cloudflare_storage.tf) in plaintext today — every Secret (Sealed-Secrets master key, hcloud/Cloudflare tokens, SMTP/Grafana creds) is readable offline. This closes it. Last outstanding item from the 2026-06-19 sweep (Phase 4).

  • Encodes the doc-cited HA enable-on-existing-cluster flow as five status-gated, resumable plays: precondition+enable → flag rollout + rolling restart → start-stage/hash-match gate + rotate-keys → post-rotate rolling restart → verify Enabled/reencrypt_finished.
  • Provider: secretbox (XSalsa20-Poly1305, AEAD) over the default aescbc (unauthenticated CBC). Set before first run so rotate-keys lands all data on a secretbox key — the documented aescbc→secretbox migration — at zero extra cost. Switching post-go-live would be a full key cycle.
  • Guards: hash-match assert before rotate-keys (k3s warns mismatched-hash rotation can permanently corrupt the cluster); -e snapshot_confirmed=true forces a pre-change etcd snapshot.
  • Version gate: enable-on-existing-cluster needs v1.33.10+/v1.34.6+/v1.35.3+k3s1; cluster is v1.36.2+k3s1. Provider needs ≥ v1.32.4+k3s1. Doc: https://docs.k3s.io/security/secrets-encryption

2. CIS hardening flags — ansible/playbooks/k3s-cis-hardening.yml

One drop-in, one rolling restart:

  • kube-apiserver --enable-admission-plugins=NodeRestriction — not in the default set; caps each kubelet to its own Node + bound pods. Meaningful here since all 3 nodes are cp_worker, so every kubelet holds node creds. Additive to apiserver defaults; hccm/Cilium/kured use their own SAs.
  • kubelet --streaming-connection-idle-timeout=5m + strong AEAD --tls-cipher-suites — both real kubelet CLI flags per the k3s hardening guide (no max-parallel-image-pulls-style crash-loop).
  • Safety: adds a local-apiserver /readyz gate per node — a bad kube-apiserver-arg crash-loops the in-process apiserver while the kubelet still reports Ready via the LB, so a plain node-Ready gate would mask it. serial:1 halts before the next node.
  • Doc: https://docs.k3s.io/security/hardening-guide

How to run (per playbook)

just ansible-check    <playbook>        # dry-run
just ansible-converge <playbook> [args] # apply, rolling

For secrets-encryption: k3s etcd-snapshot save --name pre-secrets-encryption first, then just ansible-converge k3s-secrets-encryption -e snapshot_confirmed=true.

Deliberately NOT here

  • Audit logging — deferred until there's a log pipeline (on-disk logs die with cattle nodes).
  • Memory levers — a dedicated doc-backed pass found the k3s daemon memory budget is small (~20–40 MB) and the real pressure is workload-side; the two candidate flags (--disable local-storage, GOMEMLIMIT governor) each carry a caveat (default-SC semantics / GC-thrash) — see the PR thread, opt-in.
  • PSA baseline, Cilium policyAuditMode → enforce, CoreDNS ArgoCD durability — non-k3s-node-config; separate PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DSsgEHGrf6oKAQoerbB5Lt

claude added 3 commits July 14, 2026 06:33
etcd snapshots ship to Cloudflare R2 in plaintext today, so every Secret
in etcd (Sealed-Secrets master key, hcloud token, Cloudflare tunnel token,
SMTP/Grafana creds) is readable offline from a snapshot. Enable AES-CBC
secrets encryption at rest to close that exposure in every future snapshot.

This is the last outstanding item from the 2026-06-19 optimization sweep.

New operator-supervised one-shot `ansible/playbooks/k3s-secrets-encryption.yml`
encoding the doc-cited HA "enable on existing cluster" flow as five phased
plays: precondition+enable -> flag rollout + rolling restart -> start-stage/
hash-match gate + rotate-keys -> post-rotate rolling restart -> verify Enabled.

Safety:
- Destructive CLI stages (enable, rotate-keys) are status-gated so an
  interrupted run resumes rather than double-applies.
- Hash-match assert guards rotate-keys (k3s: mismatched-hash rotation can
  permanently corrupt the cluster).
- Every rolling play is serial:1 + max_fail_percentage:0 (etcd quorum-safe);
  Ready-wait uses default([]) since the operator kubeconfig points at cp-1.
- Requires -e snapshot_confirmed=true to force a pre-change etcd snapshot.

Version gate: modern enable-on-existing-cluster flow needs
v1.33.10+/v1.34.6+/v1.35.3+k3s1; cluster runs v1.36.2+k3s1 -> supported.
Doc: https://docs.k3s.io/cli/secrets-encrypt

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DSsgEHGrf6oKAQoerbB5Lt
Set secrets-encryption-provider: secretbox (XSalsa20-Poly1305, AEAD)
instead of the k3s default aescbc (AES-CBC, unauthenticated). Chosen
before the first run: the Play 3 rotate-keys step lands all data on a
secretbox key (the documented aescbc->secretbox migration), so we never
pay a later rotate+reencrypt to switch providers. Free now, expensive
after go-live. Provider support: k3s >= v1.32.4+k3s1; cluster v1.36.2+k3s1.
Doc: https://docs.k3s.io/security/secrets-encryption

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DSsgEHGrf6oKAQoerbB5Lt
…pher)

Fold k3s node-config hardening into the secrets-encryption PR. One drop-in,
one rolling restart, delivered via a new ansible one-shot:

- kube-apiserver: enable-admission-plugins=NodeRestriction (not in the
  default set; caps each kubelet to its own Node + bound pods — meaningful
  here since all 3 nodes are cp_worker so every kubelet holds node creds).
- kubelet: streaming-connection-idle-timeout=5m + strong AEAD tls-cipher-suites
  (both real kubelet CLI flags per the k3s hardening guide).

Safety: adds a LOCAL-apiserver /readyz gate per node, because a bad
kube-apiserver-arg crash-loops the in-process apiserver while the kubelet
still reports Ready via the LB — a plain node-Ready gate would mask it and
march on. serial:1 + max_fail_percentage:0 halts before the next node.

Doc: https://docs.k3s.io/security/hardening-guide

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DSsgEHGrf6oKAQoerbB5Lt
@arunanshub arunanshub changed the title harden(k3s): add secrets-encryption at-rest playbook (Phase 4) harden(k3s): node-config bundle — secrets-encryption (secretbox) + CIS hardening Jul 14, 2026
@safedep

safedep Bot commented Jul 27, 2026

Copy link
Copy Markdown

SafeDep Report Summary

Green Malicious Packages Badge Green Vulnerable Packages Badge Green Risky License Badge

No dependency changes detected. Nothing to scan.

View complete scan results →

This report is generated by SafeDep Github App

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants