Protect CHK recreate with live Raft quorum safety

[alex-zaitsev]

Summary

Fixes #2069 (CHK Raft quorum safety during rolls) and #2035 (drop pointless same-size settle delay). Partial #2059 (propagate reconcile status persist errors).

Quorum-safe Keeper rolls (#2069)

On a multi-node CHK, the operator must not take a Ready replica down if that would leave the ensemble below Raft majority, and must not keep rolling the next replica while a previous one never rejoined.

Behavior:

• Before disrupting a host, classify the pass from live Ready counts (not CR ancestor): rolling when the ensemble already has quorum (or is single-node); bootstrap when it does not.
• Rolling waits for Ready; bootstrap waits for Started only so siblings can come up together. The rolling/bootstrap decision is frozen for the host pass so a force-restart (ReadyReplicas→0) cannot flip the pass into bootstrap mid-flight.
• If disrupting this host would break quorum, wait briefly for headroom (poll ~5s, up to ~2m). If still unsafe, defer that host ([RaftQuorumUnsafe]), continue other replicas, then soft-requeue in 5s (not error backoff).
• Prefer recovering not-Ready replicas before rolling Ready peers (helps interrupted rolls and tight ensembles).
• STS create/update wait failure aborts the reconcile instead of ignoring and walking to the next replica (ClickHouseKeeperInstallation controller can recreate every replica of an ensemble without keeping a Raft quorum running #2069 root cause).

Out of scope here: committed Raft membership / mntr barriers (#2041 — hook only), staged rescale (e.g. 3→2→1), and replacing blind downscale sleeps with per-step settle.

Settle delay and status (#2035 / partial #2059)

• Same-size reconciles no longer sleep 10s.
• Downscale still uses the existing 120s pre-settle + 60s post-purge pauses.
• Reconcile start/completion status update failures are returned to the controller; completion metrics fire only after Completed is persisted.

vs CHI shard safety (#1704)

	CHI	CHK (this PR)
Gate	≥1 healthy peer in the shard	Raft majority of Ready members
On refuse	Defer	Wait up to ~2m, then defer + 5s soft requeue
Snapshot before disrupt	No	Yes (survives force-restart Ready drop)
Recovery-first	Yes	Yes

Test plan

• Unit: go test ./pkg/controller/chk/
• Regression: test_020005 xfail removed
• E2e: test_020003 / test_020005 (keeper upgrade / scale)
• E2e: test_020003_3 — 3-node CHK, broken-image roll stops on one replica, peers stay Ready through operator restart, recovery to good image (CHI test_010083 analogue)
• Close fix(chk): let healthy keeper reconciles complete #2059 after merge (remaining items intentionally not included)

Follow-ups

• Staged Raft rescale (one peer per step)
• Per-step quorum settle instead of fixed 120s+60s on downscale
• Full Raft-safe ClickHouseKeeperInstallation rescale #2041 membership barrier after join