fix(controller): record pinned executions before force-deleting versions on teardown

[tczhao]

TWD teardown can permanently strand in-flight customer workflows, and today it leaves no record of which ones. This makes it leave one.

handleDeletion force-deletes every worker deployment version with SkipDrainage: true. That flag bypasses the one server-side guard Temporal provides against deleting a version that still has open pinned executions. Those workflows are then unrecoverable without manual intervention: their tasks route by exact deployment-version match, and a task on a base (non-sticky) task queue has no schedule-to-start timeout, so once the version is gone nothing dispatches them and nothing expires them. They sit at attempt: 1 forever.

This fired on 2026-08-31 and stranded six customer workflows. One had been stranded since 08-27 and nobody noticed, because there was nothing to notice.

What this changes

Before each force-delete, query visibility for open executions pinned to that version:

ExecutionStatus="Running" AND TemporalWorkerDeploymentVersion="<deployment>:<buildID>"

If the count is non-zero, log the count, a sample of workflow IDs, and the query itself, then emit a Warning Event (PinnedExecutionsStranded).

The delete still proceeds. Honouring drainage here would hold the TWD in Terminating for as long as the longest pinned workflow runs - a 72-hour Snowflake crawl blocks the namespace and any reinstall for 72 hours. That trades a silent data-integrity failure for a loud availability failure. This PR does not make that trade; it only stops the failure being silent. Bounded drainage is ARUN-1232 item 2 and is deliberately out of scope here.

Why the log line, not a status condition

ARUN-1232 proposes setting a status condition. That does not work on this path and the ticket should be read with this correction:

• This runs inside the finalizer. On success the finalizer is removed and the API server deletes the TWD immediately, so a condition written here survives for microseconds.
• Cluster Events expire on the apiserver's event-ttl, commonly 1h. Against an incident that went unnoticed for 20 hours and 5 days, that is weak.
• Controller logs are already scraped into otel_logs.filelogs_deployment and are exactly where the DISTR-947 detection queries look.

So the log line is the contract. The Event is the human-facing nudge. The status condition belongs to item 2, where the object stays around to carry it.

Grep target for detection:

msg = "force-deleting worker deployment version with open pinned executions"

Deliberate choices

• Visibility failure never blocks teardown. It reports that the check could not run (PinnedExecutionCheckFailed) and returns. Proceeding keeps teardown reliable at the cost of reintroducing the silent case exactly when Temporal is unhealthy - the alternative risks wedging teardown on an unrelated outage, which is the failure mode ARUN-1182 is already about.
• planner.go is untouched. The steady-state reaper only deletes VersionStatusDrained, which by definition has no open pinned executions. Adding the check there would be cost with no signal.
• Query precision. Uses the query validated in DISTR-947 rather than also filtering TemporalWorkflowVersioningBehavior="Pinned". AutoUpgrade workflows on a doomed version are not stranded and will inflate the count, but the extra built-in search attribute may not be registered on every namespace, and a failed query degrades to the check-failed path. Over-reporting on a warning is the safer side to err on.
• Cost. Two visibility RPCs per version per reconcile attempt. Teardown requeues every 10s while pollers persist (58 attempts over ~20 minutes in the 08-31 incident), so a stuck teardown could issue a few thousand count queries. ES count queries are cheap and this is a rare path, but flagging it explicitly.

Testing

go test ./internal/controller/... - 137 pass. Four new tests in teardown_pinned_test.go cover: open pinned executions logged and evented with IDs; nothing pinned stays silent and skips the list call; count failure proceeds with a warning; list failure still reports the count.

handleDeletion itself is not unit-testable - it dials the client pool directly - so "the delete still proceeds" is guaranteed structurally by recordPinnedExecutions having no return value, not by a test.

make lint-code reports one pre-existing finding, execplan.go:355 cyclomatic complexity, identical on the base branch and in an untouched file.

Scope

No CRD change, no API change, no new flag, no behaviour change to the delete itself. Should rebase cleanly and is a reasonable upstream candidate.

Refs ARUN-1232. The trigger for the accidental teardown is tracked separately in DISTR-947 (Flux remediation.strategy: uninstall), which is still necessary - this PR does not remove the need for it.

🤖 Generated with Claude Code

[linear]

ARUN-1232