exporterset: disable-before-delete grace for ExitAndReplace recycle

[mangelajo]

Summary

Follow-up to #929: cleanupTerminalExporters currently deletes unleased exporters as soon as their Pods are Succeeded/Failed. That races with the lease controller (separate process): after LeaseRef clears, the instance can still briefly look Online/Available and accept a new lease while it is being deleted, especially if unregister/Offline lags or the process dies uncleanly.

Scale-down already does disable → later delete. ExitAndReplace should get a similar fence without bloating #929.

Goals

1. Fence new leases by setting spec.enabled: false before delete.
2. Only delete after Offline OR disabled for ≥ 10s (whichever comes first).
3. Keep a single owner for recycle final cleanup: cleanupTerminalExporters.

Proposed design

Ownership (prefer B)

Helper	Role
cleanupTerminalExporters	Only place that deletes ExitAndReplace recycle instances: disable+annotate → wait → delete
cleanupDisabledExporters	Scale-down only; skip exporters stamped for recycle (comment: owned by cleanupTerminalExporters)

Do not teach cleanupDisabledExporters the grace logic — otherwise disable on reconcile N would be deleted immediately on N+1 with no wait.

Behavior for terminal + unleased exporters

State	Action
Still enabled	Set enabled=false, stamp recycle annotation, RequeueAfter: 10s
Disabled + (Online=False or stamp age ≥ 10s) + still unleased	Provisioner cleanup + delete
Disabled + still Online + stamp age < 10s	RequeueAfter remaining
LeaseRef set	Do not delete

Annotation

When recycle disables an exporter, stamp e.g.:

exporterset.jumpstarter.dev/recycle-disabled-at: <RFC3339>

Comment at the annotate/disable site that final delete is exclusively cleanupTerminalExporters, and that cleanupDisabledExporters must skip this annotation.

Offline check

Treat as offline when Online condition is False and/or exporterStatus == Offline (same signals unregister / lease filtering use).

Why 10s + Offline OR

• Disable → lease controller stops selecting the exporter immediately.
• Offline → fast path when unregister succeeds.
• 10s floor → crash/unregister lag safety net (LastSeen Online timeout is ~1m; we should not wait that long to recycle).

Implementation sketch

1. Add annotation constant + helpers (recycleDisabledAt, readyToDeleteRecycledExporter).
2. Update cleanupTerminalExporters for disable → wait → delete; return RequeueAfter when waiting.
3. Update cleanupDisabledExporters to continue when recycle annotation is present (with comment).
4. Unit tests in reconciler_test.go:
   • terminal + enabled → disabled, not deleted
   • disabled < 10s + Online → not deleted
   • disabled + Offline before 10s → deleted
   • disabled ≥ 10s + still Online → deleted
   • lease after disable → not deleted
   • cleanupDisabledExporters skips recycle-stamped exporters
5. Short JEP-0014 / recycle doc note aligning with scale-down disable-before-delete.

Out of scope / non-goals

• Changing scale-down cooldown semantics (unless we later unify all disabled deletes under the same grace — optional polish, not required).
• Moving lease assignment into the exporterset controller.

References

• PR feat: exporter-as-main QEMU pods with ExitAndReplace recycle e2e #929 (cleanupTerminalExporters)
• Race analysis: lease assignment runs in jumpstarter-controller; recycle/delete in exporter-set-controller — parallel even with single replicas / MaxConcurrentReconciles=1 per controller.