feat(executor): fall back to non-interruptible capacity on the last attempt

[chris-thomas-dev]

Why are the changes needed?

The Interruptible tasks docs describe a spot-to-on-demand fallback: the final attempt of an interruptible task runs on an on-demand instance, so a task whose nodes keep being reclaimed cannot burn its whole retry budget without ever landing on capacity that survives. The v2 executor schedules interruptible tasks on interruptible capacity for every attempt, so that fallback never happens. Requested in the review of #7885.

Note

Stacked on #7885 — the first commit is that PR (the fallback needs it: without it, env/task-level interruptible never reaches TaskExecutionMetadata). Only the last commit is new here. I'll rebase once #7885 merges.

What changes were proposed in this pull request?

A new executor config key, interruptibleFailureThreshold, ports flytepropeller's interruptible-failure-threshold semantics to the v2 executor:

• Positive values are an absolute count of burnt attempts; negative values are complementary to the maximum, so the default -1 moves only the final attempt to non-interruptible capacity. A value above the maximum disables the fallback; zero is rejected at startup.
• Either budget trips it: the task's own retries (against the template's retry count) or the system-failure budget (against maxSystemFailures), since a reclaimed node may be charged to either. This covers both of v1's IgnoreRetryCause branches without needing that flag.
• An interruptible task always gets at least one attempt on interruptible capacity, even with no retries.
• GetInterruptibleFailureThreshold() reports the configured value instead of a hardcoded 0, so connector-backed tasks receive the real threshold in TaskExecutionMetadata.
• DefaultMaxSystemFailures moved from executor/pkg/controller to executor/pkg/config (next to the field it defaults) so the plugin package can use it without an import cycle.

Notes for reviewers:

• With the -1 default, existing interruptible tasks change behaviour on upgrade: their final attempt moves to on-demand. This matches the documented behaviour and v1's default.
• Clusters need non-interruptible capacity (an on-demand node group/pool matching the non-interruptible-node-selector-requirement) or the fallback pod is unschedulable.
• config_flags.go was updated by hand in the generator's style; pflags isn't installed locally, so re-running go generate ./executor/pkg/config/ to confirm identical output would be a good check.

How was this patch tested?

Table tests over the threshold arithmetic (aboveInterruptibleFailureThreshold, interruptibleThresholdReached, maxSystemAttempts) and end-to-end NewTaskExecutionMetadata tests driving attempts/system-failures against a swapped-in config: last user attempt falls back, last system attempt falls back, single-attempt tasks keep their interruptible attempt, a threshold above both budgets disables the fallback, non-interruptible tasks are unaffected, and the threshold is reported to plugins. go test ./executor/pkg/plugin/ ./executor/pkg/config/ ./executor/ and go vet ./executor/... pass (the envtest suites were not run locally — no kubebuilder test binaries on this machine).

Labels

• added: For any new feature added.

Check all the applicable boxes

• I updated the documentation accordingly. (No docs change needed — this implements the documented spot-to-on-demand fallback.)
• All new and existing tests passed.
• All commits are signed-off.

🤖 Generated with Claude Code