Skip to content

fix(executor): honor task-level interruptible when the run-level flag is unset - #7885

Open
chris-thomas-dev wants to merge 1 commit into
flyteorg:mainfrom
chris-thomas-dev:fix/executor-task-level-interruptible
Open

fix(executor): honor task-level interruptible when the run-level flag is unset#7885
chris-thomas-dev wants to merge 1 commit into
flyteorg:mainfrom
chris-thomas-dev:fix/executor-task-level-interruptible

Conversation

@chris-thomas-dev

Copy link
Copy Markdown

Why are the changes needed?

interruptible set at the TaskEnvironment or @env.task level has no effect on pod scheduling; only the run-level flag (flyte.with_runcontext(interruptible=True), or a Trigger's interruptible) is honoured. The Interruptible tasks docs state that interruptible can be set at the TaskEnvironment, @env.task, and task.override() levels.

The SDK serialises env/task-level interruptible into TaskTemplate.metadata.interruptible, and the template is embedded verbatim in the TaskAction CR. However, the executor builds TaskExecutionMetadata from TaskAction.Spec.Interruptible only, which is populated from the run-level RunSpec (root action, inherited by child actions) — the embedded template's metadata is never consulted, so flytek8s never applies the plugins.k8s interruptible node affinity/tolerations for env/task-level flags.

Reproduced on EKS (self-hosted flyte-binary-v2, chart v2.0.42) with plugins.k8s.interruptible-node-selector-requirement / non-interruptible-node-selector-requirement / interruptible-tolerations configured against Karpenter karpenter.sh/capacity-type labels. No run-level flag was set for the env/task-level cases:

Flag level Pod affinity stamped
TaskEnvironment(interruptible=True) capacity-type In [on-demand] (non-interruptible) ❌
@env.task(interruptible=True) capacity-type In [on-demand] (non-interruptible) ❌
with_runcontext(interruptible=True) (control) capacity-type In [spot] + interruptible toleration ✅

What changes were proposed in this pull request?

In NewTaskExecutionMetadata, when TaskAction.Spec.Interruptible is unset, interruptibility now falls back to the embedded task template's metadata.interruptible (a new interruptibleFromTaskTemplate helper, mirroring the existing maxAttemptsFromTaskTemplate). An explicit run-level value still wins. The template's interruptible_value oneof distinguishes "declared false" from "not declared", and both leave the task non-interruptible when the run-level flag is unset.

How was this patch tested?

Added TestNewTaskExecutionMetadata_Interruptible covering: default false with no flags; template-level true/false applying when the run level is unset; a template without the oneof leaving the default; and the run-level flag winning over template metadata in both directions. go test ./executor/pkg/plugin/ and go vet ./executor/... pass (the executor/test/integration envtest suite was not run locally — no kubebuilder test binaries on this machine).

Labels

  • fixed: For any bug fixed.

Check all the applicable boxes

  • I updated the documentation accordingly. (No docs change needed — this makes behaviour match the existing docs.)
  • All new and existing tests passed.
  • All commits are signed-off.

… is unset

The executor read interruptibility only from TaskAction.Spec.Interruptible,
which is populated from the run-level RunSpec. Interruptible set at the
TaskEnvironment, @env.task, or task.override() level serializes into the
embedded TaskTemplate metadata but never reached pod scheduling, so the
plugins.k8s interruptible node affinity and tolerations were not applied.

The task template's metadata now decides interruptibility whenever the
run-level flag is unset; an explicit run-level value still wins.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Chris Thomas <chris.john.thomas@gmail.com>
@chris-thomas-dev
chris-thomas-dev force-pushed the fix/executor-task-level-interruptible branch from 71828f4 to 185213d Compare August 20, 2026 20:05

@popojk popojk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the doc, we should schedule the pod to on demand node in the last attempt. Mind adding that in the PR?

@chris-thomas-dev

Copy link
Copy Markdown
Author

I would consider this PR is a pure bugfix, so I haven't implemented your on demand fallback doco alignment request in this pr, it has alot more scope because adds a config surface and changes default scheduling for existing interruptible tasks. I've implemented your requerst in #7911 (stacked on this PR, since the fallback needs task-level interruptible to reach TaskExecutionMetadata at all). It ports v1's interruptible-failure-threshold semantics: defaults to the last attempt on on-demand, configurable/disable-able via a new executor config key.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants