fix(executor): honor task-level interruptible when the run-level flag is unset - #7885
Open
chris-thomas-dev wants to merge 1 commit into
Open
Conversation
… is unset The executor read interruptibility only from TaskAction.Spec.Interruptible, which is populated from the run-level RunSpec. Interruptible set at the TaskEnvironment, @env.task, or task.override() level serializes into the embedded TaskTemplate metadata but never reached pod scheduling, so the plugins.k8s interruptible node affinity and tolerations were not applied. The task template's metadata now decides interruptibility whenever the run-level flag is unset; an explicit run-level value still wins. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Chris Thomas <chris.john.thomas@gmail.com>
chris-thomas-dev
force-pushed
the
fix/executor-task-level-interruptible
branch
from
August 20, 2026 20:05
71828f4 to
185213d
Compare
3 tasks
Author
|
I would consider this PR is a pure bugfix, so I haven't implemented your on demand fallback doco alignment request in this pr, it has alot more scope because adds a config surface and changes default scheduling for existing interruptible tasks. I've implemented your requerst in #7911 (stacked on this PR, since the fallback needs task-level interruptible to reach TaskExecutionMetadata at all). It ports v1's interruptible-failure-threshold semantics: defaults to the last attempt on on-demand, configurable/disable-able via a new executor config key. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are the changes needed?
interruptibleset at theTaskEnvironmentor@env.tasklevel has no effect on pod scheduling; only the run-level flag (flyte.with_runcontext(interruptible=True), or aTrigger'sinterruptible) is honoured. The Interruptible tasks docs state thatinterruptiblecan be set at theTaskEnvironment,@env.task, andtask.override()levels.The SDK serialises env/task-level
interruptibleintoTaskTemplate.metadata.interruptible, and the template is embedded verbatim in the TaskAction CR. However, the executor buildsTaskExecutionMetadatafromTaskAction.Spec.Interruptibleonly, which is populated from the run-levelRunSpec(root action, inherited by child actions) — the embedded template's metadata is never consulted, soflytek8snever applies theplugins.k8sinterruptible node affinity/tolerations for env/task-level flags.Reproduced on EKS (self-hosted
flyte-binary-v2, chartv2.0.42) withplugins.k8s.interruptible-node-selector-requirement/non-interruptible-node-selector-requirement/interruptible-tolerationsconfigured against Karpenterkarpenter.sh/capacity-typelabels. No run-level flag was set for the env/task-level cases:TaskEnvironment(interruptible=True)capacity-type In [on-demand](non-interruptible) ❌@env.task(interruptible=True)capacity-type In [on-demand](non-interruptible) ❌with_runcontext(interruptible=True)(control)capacity-type In [spot]+ interruptible toleration ✅What changes were proposed in this pull request?
In
NewTaskExecutionMetadata, whenTaskAction.Spec.Interruptibleis unset, interruptibility now falls back to the embedded task template'smetadata.interruptible(a newinterruptibleFromTaskTemplatehelper, mirroring the existingmaxAttemptsFromTaskTemplate). An explicit run-level value still wins. The template'sinterruptible_valueoneof distinguishes "declared false" from "not declared", and both leave the task non-interruptible when the run-level flag is unset.How was this patch tested?
Added
TestNewTaskExecutionMetadata_Interruptiblecovering: default false with no flags; template-level true/false applying when the run level is unset; a template without the oneof leaving the default; and the run-level flag winning over template metadata in both directions.go test ./executor/pkg/plugin/andgo vet ./executor/...pass (theexecutor/test/integrationenvtest suite was not run locally — no kubebuilder test binaries on this machine).Labels
Check all the applicable boxes