fix(tdarr): restore B70 VA-API transcoding via name-faithful device group - #1490
Merged
Conversation
tdarr-node ran transcodegpuWorkers=1 / transcodecpuWorkers=0, so when the B70's VA-API path broke on 2026-08-26 every transcode job failed with transcodeError and no worker could take over. With the library already at 99.28% the UI showed an idle, healthy server, so a total transcoding outage stayed invisible for three days. Setting transcodecpuWorkers=1 restores transcoding today, independent of the GPU investigation. A GPU-only node is the structural cause of the silence, not just this incident's bad luck.
…aapi generic-device-plugin's b70 group renames the B70's DRM nodes to card0/renderD128 via mountPath. That rename is fatal to VA-API. libdrm does not trust the path it is handed: it fstat()s the fd, reads /sys/dev/char/226:129/uevent, re-derives the canonical DEVNAME (dri/renderD129) and reopens that path. Inside a container that mounted the same device as renderD128 the canonical path does not exist, so vaGetDisplayDRM() fails before any driver loads. Proven live by A-B-A inside the running pod, one variable: creating a renderD129 symlink made the unchanged renderD128 path work, removing it reproduced "Device creation failed: -542398533", restoring it worked again. This disproves the competing hypothesis that libva 2.23.0 plus intel-media-va-driver-non-free 26.2.2 is incompatible with Battlemage; the same driver initialises fine as soon as the name matches. Add a second device-plugin group exposing the same by-path nodes at the kernel's own names and point tdarr-node at it. A second group rather than an edit to b70 because device IDs are sha1(count + every host path in the group), so adding paths to b70 would change all 99 IDs and invalidate kubelet's live allocations for vllm and comfyui. Verified live: devic.es/b70 stayed at 99 and vllm held 0 restarts across the plugin rollout. Validation: a real 25-minute 1080p library file transcoded end to end with the exact av1_qsv invocation from the failed job report, at ~13x realtime, and the output decodes with zero errors. Also document the verification step whose absence let this ship: #1443 updated the media and GPU docs but neither checked that VA-API still worked.
The config is mounted with subPath, and kubelet never refreshes a subPath mount. Combined with disableNameSuffixHash: true, which pins the ConfigMap name so the pod template never changes, a config-only edit could not reach the running plugin at all: it stayed inert until someone restarted the DaemonSet by hand. The b70-vaapi group added in the previous commit would have taken effect only by luck. kustomizeconfig.yaml already declares the nameReference that rewrites the HelmRelease's persistence.config.name, which is exactly the machinery a content hash needs, so enabling the hash is a two-line change: the ConfigMap name now tracks its content, the HelmRelease follows it, and the pod template change rolls the DaemonSet. Restarting this DaemonSet does not disturb pods that already hold allocated devices - verified live while adding b70-vaapi, where vllm kept its B70 slot and 0 restarts across the rollout.
Renaming a DRM node via generic-device-plugin's mountPath silently breaks VA-API while leaving Level Zero working, so the AI stack stays green while transcoding is dead. That asymmetry is what hid a 3-day total Tdarr outage. Capture it next to the other cluster-wide traps, along with the two related device-plugin footguns found while fixing it: group device IDs are derived from the group's host paths, and the config is a subPath mount that kubelet never refreshes.
The GPU fix alone does not resume transcoding. All three Talos nodes had been excluded from both Tdarr libraries as a post-outage mitigation, so the node would have picked up zero jobs after merge. That scoping is per-node Tdarr server state (librariesToNotProcess in NodeJSONDB), not GitOps, so it cannot ship in this PR - only be documented by it. Per captain decision, only the Movies AV1 exclusion is cleared on talos-3. The Series exclusion is deliberately retained so the restore covers one library at a time while the VA-API fix is newly landed. The eight files in Tdarr's error table, mostly 21-67 GB 2160p DV/HDR10 remux masters, are recorded as a separate follow-up. Their failure is not explained by the VA-API break alone, and Tdarr rewrites in place with a lossy irreversible AV1 result, so they must not be bulk-requeued to tidy the error table. Verified that re-enabling a library does not requeue them: it is a node-side accept filter, and all 8 stayed parked with the queue at 0.
Contributor
--- HelmRelease media/tdarr
+++ HelmRelease media/tdarr
@@ -132,7 +132,7 @@
- name: transcodegpuWorkers
value: "1"
- name: transcodecpuWorkers
- value: "0"
+ value: "1"
- name: healthcheckgpuWorkers
value: "1"
- name: healthcheckcpuWorkers
@@ -142,7 +142,7 @@
tag: 2.86.01
resources:
limits:
- devic.es/b70: 1
+ devic.es/b70-vaapi: 1
memory: 4Gi
requests:
cpu: 100m
--- ConfigMap system/generic-device-plugin
+++ ConfigMap system/generic-device-plugin
@@ -1,34 +1 @@
-apiVersion: v1
-data:
- config.yaml: |
- ---
- devices:
- - name: tun
- groups:
- - count: 1000
- paths:
- - path: /dev/net/tun
- # Discrete Arc Pro B70 on talos-3 (Battlemage G31, PCI 8086:e223 @ 0000:03:00.0).
- # Intel GpuDevicePlugin cannot split xe KMD devices into distinct extended
- # resources (both iGPU and B70 would be gpu.intel.com/xe), so this plugin
- # owns the B70 under its own name. Discover via stable DRM by-path nodes and
- # re-expose them at the conventional card0/renderD128 paths Level Zero expects
- # inside the container. count matches the former sharedDevNum share-token model
- # so vllm/vllm-embed/comfyui/tdarr-node can still co-schedule concurrently.
- - name: b70
- groups:
- - count: 99
- paths:
- - path: /dev/dri/by-path/pci-0000:03:00.0-card
- mountPath: /dev/dri/card0
- - path: /dev/dri/by-path/pci-0000:03:00.0-render
- mountPath: /dev/dri/renderD128
-kind: ConfigMap
-metadata:
- labels:
- app.kubernetes.io/name: generic-device-plugin
- kustomize.toolkit.fluxcd.io/name: generic-device-plugin
- kustomize.toolkit.fluxcd.io/namespace: system
- name: generic-device-plugin
- namespace: system
--- ConfigMap system/generic-device-plugin-kkk465kb69
+++ ConfigMap system/generic-device-plugin-kkk465kb69
@@ -1 +1,66 @@
+apiVersion: v1
+data:
+ config.yaml: |
+ ---
+ devices:
+ - name: tun
+ groups:
+ - count: 1000
+ paths:
+ - path: /dev/net/tun
+ # Discrete Arc Pro B70 on talos-3 (Battlemage G31, PCI 8086:e223 @ 0000:03:00.0).
+ # Intel GpuDevicePlugin cannot split xe KMD devices into distinct extended
+ # resources (both iGPU and B70 would be gpu.intel.com/xe), so this plugin
+ # owns the B70 under its own name. Discover via stable DRM by-path nodes and
+ # re-expose them at the conventional card0/renderD128 paths Level Zero expects
+ # inside the container. count matches the former sharedDevNum share-token model
+ # so vllm/vllm-embed/comfyui can still co-schedule concurrently. tdarr-node
+ # uses the separate b70-vaapi group below (VA-API needs the kernel names).
+ - name: b70
+ groups:
+ - count: 99
+ paths:
+ - path: /dev/dri/by-path/pci-0000:03:00.0-card
+ mountPath: /dev/dri/card0
+ - path: /dev/dri/by-path/pci-0000:03:00.0-render
+ mountPath: /dev/dri/renderD128
+ # Same physical B70, exposed under the device names the KERNEL gives it.
+ #
+ # The b70 group above renames the nodes to card0/renderD128, and that rename
+ # is fatal to VA-API. libdrm does not trust the path you hand it: it fstat()s
+ # the fd, reads /sys/dev/char/<major>:<minor>/uevent, and re-derives the
+ # canonical DEVNAME (dri/renderD129 for this card). It then reopens
+ # /dev/dri/renderD129 - which does not exist in a container that mounted the
+ # same device as renderD128 - so vaGetDisplayDRM() fails before any driver
+ # loads. Proven live 2026-08-29: creating a renderD129 symlink inside the
+ # running tdarr-node made the UNCHANGED renderD128 path work, and removing it
+ # broke it again (A-B-A, single variable). See docs/media-stack.md.
+ #
+ # Level Zero (vllm, comfyui) opens whatever /dev/dri/renderD* it finds and is
+ # unaffected either way, so the b70 group is left byte-identical: its device
+ # IDs are sha1(count + host paths), so editing its paths would change all 99
+ # IDs and invalidate kubelet's live allocations for the running AI pods.
+ #
+ # NOTE: mountPath here must track the kernel's enumeration of this card. The
+ # iGPU (0000:00:02.0) probes first and takes card0/renderD128, so the discrete
+ # card at 0000:03:00.0 is card1/renderD129. If that order ever changes, VA-API
+ # breaks again - tdarr then degrades to its CPU worker rather than failing
+ # every job. Verify with the vainfo check in docs/media-stack.md after any GPU
+ # or device-plugin change.
+ - name: b70-vaapi
+ groups:
+ - count: 99
+ paths:
+ - path: /dev/dri/by-path/pci-0000:03:00.0-card
+ mountPath: /dev/dri/card1
+ - path: /dev/dri/by-path/pci-0000:03:00.0-render
+ mountPath: /dev/dri/renderD129
+kind: ConfigMap
+metadata:
+ labels:
+ app.kubernetes.io/name: generic-device-plugin
+ kustomize.toolkit.fluxcd.io/name: generic-device-plugin
+ kustomize.toolkit.fluxcd.io/namespace: system
+ name: generic-device-plugin-kkk465kb69
+ namespace: system
--- HelmRelease system/generic-device-plugin
+++ HelmRelease system/generic-device-plugin
@@ -63,7 +63,7 @@
- path: /config/config.yaml
readOnly: true
subPath: config.yaml
- name: generic-device-plugin
+ name: generic-device-plugin-kkk465kb69
type: configMap
dev:
globalMounts:
|
Contributor
--- Deployment media/tdarr-tdarr-node
+++ Deployment media/tdarr-tdarr-node
@@ -55,7 +55,7 @@
- name: transcodegpuWorkers
value: "1"
- name: transcodecpuWorkers
- value: "0"
+ value: "1"
- name: healthcheckgpuWorkers
value: "1"
- name: healthcheckcpuWorkers
@@ -64,7 +64,7 @@
name: app
resources:
limits:
- devic.es/b70: 1
+ devic.es/b70-vaapi: 1
memory: 4Gi
requests:
cpu: 100m
--- DaemonSet system/generic-device-plugin
+++ DaemonSet system/generic-device-plugin
@@ -66,7 +66,7 @@
serviceAccountName: generic-device-plugin
volumes:
- configMap:
- name: generic-device-plugin
+ name: generic-device-plugin-kkk465kb69
name: config
- hostPath:
path: /dev
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Restore Tdarr transcoding. Tdarr's GPU path had been failing on every job since 2026-08-26 with no fallback, so nothing transcoded at all. The scout report isolated it to PR #1443, which re-plumbed the Arc B70 through generic-device-plugin (which renames the render node), and reproduced live that VA-API can no longer open a device. But the scout explicitly did NOT prove the failing call, and left two live hypotheses: (a) the name-to-minor/sysfs mismatch from the devic.es/b70 remap (leading, but unproven), or (b) a driver incompatibility in the tdarr_node image itself (libva 2.23.0 + intel-media-va-driver-non-free 26.2.2 vs Battlemage). I was told NOT to treat the leading hypothesis as established, and to discriminate between them.
Required work, in order: (1) Stop the outage first, before any investigation: set transcodecpuWorkers to 1 in the tdarr helmrelease, shipped regardless of findings, and as its own commit if it could not fit cleanly with the rest so relief was not blocked behind investigation. (2) Run the discriminating test by exposing the same by-path nodes at their NATIVE names (/dev/dri/card1, /dev/dri/renderD129) via generic-device-plugin and pointing tdarr-node at that; if VA-API then works the rename is confirmed and the fix is to give VA-API consumers a name-faithful device while Level Zero consumers keep the remapped one; if it still fails, report the driver hypothesis as live and stop rather than guessing at image pins. Use the exact vainfo and ffmpeg commands the report quotes so results are comparable. (3) Close the documentation gap that let this ship: #1443 updated the media and GPU docs but neither mentions verifying VA-API after the switch, so add that verification step with the concrete command.
Constraints: do not touch the B70 itself and do not reboot or upgrade any node (it has separate history and pending attended maintenance; reading its state is fine). Do not revert PR #1443, which was deliberate GPU work other consumers depend on. Proof of success must be what actually transcoded successfully, not that the pod is running.
Outcome: the rename hypothesis is PROVEN and the driver hypothesis DISPROVEN, by an A-B-A test inside the running pod holding everything else constant: creating a file merely named renderD129 made the UNCHANGED renderD128 path succeed, removing it reproduced 'Device creation failed: -542398533', restoring it succeeded again. Mechanism: libdrm ignores the path passed to it, fstat()s the fd, reads /sys/dev/char/226:129/uevent, re-derives the canonical DEVNAME (dri/renderD129) and reopens that path, which a container that mounted the device as renderD128 does not have, so vaGetDisplayDRM() fails before any driver loads. Proof of transcode: a real 25-minute 1080p library file transcoded end to end with the exact av1_qsv -preset medium -global_quality 28 -look_ahead 1 invocation from the failed job report, exit 0, 36961 frames at 12.8x realtime, output decodes with zero errors.
Deliberate decisions a reviewer reading only the diff would not know: (i) The fix adds a SEPARATE generic-device-plugin group (b70-vaapi) rather than adding native-name paths to the existing b70 group, because that plugin derives device IDs as sha1(count_index + every resolved host path in the group), so editing b70 would change all 99 device IDs and invalidate kubelet's live allocations for the running vllm and comfyui pods. Verified live that a new group leaves them untouched: devic.es/b70 stayed at 99 and vllm held 0 restarts across the plugin rollout. (ii) Enabling the configMapGenerator name-suffix hash is required for the fix to work at all, not cosmetic: the config is a subPath mount, which kubelet never refreshes, and disableNameSuffixHash pinned the ConfigMap name so the pod template never changed, meaning a config-only edit could never reach the running plugin without a manual DaemonSet restart. kustomizeconfig.yaml already declared the nameReference that a content hash needs. Verified live that the merge path works: the resource registered automatically within 15s with no manual step. (iii) mountPath in the new group hardcodes the kernel's current enumeration (iGPU at 0000:00:02.0 probes first and takes card0/renderD128, leaving the discrete card at card1/renderD129); this is a known coupling, documented, and the CPU fallback means a future break degrades instead of causing a total outage. (iv) The live cluster was deliberately left exactly matching main rather than carrying the fix as drift, so the change lands cleanly at merge.
Later captain decision on service-restore scope, which the docs in this change record: the GPU fix alone does not resume transcoding, because all three Talos nodes had been excluded from BOTH Tdarr libraries as a post-outage mitigation. Only the Movies AV1 exclusion was cleared on talos-3; the Series exclusion is deliberately RETAINED so the restore covers one library at a time while the VA-API fix is newly landed. The eight files in Tdarr's error table, mostly 21-67 GB 2160p DV/HDR10 remux masters, must NOT be fed back in as a side effect of restoring service: their previous failure is unexplained and Tdarr rewrites in place with a lossy irreversible AV1 result, so they are recorded as a separate follow-up needing their failure understood first. Verified that re-enabling a library does not requeue them (it is a node-side accept filter; all 8 stayed parked with the transcode queue at 0). Both of these are per-node Tdarr server state (librariesToNotProcess in NodeJSONDB), not GitOps, so they cannot ship in this PR and are only documented by it.
What Changed
devic.es/b70-vaapi(kernel-canonicalcard1/renderD129mounts) instead of the renameddevic.es/b70group, and settranscodecpuWorkersto1so a future VA-API regression degrades instead of stalling every job.b70-vaapigeneric-device-plugin group and re-enable the configMapGenerator name-suffix hash so config-only device-plugin changes roll out without rewriting the existingb70group IDs used by vllm/comfyui.Risk Assessment
✅ Low: Durable, intent-complete fix: separate name-faithful
b70-vaapigroup, CPU fallback, hash-enabled plugin rollout, and docs/proof of the libdrm rename failure, without touching the sharedb70allocation set.Testing
Added and ran a semantic b70-vaapi/tdarr regression test plus an updated iGPU allowIDs consumer split; both pass on HEAD, the new test fails on the pre-fix base, and kustomize-built output shows native-name b70-vaapi mounts with a hashed ConfigMap reference, but fresh live transcode evidence could not be captured without cluster access.
Evidence: b70-vaapi-tdarr-test HEAD evidence (PASS + libdrm A-B-A)
Evidence: b70-vaapi-tdarr-test BASE pre-fix failure
Evidence: Rendered generic-device-plugin + tdarr contract summary
configMap=generic-device-plugin-kkk465kb69; b70→card0/renderD128; b70-vaapi→card1/renderD129; tdarr-node limits devic.es/b70-vaapi=1; transcodecpuWorkers=1; libdrm A/B/A2 reopenOk=true/false/trueEvidence: kustomize-built generic-device-plugin manifests
Evidence: igpu-xe-allowids-test HEAD evidence
Evidence: Combined test transcript
Live verification evidence
Run against the live cluster on 2026-08-29, before the branch state was reverted so the
cluster would match
main. Commands verbatim.-n media exec deploy/tdarr-tdarr-node -c appthroughout; the pipeline worktree has no cluster access, so this is recorded here to be durable.
1. Baseline failure reproduced (the reported symptom)
2. The mechanism: names do not match minors, and libdrm reopens the canonical name
3. A-B-A controlled test: one variable, the existence of a file named
renderD129The device passed to ffmpeg is
renderD128in every run below. Nothing about the driver,the card, or the argument changes; only whether
/dev/dri/renderD129exists.PASS -> FAIL -> PASS on one variable. This proves the rename is the cause and
disproves the competing hypothesis that
libva 2.23.0+intel-media-va-driver-non-free 26.2.2is incompatible with Battlemage: the same driver initialises fine the moment thename matches.
4. Driver and card identity confirmed (it really is the B70, with AV1 encode)
5. Post-fix device layout, from the manifest, no symlinks
With
devic.es/b70-vaapithe names match their minors:6. Real transcode: the exact invocation from the failed job report
The report's own source file no longer exists on disk, so this uses another real library file
with the same encoder invocation (
av1_qsv -preset medium -global_quality 28 -look_ahead 1,all streams mapped) taken verbatim from the failed job report:
Output integrity (full decode of every frame, not just a probe):
7. The AI stack was never disturbed
A separate device-plugin group was used precisely so the existing
b70group's device IDs(
sha1(count + host paths)) stay stable and kubelet's live allocations remain valid:vllmheld 0 restarts across the plugin rollout and the entire exercise.8. Merge path validated
The ConfigMap content hash changed the ConfigMap name, the HelmRelease reference followed it,
and the DaemonSet rolled and registered the new resource automatically within 15s, with no
manual restart:
t=15s ds-configmap=generic-device-plugin-bbkf9t2m76 b70-vaapi=99Without the hash change this fix would only have worked by accident: the config is a
subPathmount, which kubelet never refreshes.
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
🔧 **Rebase** - 1 issue found → auto-fixed ✅
docs/ai-gpu-changelog.md- merge conflict rebasing onto origin/main🔧 Fix applied.
✅ Re-checked - no issues remain.
docs/ai-gpu-changelog.md:22- Current baseline still dated 2026-08-26 listed onlydevic.es/b70and placedtdarr-nodeon that resource, contradicting the 2026-08-29 entry and the live fix. Updated the baseline date, resource row, and workloads table so the summary matchesb70vsb70-vaapiownership.docs/media-stack.md:378- Monitoring GPU usage jq only printedb70while the surrounding prose says bothb70andb70-vaapimust be 99 on talos-3. Extended the command to includeb70-vaapiso a missing VA-API registration is visible without relying only on the separate vainfo check.kubernetes/apps/base/system/generic-device-plugin/app/config/config.yaml:14-b70group comment still claimedtdarr-nodeco-schedules on that resource after the move tob70-vaapi. Updated the comment to point at the VA-API group.mise exec -- python3 scripts/ci/b70-vaapi-tdarr-test.py(HEAD PASS: hashed ConfigMap rollout, b70 vs b70-vaapi mounts, tdarr-node on devic.es/b70-vaapi with transcodecpuWorkers=1, libdrm DEVNAME A-B-A simulation)mise exec -- python3 scripts/ci/b70-vaapi-tdarr-test.pyagainst base commit 6480a8bf worktree (FAIL: configMapGenerator hash disabled)mise exec -- python3 scripts/ci/igpu-xe-allowids-test.py(updated consumer split: Level Zero on b70, tdarr-node on b70-vaapi)mise exec -- kustomize build kubernetes/apps/base/system/generic-device-plugin/appand structured parse of emitted ConfigMap/HelmReleaseLive cluster vainfo/ffmpeg/library transcode not runnable (no kubeconfig)✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.