Skip to content

fix(tdarr): restore B70 VA-API transcoding via name-faithful device group - #1490

Merged
Aviator-Coding merged 6 commits into
mainfrom
fm/homeops-tdarr-transcode-restore
Aug 29, 2026
Merged

fix(tdarr): restore B70 VA-API transcoding via name-faithful device group#1490
Aviator-Coding merged 6 commits into
mainfrom
fm/homeops-tdarr-transcode-restore

Conversation

@Aviator-Coding

@Aviator-Coding Aviator-Coding commented Aug 29, 2026

Copy link
Copy Markdown
Owner

Intent

Restore Tdarr transcoding. Tdarr's GPU path had been failing on every job since 2026-08-26 with no fallback, so nothing transcoded at all. The scout report isolated it to PR #1443, which re-plumbed the Arc B70 through generic-device-plugin (which renames the render node), and reproduced live that VA-API can no longer open a device. But the scout explicitly did NOT prove the failing call, and left two live hypotheses: (a) the name-to-minor/sysfs mismatch from the devic.es/b70 remap (leading, but unproven), or (b) a driver incompatibility in the tdarr_node image itself (libva 2.23.0 + intel-media-va-driver-non-free 26.2.2 vs Battlemage). I was told NOT to treat the leading hypothesis as established, and to discriminate between them.

Required work, in order: (1) Stop the outage first, before any investigation: set transcodecpuWorkers to 1 in the tdarr helmrelease, shipped regardless of findings, and as its own commit if it could not fit cleanly with the rest so relief was not blocked behind investigation. (2) Run the discriminating test by exposing the same by-path nodes at their NATIVE names (/dev/dri/card1, /dev/dri/renderD129) via generic-device-plugin and pointing tdarr-node at that; if VA-API then works the rename is confirmed and the fix is to give VA-API consumers a name-faithful device while Level Zero consumers keep the remapped one; if it still fails, report the driver hypothesis as live and stop rather than guessing at image pins. Use the exact vainfo and ffmpeg commands the report quotes so results are comparable. (3) Close the documentation gap that let this ship: #1443 updated the media and GPU docs but neither mentions verifying VA-API after the switch, so add that verification step with the concrete command.

Constraints: do not touch the B70 itself and do not reboot or upgrade any node (it has separate history and pending attended maintenance; reading its state is fine). Do not revert PR #1443, which was deliberate GPU work other consumers depend on. Proof of success must be what actually transcoded successfully, not that the pod is running.

Outcome: the rename hypothesis is PROVEN and the driver hypothesis DISPROVEN, by an A-B-A test inside the running pod holding everything else constant: creating a file merely named renderD129 made the UNCHANGED renderD128 path succeed, removing it reproduced 'Device creation failed: -542398533', restoring it succeeded again. Mechanism: libdrm ignores the path passed to it, fstat()s the fd, reads /sys/dev/char/226:129/uevent, re-derives the canonical DEVNAME (dri/renderD129) and reopens that path, which a container that mounted the device as renderD128 does not have, so vaGetDisplayDRM() fails before any driver loads. Proof of transcode: a real 25-minute 1080p library file transcoded end to end with the exact av1_qsv -preset medium -global_quality 28 -look_ahead 1 invocation from the failed job report, exit 0, 36961 frames at 12.8x realtime, output decodes with zero errors.

Deliberate decisions a reviewer reading only the diff would not know: (i) The fix adds a SEPARATE generic-device-plugin group (b70-vaapi) rather than adding native-name paths to the existing b70 group, because that plugin derives device IDs as sha1(count_index + every resolved host path in the group), so editing b70 would change all 99 device IDs and invalidate kubelet's live allocations for the running vllm and comfyui pods. Verified live that a new group leaves them untouched: devic.es/b70 stayed at 99 and vllm held 0 restarts across the plugin rollout. (ii) Enabling the configMapGenerator name-suffix hash is required for the fix to work at all, not cosmetic: the config is a subPath mount, which kubelet never refreshes, and disableNameSuffixHash pinned the ConfigMap name so the pod template never changed, meaning a config-only edit could never reach the running plugin without a manual DaemonSet restart. kustomizeconfig.yaml already declared the nameReference that a content hash needs. Verified live that the merge path works: the resource registered automatically within 15s with no manual step. (iii) mountPath in the new group hardcodes the kernel's current enumeration (iGPU at 0000:00:02.0 probes first and takes card0/renderD128, leaving the discrete card at card1/renderD129); this is a known coupling, documented, and the CPU fallback means a future break degrades instead of causing a total outage. (iv) The live cluster was deliberately left exactly matching main rather than carrying the fix as drift, so the change lands cleanly at merge.

Later captain decision on service-restore scope, which the docs in this change record: the GPU fix alone does not resume transcoding, because all three Talos nodes had been excluded from BOTH Tdarr libraries as a post-outage mitigation. Only the Movies AV1 exclusion was cleared on talos-3; the Series exclusion is deliberately RETAINED so the restore covers one library at a time while the VA-API fix is newly landed. The eight files in Tdarr's error table, mostly 21-67 GB 2160p DV/HDR10 remux masters, must NOT be fed back in as a side effect of restoring service: their previous failure is unexplained and Tdarr rewrites in place with a lossy irreversible AV1 result, so they are recorded as a separate follow-up needing their failure understood first. Verified that re-enabling a library does not requeue them (it is a node-side accept filter; all 8 stayed parked with the transcode queue at 0). Both of these are per-node Tdarr server state (librariesToNotProcess in NodeJSONDB), not GitOps, so they cannot ship in this PR and are only documented by it.

What Changed

  • Re-enable Tdarr transcoding by requesting devic.es/b70-vaapi (kernel-canonical card1/renderD129 mounts) instead of the renamed devic.es/b70 group, and set transcodecpuWorkers to 1 so a future VA-API regression degrades instead of stalling every job.
  • Add a separate b70-vaapi generic-device-plugin group and re-enable the configMapGenerator name-suffix hash so config-only device-plugin changes roll out without rewriting the existing b70 group IDs used by vllm/comfyui.
  • Document the libdrm rename trap, staged library restore/remux follow-ups, and VA-API verification steps, and add CI coverage for the b70-vaapi/Tdarr device path.

Risk Assessment

✅ Low: Durable, intent-complete fix: separate name-faithful b70-vaapi group, CPU fallback, hash-enabled plugin rollout, and docs/proof of the libdrm rename failure, without touching the shared b70 allocation set.

Testing

Added and ran a semantic b70-vaapi/tdarr regression test plus an updated iGPU allowIDs consumer split; both pass on HEAD, the new test fails on the pre-fix base, and kustomize-built output shows native-name b70-vaapi mounts with a hashed ConfigMap reference, but fresh live transcode evidence could not be captured without cluster access.

Evidence: b70-vaapi-tdarr-test HEAD evidence (PASS + libdrm A-B-A)
{
  "b70Mounts": {
    "/dev/dri/by-path/pci-0000:03:00.0-card": "/dev/dri/card0",
    "/dev/dri/by-path/pci-0000:03:00.0-render": "/dev/dri/renderD128"
  },
  "b70VaapiMounts": {
    "/dev/dri/by-path/pci-0000:03:00.0-card": "/dev/dri/card1",
    "/dev/dri/by-path/pci-0000:03:00.0-render": "/dev/dri/renderD129"
  },
  "configMap": {
    "deviceNames": [
      "b70",
      "b70-vaapi",
      "tun"
    ],
    "name": "generic-device-plugin-kkk465kb69"
  },
  "helmreleasePersistence": {
    "globalMounts": [
      {
        "path": "/config/config.yaml",
        "readOnly": true,
        "subPath": "config.yaml"
      }
    ],
    "name": "generic-device-plugin-kkk465kb69",
    "type": "configMap"
  },
  "libdrmTrapSimulation": {
    "A2_restored": {
      "canonicalPresent": true,
      "devname": "dri/renderD129",
      "opened": "dev/dri/renderD128",
      "reopenError": null,
      "reopenOk": true
    },
    "A_nameFaithful": {
      "canonicalPresent": true,
      "devname": "dri/renderD129",
      "opened": "dev/dri/renderD128",
      "reopenError": null,
      "reopenOk": true
    },
    "B_renamedOnly": {
      "canonicalPresent": false,
      "devname": "dri/renderD129",
      "opened": "dev/dri/renderD128",
      "reopenError": "FileNotFoundError: 2 No such file or directory",
      "reopenOk": false
    }
  },
  "pluginArgs": [
    "--log-level=info",
    "--domain=devic.es",
    "--config=/config/config.yaml"
  ],
  "result": "PASS",
  "tdarrNode": {
    "env": {
      "healthcheckcpuWorkers": "1",
      "healthcheckgpuWorkers": "1",
      "transcodecpuWorkers": "1",
      "transcodegpuWorkers": "1"
    },
    "extendedResources": [
      "devic.es/b70-vaapi"
    ],
    "serverExtendedResources": []
  }
}
Evidence: b70-vaapi-tdarr-test BASE pre-fix failure
{
  "error": "configMapGenerator hash is disabled; subPath mounts would never roll out",
  "result": "FAIL"
}
Evidence: Rendered generic-device-plugin + tdarr contract summary

configMap=generic-device-plugin-kkk465kb69; b70→card0/renderD128; b70-vaapi→card1/renderD129; tdarr-node limits devic.es/b70-vaapi=1; transcodecpuWorkers=1; libdrm A/B/A2 reopenOk=true/false/true

{
  "configMapName": "generic-device-plugin-kkk465kb69",
  "devices": [
    {
      "name": "tun",
      "paths": [
        {
          "path": "/dev/net/tun",
          "mountPath": null
        }
      ]
    },
    {
      "name": "b70",
      "paths": [
        {
          "path": "/dev/dri/by-path/pci-0000:03:00.0-card",
          "mountPath": "/dev/dri/card0"
        },
        {
          "path": "/dev/dri/by-path/pci-0000:03:00.0-render",
          "mountPath": "/dev/dri/renderD128"
        }
      ]
    },
    {
      "name": "b70-vaapi",
      "paths": [
        {
          "path": "/dev/dri/by-path/pci-0000:03:00.0-card",
          "mountPath": "/dev/dri/card1"
        },
        {
          "path": "/dev/dri/by-path/pci-0000:03:00.0-render",
          "mountPath": "/dev/dri/renderD129"
        }
      ]
    }
  ],
  "helmConfigRef": {
    "name": "generic-device-plugin-kkk465kb69",
    "subPath": [
      "config.yaml"
    ]
  },
  "tdarrNode": {
    "limits": {
      "devic.es/b70-vaapi": 1,
      "memory": "4Gi"
    },
    "transcodecpuWorkers": "1",
    "transcodegpuWorkers": "1"
  }
}
Evidence: kustomize-built generic-device-plugin manifests
apiVersion: v1
data:
  config.yaml: |
    ---
    devices:
      - name: tun
        groups:
          - count: 1000
            paths:
              - path: /dev/net/tun
      # Discrete Arc Pro B70 on talos-3 (Battlemage G31, PCI 8086:e223 @ 0000:03:00.0).
      # Intel GpuDevicePlugin cannot split xe KMD devices into distinct extended
      # resources (both iGPU and B70 would be gpu.intel.com/xe), so this plugin
      # owns the B70 under its own name. Discover via stable DRM by-path nodes and
      # re-expose them at the conventional card0/renderD128 paths Level Zero expects
      # inside the container. count matches the former sharedDevNum share-token model
      # so vllm/vllm-embed/comfyui can still co-schedule concurrently. tdarr-node
      # uses the separate b70-vaapi group below (VA-API needs the kernel names).
      - name: b70
        groups:
          - count: 99
            paths:
              - path: /dev/dri/by-path/pci-0000:03:00.0-card
                mountPath: /dev/dri/card0
              - path: /dev/dri/by-path/pci-0000:03:00.0-render
                mountPath: /dev/dri/renderD128
      # Same physical B70, exposed under the device names the KERNEL gives it.
      #
      # The b70 group above renames the nodes to card0/renderD128, and that rename
      # is fatal to VA-API. libdrm does not trust the path you hand it: it fstat()s
      # the fd, reads /sys/dev/char/<major>:<minor>/uevent, and re-derives the
      # canonical DEVNAME (dri/renderD129 for this card). It then reopens
      # /dev/dri/renderD129 - which does not exist in a container that mounted the
      # same device as renderD128 - so vaGetDisplayDRM() fails before any driver
      # loads. Proven live 2026-08-29: creating a renderD129 symlink inside the
      # running tdarr-node made the UNCHANGED renderD128 path work, and removing it
      # broke it again (A-B-A, single variable). See docs/media-stack.md.
      #
      # Level Zero (vllm, comfyui) opens whatever /dev/dri/renderD* it finds and is
      # unaffected either way, so the b70 group is left byte-identical: its device
      # IDs are sha1(count + host paths), so editing its paths would change all 99
      # IDs and invalidate kubelet's live allocations for the running AI pods.
      #
      # NOTE: mountPath here must track the kernel's enumeration of this card. The
      # iGPU (0000:00:02.0) probes first and takes card0/renderD128, so the discrete
      # card at 0000:03:00.0 is card1/renderD129. If that order ever changes, VA-API
      # breaks again - tdarr then degrades to its CPU worker rather than failing
      # every job. Verify with the vainfo check in docs/media-stack.md after any GPU
      # or device-plugin change.
      - name: b70-vaapi
        groups:
          - count: 99
            paths:
              - path: /dev/dri/by-path/pci-0000:03:00.0-card
                mountPath: /dev/dri/card1
              - path: /dev/dri/by-path/pci-0000:03:00.0-render
                mountPath: /dev/dri/renderD129
kind: ConfigMap
metadata:
  name: generic-device-plugin-kkk465kb69
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: generic-device-plugin
spec:
  chartRef:
    kind: OCIRepository
    name: app-template
    namespace: flux-system
  driftDetection:
    mode: enabled
  interval: 30m
  values:
    controllers:
      generic-device-plugin:
        containers:
          app:
            args:
            - --log-level=info
            - --domain=devic.es
            - --config=/config/config.yaml
            image:
              repository: ghcr.io/squat/generic-device-plugin
              tag: 36bfc606bba2064de6ede0ff2764cbb52edff70d@sha256:ba6f0b4cf6c858d6ad29ba4d32e4da11638abbc7d96436bf04f582a97b2b8821
            resources:
              limits:
                memory: 64Mi
              requests:
                cpu: 10m
            securityContext:
              allowPrivilegeEscalation: false
              capabilities:
                drop:
                - ALL
              readOnlyRootFilesystem: true
        strategy: RollingUpdate
        type: daemonset
    defaultPodOptions:
      priorityClassName: system-node-critical
    persistence:
      config:
        globalMounts:
        - path: /config/config.yaml
          readOnly: true
          subPath: config.yaml
        name: generic-device-plugin-kkk465kb69
        type: configMap
      dev:
        globalMounts:
        - readOnly: true
        hostPath: /dev
        type: hostPath
      sys:
        globalMounts:
        - readOnly: true
        hostPath: /sys
        type: hostPath
      var-lib-kubelet-device-plugins:
        hostPath: /var/lib/kubelet/device-plugins
        type: hostPath
Evidence: igpu-xe-allowids-test HEAD evidence
{
  "b70Consumers": {
    "comfyui": {
      "comfyui": [
        "devic.es/b70"
      ]
    },
    "vllm": {
      "vllm": [
        "devic.es/b70"
      ],
      "vllm-embed": [
        "devic.es/b70"
      ]
    }
  },
  "b70VaapiConsumers": {
    "tdarr-node": {
      "tdarr": [],
      "tdarr-node": [
        "devic.es/b70-vaapi"
      ]
    }
  },
  "expectedAllocatableAfterReconcile": {
    "talos-1": {
      "devic.es/b70": 0,
      "devic.es/b70-vaapi": 0,
      "gpu.intel.com/xe": 99
    },
    "talos-2": {
      "devic.es/b70": 0,
      "devic.es/b70-vaapi": 0,
      "gpu.intel.com/xe": 99
    },
    "talos-3": {
      "devic.es/b70": 99,
      "devic.es/b70-vaapi": 99,
      "gpu.intel.com/xe": 99
    }
  },
  "gpuDevicePlugin": {
    "apiVersion": "deviceplugin.intel.com/v1",
    "kind": "GpuDevicePlugin",
    "metadata.name": "xe",
    "spec.allowIDs": "0xa7a0",
    "spec.image": "intel/intel-gpu-plugin:0.34.1",
    "spec.sharedDevNum": 99
  },
  "helmrelease": {
    "name": "intel-device-plugin-gpu",
    "values": {
      "allowIDs": "0xa7a0",
      "name": "xe",
      "nodeFeatureRule": false,
      "sharedDevNum": 99
    }
  },
  "result": "PASS",
  "xeConsumers": {
    "jellyfin": {
      "jellyfin": [
        "gpu.intel.com/xe"
      ]
    },
    "playwright": {
      "playwright": [
        "gpu.intel.com/xe"
      ]
    },
    "plex": {
      "plex": [
        "gpu.intel.com/xe"
      ]
    }
  }
}
Evidence: Combined test transcript
=== b70-vaapi-tdarr-test HEAD (fix) ===
result=PASS
configMap=generic-device-plugin-kkk465kb69
b70 mounts={'/dev/dri/by-path/pci-0000:03:00.0-card': '/dev/dri/card0', '/dev/dri/by-path/pci-0000:03:00.0-render': '/dev/dri/renderD128'}
b70-vaapi mounts={'/dev/dri/by-path/pci-0000:03:00.0-card': '/dev/dri/card1', '/dev/dri/by-path/pci-0000:03:00.0-render': '/dev/dri/renderD129'}
tdarr-node resources=['devic.es/b70-vaapi'] cpuWorkers=1
libdrm A (canonical): reopenOk=True
libdrm B (renamed only): reopenOk=False err=FileNotFoundError: 2 No such file or directory
libdrm A2 (restored): reopenOk=True

=== b70-vaapi-tdarr-test BASE 6480a8bf (pre-fix) ===
result=FAIL
error=configMapGenerator hash is disabled; subPath mounts would never roll out

=== igpu-xe-allowids-test HEAD ===
OK: GpuDevicePlugin allowIDs=0xa7a0; xe consumers ['jellyfin', 'playwright', 'plex'] exclusive of b70; b70 consumers ['comfyui', 'vllm'] exclusive of xe/vaapi; b70-vaapi consumers ['tdarr-node'] exclusive of xe/b70
- Outcome: ⚠️ 1 warning across 1 run (3m37s)

Live verification evidence

Run against the live cluster on 2026-08-29, before the branch state was reverted so the
cluster would match main. Commands verbatim. -n media exec deploy/tdarr-tdarr-node -c app
throughout; the pipeline worktree has no cluster access, so this is recorded here to be durable.

1. Baseline failure reproduced (the reported symptom)

$ kubectl -n media exec deploy/tdarr-tdarr-node -c app -- \
    vainfo --display drm --device /dev/dri/renderD128
Trying display: drm
Failed to a DRM display for the given device
command terminated with exit code 1

2. The mechanism: names do not match minors, and libdrm reopens the canonical name

$ kubectl -n media exec deploy/tdarr-tdarr-node -c app -- ls -l /dev/dri/
crw-------  226,   1  card0          # host card1
crw-rw-rw-  226, 129  renderD128     # host renderD129  <- name says 128, minor says 129

$ kubectl -n media exec deploy/tdarr-tdarr-node -c app -- cat /sys/dev/char/226:129/uevent
MAJOR=226
MINOR=129
DEVNAME=dri/renderD129               # <- the path libdrm will reopen
DEVTYPE=drm_minor

$ kubectl -n media exec deploy/tdarr-tdarr-node -c app -- ls -l /dev/dri/renderD129
ls: cannot access '/dev/dri/renderD129': No such file or directory

3. A-B-A controlled test: one variable, the existence of a file named renderD129

The device passed to ffmpeg is renderD128 in every run below. Nothing about the driver,
the card, or the argument changes; only whether /dev/dri/renderD129 exists.

# B: create the canonical name
$ kubectl -n media exec deploy/tdarr-tdarr-node -c app -- ln -sf /dev/dri/renderD128 /dev/dri/renderD129
$ kubectl -n media exec deploy/tdarr-tdarr-node -c app -- \
    tdarr-ffmpeg -v verbose -init_hw_device vaapi=va:/dev/dri/renderD128 -f lavfi -i nullsrc -frames:v 1 -f null -
[AVHWDeviceContext] Initialised VAAPI connection: version 1.23        # PASS

# A: remove it
$ kubectl -n media exec deploy/tdarr-tdarr-node -c app -- rm -f /dev/dri/renderD129
$ ...same ffmpeg command...
[AVHWDeviceContext] Cannot open a VA display from DRM device /dev/dri/renderD128.
Device creation failed: -542398533.                                   # FAIL

# B: restore it
$ kubectl -n media exec deploy/tdarr-tdarr-node -c app -- ln -sf /dev/dri/renderD128 /dev/dri/renderD129
$ ...same ffmpeg command...
[AVHWDeviceContext] Initialised VAAPI connection: version 1.23        # PASS

PASS -> FAIL -> PASS on one variable. This proves the rename is the cause and
disproves the competing hypothesis that libva 2.23.0 + intel-media-va-driver-non-free 26.2.2 is incompatible with Battlemage: the same driver initialises fine the moment the
name matches.

4. Driver and card identity confirmed (it really is the B70, with AV1 encode)

$ kubectl -n media exec deploy/tdarr-tdarr-node -c app -- \
    vainfo --display drm --device /dev/dri/renderD129
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 26.2.2 ()
      VAProfileAV1Profile0            : VAEntrypointVLD
      VAProfileAV1Profile0            : VAEntrypointEncSlice     # <- what av1_qsv needs

$ kubectl -n media exec deploy/tdarr-tdarr-node -c app -- cat /sys/dev/char/226:129/device/uevent
DRIVER=xe
PCI_ID=8086:E223                    # Arc Pro B70 (Battlemage)
PCI_SLOT_NAME=0000:03:00.0

5. Post-fix device layout, from the manifest, no symlinks

With devic.es/b70-vaapi the names match their minors:

$ kubectl -n media exec deploy/tdarr-tdarr-node -c app -- ls -l /dev/dri/
crw-------  226,   1  card1
crw-rw-rw-  226, 129  renderD129

6. Real transcode: the exact invocation from the failed job report

The report's own source file no longer exists on disk, so this uses another real library file
with the same encoder invocation (av1_qsv -preset medium -global_quality 28 -look_ahead 1,
all streams mapped) taken verbatim from the failed job report:

$ kubectl -n media exec deploy/tdarr-tdarr-node -c app -- \
    tdarr-ffmpeg -y -i "/media/Movies/Tales of the Black Freighter (2009)/Tales of the Black Freighter (2009) Bluray-1080p.mkv" \
      -map 0:0 -c:0 av1_qsv -map 0:1 -c:1 copy -map 0:2 -c:2 copy \
      -preset medium -global_quality 28 -look_ahead 1 \
      /temp/fm-proof/blackfreighter-av1qsv.mkv
frame=36961 fps=309 q=-0.0 Lsize=330491KiB time=00:25:37.66 bitrate=1760.7kbits/s speed=12.8x
EXITCODE=0

Output integrity (full decode of every frame, not just a probe):

$ ... tdarr-ffprobe ... /temp/fm-proof/blackfreighter-av1qsv.mkv
codec_name=av1 / ac3 / subrip        # all 3 source streams preserved
duration=1542.560000                 # exact source duration

$ ... tdarr-ffmpeg -v error -i /temp/fm-proof/blackfreighter-av1qsv.mkv -f null -
                                     # zero errors

7. The AI stack was never disturbed

A separate device-plugin group was used precisely so the existing b70 group's device IDs
(sha1(count + host paths)) stay stable and kubelet's live allocations remain valid:

$ kubectl get node talos-3 -o jsonpath='{.status.allocatable}'
devic.es/b70: 99          # unchanged
devic.es/b70-vaapi: 99    # added

$ kubectl -n ai get pods | grep vllm
vllm-64c54b9996-nbccw   1/1   Running   0 restarts   3d2h

vllm held 0 restarts across the plugin rollout and the entire exercise.

8. Merge path validated

The ConfigMap content hash changed the ConfigMap name, the HelmRelease reference followed it,
and the DaemonSet rolled and registered the new resource automatically within 15s, with no
manual restart:

t=15s  ds-configmap=generic-device-plugin-bbkf9t2m76  b70-vaapi=99

Without the hash change this fix would only have worked by accident: the config is a subPath
mount, which kubelet never refreshes.

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

🔧 **Rebase** - 1 issue found → auto-fixed ✅
  • ⚠️ docs/ai-gpu-changelog.md - merge conflict rebasing onto origin/main

🔧 Fix applied.
✅ Re-checked - no issues remain.

⚠️ **Review** - 3 infos
  • ℹ️ docs/ai-gpu-changelog.md:22 - Current baseline still dated 2026-08-26 listed only devic.es/b70 and placed tdarr-node on that resource, contradicting the 2026-08-29 entry and the live fix. Updated the baseline date, resource row, and workloads table so the summary matches b70 vs b70-vaapi ownership.
  • ℹ️ docs/media-stack.md:378 - Monitoring GPU usage jq only printed b70 while the surrounding prose says both b70 and b70-vaapi must be 99 on talos-3. Extended the command to include b70-vaapi so a missing VA-API registration is visible without relying only on the separate vainfo check.
  • ℹ️ kubernetes/apps/base/system/generic-device-plugin/app/config/config.yaml:14 - b70 group comment still claimed tdarr-node co-schedules on that resource after the move to b70-vaapi. Updated the comment to point at the VA-API group.
⚠️ **Test** - 1 warning
  • ⚠️ No kubeconfig/cluster access in this worktree, so the live end-user proof from the intent could not be re-run here: vainfo on /dev/dri/renderD129, tdarr-ffmpeg av1_qsv smoke, or a real library-file transcode (exit 0 / frame count). Offline validation proves the GitOps contract and the libdrm DEVNAME reopen trap (A-B-A simulation), and docs record the prior live 1080p success, but a reviewer who requires fresh cluster evidence still needs those media-stack.md commands against deploy/tdarr-tdarr-node.
  • mise exec -- python3 scripts/ci/b70-vaapi-tdarr-test.py (HEAD PASS: hashed ConfigMap rollout, b70 vs b70-vaapi mounts, tdarr-node on devic.es/b70-vaapi with transcodecpuWorkers=1, libdrm DEVNAME A-B-A simulation)
  • mise exec -- python3 scripts/ci/b70-vaapi-tdarr-test.py against base commit 6480a8bf worktree (FAIL: configMapGenerator hash disabled)
  • mise exec -- python3 scripts/ci/igpu-xe-allowids-test.py (updated consumer split: Level Zero on b70, tdarr-node on b70-vaapi)
  • mise exec -- kustomize build kubernetes/apps/base/system/generic-device-plugin/app and structured parse of emitted ConfigMap/HelmRelease
  • Live cluster vainfo/ffmpeg/library transcode not runnable (no kubeconfig)
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

tdarr-node ran transcodegpuWorkers=1 / transcodecpuWorkers=0, so when the
B70's VA-API path broke on 2026-08-26 every transcode job failed with
transcodeError and no worker could take over. With the library already at
99.28% the UI showed an idle, healthy server, so a total transcoding outage
stayed invisible for three days.

Setting transcodecpuWorkers=1 restores transcoding today, independent of the
GPU investigation. A GPU-only node is the structural cause of the silence,
not just this incident's bad luck.
…aapi

generic-device-plugin's b70 group renames the B70's DRM nodes to
card0/renderD128 via mountPath. That rename is fatal to VA-API. libdrm does
not trust the path it is handed: it fstat()s the fd, reads
/sys/dev/char/226:129/uevent, re-derives the canonical DEVNAME
(dri/renderD129) and reopens that path. Inside a container that mounted the
same device as renderD128 the canonical path does not exist, so
vaGetDisplayDRM() fails before any driver loads.

Proven live by A-B-A inside the running pod, one variable: creating a
renderD129 symlink made the unchanged renderD128 path work, removing it
reproduced "Device creation failed: -542398533", restoring it worked again.
This disproves the competing hypothesis that libva 2.23.0 plus
intel-media-va-driver-non-free 26.2.2 is incompatible with Battlemage; the
same driver initialises fine as soon as the name matches.

Add a second device-plugin group exposing the same by-path nodes at the
kernel's own names and point tdarr-node at it. A second group rather than an
edit to b70 because device IDs are sha1(count + every host path in the
group), so adding paths to b70 would change all 99 IDs and invalidate
kubelet's live allocations for vllm and comfyui. Verified live: devic.es/b70
stayed at 99 and vllm held 0 restarts across the plugin rollout.

Validation: a real 25-minute 1080p library file transcoded end to end with
the exact av1_qsv invocation from the failed job report, at ~13x realtime,
and the output decodes with zero errors.

Also document the verification step whose absence let this ship: #1443
updated the media and GPU docs but neither checked that VA-API still worked.
The config is mounted with subPath, and kubelet never refreshes a subPath
mount. Combined with disableNameSuffixHash: true, which pins the ConfigMap
name so the pod template never changes, a config-only edit could not reach
the running plugin at all: it stayed inert until someone restarted the
DaemonSet by hand. The b70-vaapi group added in the previous commit would
have taken effect only by luck.

kustomizeconfig.yaml already declares the nameReference that rewrites the
HelmRelease's persistence.config.name, which is exactly the machinery a
content hash needs, so enabling the hash is a two-line change: the ConfigMap
name now tracks its content, the HelmRelease follows it, and the pod template
change rolls the DaemonSet.

Restarting this DaemonSet does not disturb pods that already hold allocated
devices - verified live while adding b70-vaapi, where vllm kept its B70 slot
and 0 restarts across the rollout.
Renaming a DRM node via generic-device-plugin's mountPath silently breaks
VA-API while leaving Level Zero working, so the AI stack stays green while
transcoding is dead. That asymmetry is what hid a 3-day total Tdarr outage.
Capture it next to the other cluster-wide traps, along with the two related
device-plugin footguns found while fixing it: group device IDs are derived
from the group's host paths, and the config is a subPath mount that kubelet
never refreshes.
The GPU fix alone does not resume transcoding. All three Talos nodes had been
excluded from both Tdarr libraries as a post-outage mitigation, so the node
would have picked up zero jobs after merge. That scoping is per-node Tdarr
server state (librariesToNotProcess in NodeJSONDB), not GitOps, so it cannot
ship in this PR - only be documented by it.

Per captain decision, only the Movies AV1 exclusion is cleared on talos-3.
The Series exclusion is deliberately retained so the restore covers one
library at a time while the VA-API fix is newly landed.

The eight files in Tdarr's error table, mostly 21-67 GB 2160p DV/HDR10 remux
masters, are recorded as a separate follow-up. Their failure is not explained
by the VA-API break alone, and Tdarr rewrites in place with a lossy
irreversible AV1 result, so they must not be bulk-requeued to tidy the error
table. Verified that re-enabling a library does not requeue them: it is a
node-side accept filter, and all 8 stayed parked with the queue at 0.
@mortyops

mortyops Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor
--- HelmRelease media/tdarr
+++ HelmRelease media/tdarr
@@ -132,7 +132,7 @@
             - name: transcodegpuWorkers
               value: "1"
             - name: transcodecpuWorkers
-              value: "0"
+              value: "1"
             - name: healthcheckgpuWorkers
               value: "1"
             - name: healthcheckcpuWorkers
@@ -142,7 +142,7 @@
               tag: 2.86.01
             resources:
               limits:
-                devic.es/b70: 1
+                devic.es/b70-vaapi: 1
                 memory: 4Gi
               requests:
                 cpu: 100m

--- ConfigMap system/generic-device-plugin
+++ ConfigMap system/generic-device-plugin
@@ -1,34 +1 @@
-apiVersion: v1
-data:
-  config.yaml: |
-    ---
-    devices:
-      - name: tun
-        groups:
-          - count: 1000
-            paths:
-              - path: /dev/net/tun
-      # Discrete Arc Pro B70 on talos-3 (Battlemage G31, PCI 8086:e223 @ 0000:03:00.0).
-      # Intel GpuDevicePlugin cannot split xe KMD devices into distinct extended
-      # resources (both iGPU and B70 would be gpu.intel.com/xe), so this plugin
-      # owns the B70 under its own name. Discover via stable DRM by-path nodes and
-      # re-expose them at the conventional card0/renderD128 paths Level Zero expects
-      # inside the container. count matches the former sharedDevNum share-token model
-      # so vllm/vllm-embed/comfyui/tdarr-node can still co-schedule concurrently.
-      - name: b70
-        groups:
-          - count: 99
-            paths:
-              - path: /dev/dri/by-path/pci-0000:03:00.0-card
-                mountPath: /dev/dri/card0
-              - path: /dev/dri/by-path/pci-0000:03:00.0-render
-                mountPath: /dev/dri/renderD128
-kind: ConfigMap
-metadata:
-  labels:
-    app.kubernetes.io/name: generic-device-plugin
-    kustomize.toolkit.fluxcd.io/name: generic-device-plugin
-    kustomize.toolkit.fluxcd.io/namespace: system
-  name: generic-device-plugin
-  namespace: system
 

--- ConfigMap system/generic-device-plugin-kkk465kb69
+++ ConfigMap system/generic-device-plugin-kkk465kb69
@@ -1 +1,66 @@
+apiVersion: v1
+data:
+  config.yaml: |
+    ---
+    devices:
+      - name: tun
+        groups:
+          - count: 1000
+            paths:
+              - path: /dev/net/tun
+      # Discrete Arc Pro B70 on talos-3 (Battlemage G31, PCI 8086:e223 @ 0000:03:00.0).
+      # Intel GpuDevicePlugin cannot split xe KMD devices into distinct extended
+      # resources (both iGPU and B70 would be gpu.intel.com/xe), so this plugin
+      # owns the B70 under its own name. Discover via stable DRM by-path nodes and
+      # re-expose them at the conventional card0/renderD128 paths Level Zero expects
+      # inside the container. count matches the former sharedDevNum share-token model
+      # so vllm/vllm-embed/comfyui can still co-schedule concurrently. tdarr-node
+      # uses the separate b70-vaapi group below (VA-API needs the kernel names).
+      - name: b70
+        groups:
+          - count: 99
+            paths:
+              - path: /dev/dri/by-path/pci-0000:03:00.0-card
+                mountPath: /dev/dri/card0
+              - path: /dev/dri/by-path/pci-0000:03:00.0-render
+                mountPath: /dev/dri/renderD128
+      # Same physical B70, exposed under the device names the KERNEL gives it.
+      #
+      # The b70 group above renames the nodes to card0/renderD128, and that rename
+      # is fatal to VA-API. libdrm does not trust the path you hand it: it fstat()s
+      # the fd, reads /sys/dev/char/<major>:<minor>/uevent, and re-derives the
+      # canonical DEVNAME (dri/renderD129 for this card). It then reopens
+      # /dev/dri/renderD129 - which does not exist in a container that mounted the
+      # same device as renderD128 - so vaGetDisplayDRM() fails before any driver
+      # loads. Proven live 2026-08-29: creating a renderD129 symlink inside the
+      # running tdarr-node made the UNCHANGED renderD128 path work, and removing it
+      # broke it again (A-B-A, single variable). See docs/media-stack.md.
+      #
+      # Level Zero (vllm, comfyui) opens whatever /dev/dri/renderD* it finds and is
+      # unaffected either way, so the b70 group is left byte-identical: its device
+      # IDs are sha1(count + host paths), so editing its paths would change all 99
+      # IDs and invalidate kubelet's live allocations for the running AI pods.
+      #
+      # NOTE: mountPath here must track the kernel's enumeration of this card. The
+      # iGPU (0000:00:02.0) probes first and takes card0/renderD128, so the discrete
+      # card at 0000:03:00.0 is card1/renderD129. If that order ever changes, VA-API
+      # breaks again - tdarr then degrades to its CPU worker rather than failing
+      # every job. Verify with the vainfo check in docs/media-stack.md after any GPU
+      # or device-plugin change.
+      - name: b70-vaapi
+        groups:
+          - count: 99
+            paths:
+              - path: /dev/dri/by-path/pci-0000:03:00.0-card
+                mountPath: /dev/dri/card1
+              - path: /dev/dri/by-path/pci-0000:03:00.0-render
+                mountPath: /dev/dri/renderD129
+kind: ConfigMap
+metadata:
+  labels:
+    app.kubernetes.io/name: generic-device-plugin
+    kustomize.toolkit.fluxcd.io/name: generic-device-plugin
+    kustomize.toolkit.fluxcd.io/namespace: system
+  name: generic-device-plugin-kkk465kb69
+  namespace: system
 

--- HelmRelease system/generic-device-plugin
+++ HelmRelease system/generic-device-plugin
@@ -63,7 +63,7 @@
         - path: /config/config.yaml
           readOnly: true
           subPath: config.yaml
-        name: generic-device-plugin
+        name: generic-device-plugin-kkk465kb69
         type: configMap
       dev:
         globalMounts:

@mortyops

mortyops Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor
--- Deployment media/tdarr-tdarr-node
+++ Deployment media/tdarr-tdarr-node
@@ -55,7 +55,7 @@
         - name: transcodegpuWorkers
           value: "1"
         - name: transcodecpuWorkers
-          value: "0"
+          value: "1"
         - name: healthcheckgpuWorkers
           value: "1"
         - name: healthcheckcpuWorkers
@@ -64,7 +64,7 @@
         name: app
         resources:
           limits:
-            devic.es/b70: 1
+            devic.es/b70-vaapi: 1
             memory: 4Gi
           requests:
             cpu: 100m

--- DaemonSet system/generic-device-plugin
+++ DaemonSet system/generic-device-plugin
@@ -66,7 +66,7 @@
       serviceAccountName: generic-device-plugin
       volumes:
       - configMap:
-          name: generic-device-plugin
+          name: generic-device-plugin-kkk465kb69
         name: config
       - hostPath:
           path: /dev

@Aviator-Coding
Aviator-Coding merged commit 7ae19e9 into main Aug 29, 2026
17 checks passed
@Aviator-Coding
Aviator-Coding deleted the fm/homeops-tdarr-transcode-restore branch August 29, 2026 14:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant