From 844670a417f5cd70ccd269dcd5d79cb2523a818c Mon Sep 17 00:00:00 2001 From: Rohini Vishu <200266036+RohiniVishu@users.noreply.github.com> Date: Tue, 18 Aug 2026 23:12:30 +0530 Subject: [PATCH 1/2] Enhance troubleshooting guide for HAMi installation Added troubleshooting sections for HAMi device plugin issues on GPU-less nodes and scheduler pod image pull failures. Signed-off-by: Rohini Vishu Signed-off-by: Rohini Vishu <200266036+RohiniVishu@users.noreply.github.com> --- docs/troubleshooting/troubleshooting.md | 219 ++++++++++++++++++++++++ 1 file changed, 219 insertions(+) diff --git a/docs/troubleshooting/troubleshooting.md b/docs/troubleshooting/troubleshooting.md index ec4851f41..018ecea60 100644 --- a/docs/troubleshooting/troubleshooting.md +++ b/docs/troubleshooting/troubleshooting.md @@ -160,3 +160,222 @@ devicePlugin: ``` ::: + +## HAMi device plugin fails on a GPU-less node + +HAMi's standard installation expects supported accelerator hardware and the corresponding vendor runtime to be available on nodes where the real device plugin is scheduled. +Use `mock-device-plugin` when the goal is to test HAMi scheduling behavior without executing workloads on a physical accelerator.The mock plugin is intended for development/testing and does not provide actual GPU execution or GPU performance validation. + +When testing HAMi on a machine without a supported NVIDIA GPU, the real NVIDIA device plugin cannot initialize successfully. For scheduler development and testing without physical accelerator hardware, use the HAMi `mock-device-plugin`. + +### Error seen + +After installing HAMi, check the pods: + +```bash +kubectl get pods -n kube-system +``` + +The HAMi device plugin may enter `CrashLoopBackOff`. + +Check its logs: + +```bash +kubectl logs -n kube-system \ + -c device-plugin --previous +``` + +The logs may contain: + +```text +Incompatible strategy detected auto +If this is a GPU node, did you configure the NVIDIA Container Toolkit? +If this is not a GPU node, you should set up a toleration or nodeSelector +to only deploy this plugin on GPU nodes +error starting plugins: ... invalid device discovery strategy +``` + +### Why it happens + +The real NVIDIA device plugin uses the NVIDIA discovery strategy and requires the NVIDIA driver/container runtime stack on the host.A GPU-less development machine does not provide that stack. If the node is labelled so that the real device plugin is scheduled there, the plugin attempts to initialize and fails. +The standard HAMi Quick Start is intended for an environment with the required accelerator prerequisites. + +--- + +## Problem 1: Mock device plugin reports devices as unhealthy + +Use this section when the HAMi `mock-device-plugin` is running but repeatedly reports configured devices as unhealthy and the corresponding vendor resources do not appear in the node's `Allocatable` resources. + +Check the mock plugin pod: + +```bash +kubectl get pods -n kube-system | grep mock-device-plugin +``` + +Then inspect its logs: + +```bash +kubectl logs -n kube-system --tail=100 +``` + +You may see repeated messages such as: + +```text +device NVIDIA is unhealthy on this node +device Ascend910A is unhealthy on this node +device Ascend910B2 is unhealthy on this node +``` + +The messages may repeat on every reconciliation cycle and waiting for the plugin to retry does not provide the missing resource. + +Check the node's resources: + +```bash +kubectl get node \ + -o json | jq '.status.capacity' +``` + +and: + +```bash +kubectl get node \ + -o json | jq '.status.allocatable' +``` + +### Why this happens + +The mock device plugin uses node metadata when constructing its virtual device resources. For a mock-only environment, two pieces of node state are important: + +1. A vendor registration annotation describing the mock device. +2. A non-zero vendor count resource in the node's `status.capacity`. + +The count resource is used by the plugin's health check. The relevant logic checks whether the node already has a non-zero value for the configured count resource and if the resource is absent, the device is considered unhealthy and resource construction returns without advertising the mock device resources. +For NVIDIA, the count resource is: + +```text +nvidia.com/gpu +``` + +On a fresh mock-only node, this resource may not exist yet which can create a bootstrap dependency: + +```text +CheckHealthy() -> count resource absent from node.Status.Capacity -> device considered unhealthy ->resource registration does not proceed ->count resource is still absent ->CheckHealthy() +``` +### Diagnose the health gate + +Check whether the count resource exists: +[code block 1 and 2] +```bash +kubectl get node \ + -o json | jq '.status.capacity["nvidia.com/gpu"]' +``` + +A missing value or a zero value indicates that the NVIDIA count resource has not satisfied the health check. +Also check the vendor registration annotation: + +```bash +kubectl get node \ + -o json | jq -r '.metadata.annotations["hami.io/node-nvidia-register"]' +``` + +The annotation should contain the mock device configuration being used for the test. + +### Bootstrap a mock-only NVIDIA test node + +For local testing, seed a non-zero NVIDIA count resource: + +```bash +kubectl patch node \ + --subresource=status \ + --type=json \ + -p '[{"op":"add","path":"/status/capacity/nvidia.com~1gpu","value":"1"}]' +``` + +Then provide the corresponding mock-device registration annotation. + +For example: + +```bash +kubectl annotate node \ + 'hami.io/node-nvidia-register=[{"id":"GPU-MOCK-0","count":1,"devmem":81920,"devcore":100,"type":"NVIDIA-A100-SXM4-80GB","health":true,"numa":0,"mode":"hami-core"}]' +``` + +Use values appropriate to the mock device you intend to simulate. The status patch is a local testing/bootstrap workaround and not a replacement for a normal device-plugin registration flow and should not be applied to production nodes as a way of claiming that hardware exists. After the mock plugin reconciles the node, verify the node's allocatable resources: + +```bash +kubectl get node \ + -o json | jq '.status.allocatable + | with_entries(select(.key | test("nvidia.com")))' +``` +For an NVIDIA mock device, resources such as the following may appear: + +```text +nvidia.com/gpu +nvidia.com/gpumem +nvidia.com/gpucores +nvidia.com/gpumem-percentage +``` + +The exact set depends on the mock device configuration. You can verify the count resource and registration annotation by doing codeblock 1 and 2. + +The vendor count resource is used as a health gate and doesnt describe the complete properties of the mock device. The mock device configuration, including the simulated device properties, is provided through the vendor registration configuration.This is suitable for testing HAMi's scheduler and resource-placement behavior without physical accelerator hardware and does not provide realtime GPU execution; NVIDIA driver/runtime validation; accelerator performance measurements; validation of hardware-specific runtime behavior.Use a real supported accelerator environment when testing functionality that depends on physical hardware or vendor runtime components. + +--- + +## Problem 2: Scheduler pod enters `ImagePullBackOff` while pulling `kube-scheduler` + +During validation of HAMi on a Fedora/`kind` environment, the scheduler pod also encountered an image-pull failure for the configured `kube-scheduler` image. + +The observed error was: + +```text +failed to resolve reference ...: +tls: failed to verify certificate: +x509: certificate signed by unknown authority +``` + +The HAMi scheduler extender image from Docker Hub pulled successfully in the same environment.This behavior was reproduced in the validation environment, but the investigation did not establish whether the TLS failure is: + +* specific to that environment's certificate trust configuration; or +* a broader issue affecting other networks using the configured registry mirror. + +For that reason, this observation is recorded here for visibility rather than presented as a general HAMi failure with a universal workaround. + +### Solution: +When this occurs, inspect the scheduler pod events: + +```bash +kubectl describe pod -n kube-system +``` + +and verify which image and registry failed: + +```bash +kubectl get pod -n kube-system \ + -o json | jq '.spec.containers[].image' +``` + +If the failure is caused by TLS certificate verification, investigate the container runtime's trust configuration and the accessibility of the configured registry from the affected node. + +--- +GPU-less kind validation environment + +The troubleshooting above was validated using: + +OS: Fedora Linux +CPU: Intel Core i3-5005U, 2 cores / 4 threads +GPU: Intel HD Graphics 5500 integrated graphics; no CUDA-capable discrete GPU +RAM: 16 GB +Cluster kind, single control-plane node +Kubernetes v1.36.1 +HAMi chart hami-charts/hami +HAMi image docker.io/projecthami/hami:v2.9.0 +mock-device-plugin main at time of testing + +This environment was intentionally selected to represent a contributor attempting to evaluate HAMi without access to a discrete accelerator. + +For a disposable kind cluster, clean up with: + +`kind delete cluster --name ` + +-- From 59b423c2d605e80f4e70ffcfc6c834c8b0ca2593 Mon Sep 17 00:00:00 2001 From: Rohini Vishu <200266036+RohiniVishu@users.noreply.github.com> Date: Mon, 31 Aug 2026 03:48:44 +0530 Subject: [PATCH 2/2] docs: address review feedback, terminology cleanup, rename Solution to Diagnosis Signed-off-by: Rohini Vishu <200266036+RohiniVishu@users.noreply.github.com> --- docs/troubleshooting/troubleshooting.md | 113 +++++++++++++++--------- 1 file changed, 70 insertions(+), 43 deletions(-) diff --git a/docs/troubleshooting/troubleshooting.md b/docs/troubleshooting/troubleshooting.md index 018ecea60..a7a883d6e 100644 --- a/docs/troubleshooting/troubleshooting.md +++ b/docs/troubleshooting/troubleshooting.md @@ -163,8 +163,7 @@ devicePlugin: ## HAMi device plugin fails on a GPU-less node -HAMi's standard installation expects supported accelerator hardware and the corresponding vendor runtime to be available on nodes where the real device plugin is scheduled. -Use `mock-device-plugin` when the goal is to test HAMi scheduling behavior without executing workloads on a physical accelerator.The mock plugin is intended for development/testing and does not provide actual GPU execution or GPU performance validation. +HAMi's standard installation expects supported accelerator hardware and the corresponding vendor runtime to be available on nodes where the real device plugin is scheduled. Use `mock-device-plugin` when the goal is to test HAMi scheduling behavior without executing workloads on a physical accelerator. The mock plugin is intended for development and testing, it does not provide actual GPU execution or GPU performance validation. When testing HAMi on a machine without a supported NVIDIA GPU, the real NVIDIA device plugin cannot initialize successfully. For scheduler development and testing without physical accelerator hardware, use the HAMi `mock-device-plugin`. @@ -197,12 +196,11 @@ error starting plugins: ... invalid device discovery strategy ### Why it happens -The real NVIDIA device plugin uses the NVIDIA discovery strategy and requires the NVIDIA driver/container runtime stack on the host.A GPU-less development machine does not provide that stack. If the node is labelled so that the real device plugin is scheduled there, the plugin attempts to initialize and fails. -The standard HAMi Quick Start is intended for an environment with the required accelerator prerequisites. +The real NVIDIA device plugin uses the NVIDIA discovery strategy and requires the NVIDIA driver/container runtime stack on the host. A GPU-less development machine does not provide that stack. If the node is labelled so that the real device plugin is scheduled there, the plugin attempts to initialize and fails. The standard HAMi Quick Start is intended for an environment with the required accelerator prerequisites. --- -## Problem 1: Mock device plugin reports devices as unhealthy +### Problem 1: Mock device plugin reports devices as unhealthy Use this section when the HAMi `mock-device-plugin` is running but repeatedly reports configured devices as unhealthy and the corresponding vendor resources do not appear in the node's `Allocatable` resources. @@ -226,7 +224,7 @@ device Ascend910A is unhealthy on this node device Ascend910B2 is unhealthy on this node ``` -The messages may repeat on every reconciliation cycle and waiting for the plugin to retry does not provide the missing resource. +The messages may repeat on every reconciliation cycle, and waiting for the plugin to retry does not provide the missing resource. Check the node's resources: @@ -242,36 +240,35 @@ kubectl get node \ -o json | jq '.status.allocatable' ``` -### Why this happens +#### Why this happens -The mock device plugin uses node metadata when constructing its virtual device resources. For a mock-only environment, two pieces of node state are important: +The mock-device-plugin uses node metadata when constructing its virtual device resources. For a mock-only environment, two pieces of node state are important: -1. A vendor registration annotation describing the mock device. -2. A non-zero vendor count resource in the node's `status.capacity`. +1. A registration annotation describing the mock device. +2. A non-zero count resource in the node's `status.capacity`. -The count resource is used by the plugin's health check. The relevant logic checks whether the node already has a non-zero value for the configured count resource and if the resource is absent, the device is considered unhealthy and resource construction returns without advertising the mock device resources. -For NVIDIA, the count resource is: +The count resource is used by the plugin's health check. The relevant logic checks whether the node already has a non-zero value for the configured count resource. If the resource is absent, the device is considered unhealthy and resource construction returns without advertising the mock device resources. For NVIDIA, the count resource is: ```text nvidia.com/gpu ``` -On a fresh mock-only node, this resource may not exist yet which can create a bootstrap dependency: +On a fresh mock-only node, this resource may not exist yet, which can create a bootstrap dependency: ```text -CheckHealthy() -> count resource absent from node.Status.Capacity -> device considered unhealthy ->resource registration does not proceed ->count resource is still absent ->CheckHealthy() +CheckHealthy() -> count resource absent from node.Status.Capacity -> device considered unhealthy -> resource registration does not proceed -> count resource is still absent -> CheckHealthy() ``` -### Diagnose the health gate + +#### Diagnose the health gate Check whether the count resource exists: -[code block 1 and 2] + ```bash kubectl get node \ -o json | jq '.status.capacity["nvidia.com/gpu"]' ``` -A missing value or a zero value indicates that the NVIDIA count resource has not satisfied the health check. -Also check the vendor registration annotation: +A missing value or a zero value indicates that the NVIDIA count resource has not satisfied the health check. Also check the registration annotation: ```bash kubectl get node \ @@ -280,7 +277,7 @@ kubectl get node \ The annotation should contain the mock device configuration being used for the test. -### Bootstrap a mock-only NVIDIA test node +#### Bootstrap a mock-only NVIDIA test node For local testing, seed a non-zero NVIDIA count resource: @@ -300,13 +297,16 @@ kubectl annotate node \ 'hami.io/node-nvidia-register=[{"id":"GPU-MOCK-0","count":1,"devmem":81920,"devcore":100,"type":"NVIDIA-A100-SXM4-80GB","health":true,"numa":0,"mode":"hami-core"}]' ``` -Use values appropriate to the mock device you intend to simulate. The status patch is a local testing/bootstrap workaround and not a replacement for a normal device-plugin registration flow and should not be applied to production nodes as a way of claiming that hardware exists. After the mock plugin reconciles the node, verify the node's allocatable resources: +Use values appropriate to the mock device you intend to simulate. The status patch is a local testing/bootstrap workaround and not a replacement for a normal device-plugin registration flow. It should not be applied to production nodes to falsely advertise hardware that is not present. + +After the mock plugin reconciles the node, verify the node's allocatable resources: ```bash kubectl get node \ -o json | jq '.status.allocatable | with_entries(select(.key | test("nvidia.com")))' ``` + For an NVIDIA mock device, resources such as the following may appear: ```text @@ -316,15 +316,33 @@ nvidia.com/gpucores nvidia.com/gpumem-percentage ``` -The exact set depends on the mock device configuration. You can verify the count resource and registration annotation by doing codeblock 1 and 2. +The exact set depends on the mock device configuration. You can verify the count resource and registration annotation using the following commands. + +Check whether the count resource exists: + +```bash +kubectl get node \ + -o json | jq '.status.capacity["nvidia.com/gpu"]' +``` + +Check the registration annotation: -The vendor count resource is used as a health gate and doesnt describe the complete properties of the mock device. The mock device configuration, including the simulated device properties, is provided through the vendor registration configuration.This is suitable for testing HAMi's scheduler and resource-placement behavior without physical accelerator hardware and does not provide realtime GPU execution; NVIDIA driver/runtime validation; accelerator performance measurements; validation of hardware-specific runtime behavior.Use a real supported accelerator environment when testing functionality that depends on physical hardware or vendor runtime components. +```bash +kubectl get node \ + -o json | jq -r '.metadata.annotations["hami.io/node-nvidia-register"]' +``` + +The count resource is used as a health gate and doesn't describe the complete properties of the mock device. The mock device configuration, including the simulated device properties, is provided through the registration annotation. + +This is suitable for testing HAMi's scheduler and resource-placement behavior without physical accelerator hardware and does not provide real-time GPU execution, NVIDIA driver/runtime validation, accelerator performance measurements, or validation of hardware-specific runtime behavior. + +Use a real supported accelerator environment when testing functionality that depends on physical hardware or vendor runtime components. --- -## Problem 2: Scheduler pod enters `ImagePullBackOff` while pulling `kube-scheduler` +### Problem 2: Scheduler pod enters `ImagePullBackOff` while pulling `kube-scheduler` -During validation of HAMi on a Fedora/`kind` environment, the scheduler pod also encountered an image-pull failure for the configured `kube-scheduler` image. +During validation of HAMi on a Fedora Linux with `kind` environment, the scheduler pod also encountered an image pull failure for the configured `kube-scheduler` image. The observed error was: @@ -334,48 +352,57 @@ tls: failed to verify certificate: x509: certificate signed by unknown authority ``` -The HAMi scheduler extender image from Docker Hub pulled successfully in the same environment.This behavior was reproduced in the validation environment, but the investigation did not establish whether the TLS failure is: +The HAMi scheduler extender image from Docker Hub pulled successfully in the same environment. This behavior was reproduced in the validation environment, but the investigation did not establish whether the TLS failure is: -* specific to that environment's certificate trust configuration; or -* a broader issue affecting other networks using the configured registry mirror. +- specific to that environment's certificate trust configuration; or +- a broader issue affecting other networks using the configured registry mirror. -For that reason, this observation is recorded here for visibility rather than presented as a general HAMi failure with a universal workaround. +For that reason, this observation is recorded here for visibility rather than being presented as a general HAMi failure with a universal workaround. + +#### Diagnosis -### Solution: When this occurs, inspect the scheduler pod events: ```bash kubectl describe pod -n kube-system ``` -and verify which image and registry failed: +and verify which image and registry were involved in the failure: ```bash kubectl get pod -n kube-system \ -o json | jq '.spec.containers[].image' ``` -If the failure is caused by TLS certificate verification, investigate the container runtime's trust configuration and the accessibility of the configured registry from the affected node. +If the failure is caused by TLS certificate verification, investigate the container runtime's trust configuration and whether the configured registry is accessible from the affected node. + +This is a diagnosis. If a fix is identified for your environment, please follow-up a PR or a comment so that it can be added here for other contributors. --- -GPU-less kind validation environment + +### GPU-less kind validation environment The troubleshooting above was validated using: -OS: Fedora Linux -CPU: Intel Core i3-5005U, 2 cores / 4 threads -GPU: Intel HD Graphics 5500 integrated graphics; no CUDA-capable discrete GPU -RAM: 16 GB -Cluster kind, single control-plane node -Kubernetes v1.36.1 -HAMi chart hami-charts/hami -HAMi image docker.io/projecthami/hami:v2.9.0 -mock-device-plugin main at time of testing +| Component | Configuration | +| ------------------ | ------------------------------------------------------------------------ | +| OS | Fedora Linux | +| CPU | Intel Core i3-5005U, 2 cores / 4 threads | +| GPU | Intel HD Graphics 5500 integrated graphics; no CUDA-capable discrete GPU | +| RAM | 16 GB | +| Cluster | kind, single control-plane node | +| Kubernetes | v1.36.1 | +| HAMi chart | `hami-charts/hami` | +| HAMi image | `docker.io/projecthami/hami:v2.9.0` | +| mock-device-plugin | `main` at time of testing | +| | | This environment was intentionally selected to represent a contributor attempting to evaluate HAMi without access to a discrete accelerator. For a disposable kind cluster, clean up with: -`kind delete cluster --name ` +```bash +kind delete cluster --name +``` --- +---