Enhance troubleshooting guide for HAMi installation

docs/troubleshooting/troubleshooting.md

@@ -160,3 +160,249 @@ devicePlugin:
 ```
 
 :::
+
+## HAMi device plugin fails on a GPU-less node
+
+HAMi's standard installation expects supported accelerator hardware and the corresponding vendor runtime to be available on nodes where the real device plugin is scheduled. Use `mock-device-plugin` when the goal is to test HAMi scheduling behavior without executing workloads on a physical accelerator. The mock plugin is intended for development and testing, it does not provide actual GPU execution or GPU performance validation.
+
+When testing HAMi on a machine without a supported NVIDIA GPU, the real NVIDIA device plugin cannot initialize successfully. For scheduler development and testing without physical accelerator hardware, use the HAMi `mock-device-plugin`.
+
+### Error seen
+
+After installing HAMi, check the pods:
+
+```bash
+kubectl get pods -n kube-system
+```
+
+The HAMi device plugin may enter `CrashLoopBackOff`.
+
+Check its logs:
+
+```bash
+kubectl logs -n kube-system <hami-device-plugin-pod> \
+  -c device-plugin --previous
+```
+
+The logs may contain:
+
+```text
+Incompatible strategy detected auto
+If this is a GPU node, did you configure the NVIDIA Container Toolkit?
+If this is not a GPU node, you should set up a toleration or nodeSelector
+to only deploy this plugin on GPU nodes
+error starting plugins: ... invalid device discovery strategy
+```
+
+### Why it happens
+
+The real NVIDIA device plugin uses the NVIDIA discovery strategy and requires the NVIDIA driver/container runtime stack on the host. A GPU-less development machine does not provide that stack. If the node is labelled so that the real device plugin is scheduled there, the plugin attempts to initialize and fails. The standard HAMi Quick Start is intended for an environment with the required accelerator prerequisites.
+
+---
+
+### Problem 1: Mock device plugin reports devices as unhealthy
+
+Use this section when the HAMi `mock-device-plugin` is running but repeatedly reports configured devices as unhealthy and the corresponding vendor resources do not appear in the node's `Allocatable` resources.
+
+Check the mock plugin pod:
+
+```bash
+kubectl get pods -n kube-system | grep mock-device-plugin
+```
+
+Then inspect its logs:
+
+```bash
+kubectl logs -n kube-system <mock-device-plugin-pod> --tail=100
+```
+
+You may see repeated messages such as:
+
+```text
+device NVIDIA is unhealthy on this node
+device Ascend910A is unhealthy on this node
+device Ascend910B2 is unhealthy on this node
+```
+
+The messages may repeat on every reconciliation cycle, and waiting for the plugin to retry does not provide the missing resource.
+
+Check the node's resources:
+
+```bash
+kubectl get node <node-name> \
+  -o json | jq '.status.capacity'
+```
+
+and:
+
+```bash
+kubectl get node <node-name> \
+  -o json | jq '.status.allocatable'
+```
+
+#### Why this happens
+
+The mock-device-plugin uses node metadata when constructing its virtual device resources. For a mock-only environment, two pieces of node state are important:
+
+1. A registration annotation describing the mock device.
+2. A non-zero count resource in the node's `status.capacity`.
+
+The count resource is used by the plugin's health check. The relevant logic checks whether the node already has a non-zero value for the configured count resource. If the resource is absent, the device is considered unhealthy and resource construction returns without advertising the mock device resources. For NVIDIA, the count resource is:
+
+```text
+nvidia.com/gpu
+```
+
+On a fresh mock-only node, this resource may not exist yet, which can create a bootstrap dependency:
+
+```text
+CheckHealthy() -> count resource absent from node.Status.Capacity -> device considered unhealthy -> resource registration does not proceed -> count resource is still absent -> CheckHealthy()
+```
+
+#### Diagnose the health gate
+
+Check whether the count resource exists:
+
+```bash
+kubectl get node <node-name> \
+  -o json | jq '.status.capacity["nvidia.com/gpu"]'
+```
+
+A missing value or a zero value indicates that the NVIDIA count resource has not satisfied the health check. Also check the registration annotation:
+
+```bash
+kubectl get node <node-name> \
+  -o json | jq -r '.metadata.annotations["hami.io/node-nvidia-register"]'
+```
+
+The annotation should contain the mock device configuration being used for the test.
+
+#### Bootstrap a mock-only NVIDIA test node
+
+For local testing, seed a non-zero NVIDIA count resource:
+
+```bash
+kubectl patch node <node-name> \
+  --subresource=status \
+  --type=json \
+  -p '[{"op":"add","path":"/status/capacity/nvidia.com~1gpu","value":"1"}]'
+```
+
+Then provide the corresponding mock-device registration annotation.
+
+For example:
+
+```bash
+kubectl annotate node <node-name> \
+  'hami.io/node-nvidia-register=[{"id":"GPU-MOCK-0","count":1,"devmem":81920,"devcore":100,"type":"NVIDIA-A100-SXM4-80GB","health":true,"numa":0,"mode":"hami-core"}]'
+```
+
+Use values appropriate to the mock device you intend to simulate. The status patch is a local testing/bootstrap workaround and not a replacement for a normal device-plugin registration flow. It should not be applied to production nodes to falsely advertise hardware that is not present.
+
+After the mock plugin reconciles the node, verify the node's allocatable resources:
+
+```bash
+kubectl get node <node-name> \
+  -o json | jq '.status.allocatable
+    | with_entries(select(.key | test("nvidia.com")))'
+```
+
+For an NVIDIA mock device, resources such as the following may appear:
+
+```text
+nvidia.com/gpu
+nvidia.com/gpumem
+nvidia.com/gpucores
+nvidia.com/gpumem-percentage
+```
+
+The exact set depends on the mock device configuration. You can verify the count resource and registration annotation using the following commands.
+
+Check whether the count resource exists:
+
+```bash
+kubectl get node <node-name> \
+  -o json | jq '.status.capacity["nvidia.com/gpu"]'
+```
+
+Check the registration annotation:
+
+```bash
+kubectl get node <node-name> \
+  -o json | jq -r '.metadata.annotations["hami.io/node-nvidia-register"]'
+```
+
+The count resource is used as a health gate and doesn't describe the complete properties of the mock device. The mock device configuration, including the simulated device properties, is provided through the registration annotation.
+
+This is suitable for testing HAMi's scheduler and resource-placement behavior without physical accelerator hardware and does not provide real-time GPU execution, NVIDIA driver/runtime validation, accelerator performance measurements, or validation of hardware-specific runtime behavior.
+
+Use a real supported accelerator environment when testing functionality that depends on physical hardware or vendor runtime components.
+
+---
+
+### Problem 2: Scheduler pod enters `ImagePullBackOff` while pulling `kube-scheduler`
+
+During validation of HAMi on a Fedora Linux with `kind` environment, the scheduler pod also encountered an image pull failure for the configured `kube-scheduler` image.
+
+The observed error was:
+
+```text
+failed to resolve reference ...:
+tls: failed to verify certificate:
+x509: certificate signed by unknown authority
+```
+
+The HAMi scheduler extender image from Docker Hub pulled successfully in the same environment. This behavior was reproduced in the validation environment, but the investigation did not establish whether the TLS failure is:
+
+- specific to that environment's certificate trust configuration; or
+- a broader issue affecting other networks using the configured registry mirror.
+
+For that reason, this observation is recorded here for visibility rather than being presented as a general HAMi failure with a universal workaround.
+
+#### Diagnosis
+
+When this occurs, inspect the scheduler pod events:
+
+```bash
+kubectl describe pod -n kube-system <hami-scheduler-pod>
+```
+
+and verify which image and registry were involved in the failure:
+
+```bash
+kubectl get pod -n kube-system <hami-scheduler-pod> \
+  -o json | jq '.spec.containers[].image'
+```
+
+If the failure is caused by TLS certificate verification, investigate the container runtime's trust configuration and whether the configured registry is accessible from the affected node.
+
+This is a diagnosis. If a fix is identified for your environment, please follow-up a PR or a comment so that it can be added here for other contributors.
+
+---
+
+### GPU-less kind validation environment
+
+The troubleshooting above was validated using:
+
+| Component          | Configuration                                                            |
+| ------------------ | ------------------------------------------------------------------------ |
+| OS                 | Fedora Linux                                                             |
+| CPU                | Intel Core i3-5005U, 2 cores / 4 threads                                 |
+| GPU                | Intel HD Graphics 5500 integrated graphics; no CUDA-capable discrete GPU |
+| RAM                | 16 GB                                                                    |
+| Cluster            | kind, single control-plane node                                          |
+| Kubernetes         | v1.36.1                                                                  |
+| HAMi chart         | `hami-charts/hami`                                                       |
+| HAMi image         | `docker.io/projecthami/hami:v2.9.0`                                      |
+| mock-device-plugin | `main` at time of testing                                                |
+|                    |                                                                          |
+
+This environment was intentionally selected to represent a contributor attempting to evaluate HAMi without access to a discrete accelerator.
+
+For a disposable kind cluster, clean up with:
+
+```bash
+kind delete cluster --name <cluster-name>
+```
+
+---