From 11b09f9cfeff6f8fe3dfd293c572b4b12a9c0dab Mon Sep 17 00:00:00 2001 From: blackdragoon26 Date: Tue, 11 Aug 2026 18:35:06 +0530 Subject: [PATCH 1/4] docs(scheduler): document per-pod scoring weights Signed-off-by: blackdragoon26 --- docs/developers/scheduling.md | 65 ++++++++++++++++++++++++++++++----- 1 file changed, 57 insertions(+), 8 deletions(-) diff --git a/docs/developers/scheduling.md b/docs/developers/scheduling.md index 4597384e0..62982c106 100644 --- a/docs/developers/scheduling.md +++ b/docs/developers/scheduling.md @@ -132,46 +132,95 @@ In `Spread` policy, `Node2` is selected. ![HAMi GPU scheduler policy diagram, comparing Binpack and Spread scores on each card](/img/docs/common/developers/scheduling/gpu-scheduler-policy-demo.png) +#### Per-Pod device scoring weights + +By default, HAMi gives equal influence to predicted virtual-device slot, device-core, and device-memory utilization when it scores a physical device. To change that balance for one workload, add the `hami.io/device-scoring-weights` annotation to the Pod: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: memory-weighted-gpu-pod + annotations: + hami.io/device-scoring-weights: "slot=1,core=1,memory=3" +spec: + containers: + - name: workload + image: ubuntu:22.04 + command: ["bash", "-c", "sleep 86400"] + resources: + limits: + nvidia.com/gpu: 1 + nvidia.com/gpumem-percentage: 40 +``` + +HAMi predicts each candidate device's utilization after placing the request, then calculates its device score as follows: + +```text +score = 10 * ( + slotWeight * predictedSlotUtilization + + coreWeight * predictedCoreUtilization + + memoryWeight * predictedMemoryUtilization +) +``` + +The annotation must contain the `slot`, `core`, and `memory` keys. Each value must be a non-negative integer, and at least one value must be greater than zero. Key order and surrounding whitespace do not matter. If the annotation is absent, HAMi uses `slot=1,core=1,memory=1`, which preserves the default scoring behavior. An invalid annotation prevents the Pod from being scheduled until the annotation is corrected. + +For example, consider two candidate GPUs after accounting for a Pod that requests one vGPU and 40% device memory: + +| Device | Predicted slot utilization | Predicted core utilization | Predicted memory utilization | +| ------ | -------------------------: | -------------------------: | ---------------------------: | +| GPU A | 0.2 | 0.9 | 0.5 | +| GPU B | 0.8 | 0.1 | 0.6 | + +With the default `1:1:1` weights, GPU A scores `16` and GPU B scores `15`, so `binpack` prefers GPU A. With `slot=1,core=1,memory=3`, GPU A scores `26` and GPU B scores `27`, so `binpack` prefers GPU B. Under `spread`, the lower score is preferred instead. + +The annotation changes only the utilization score used to order candidate devices. It does not bypass device fit or capacity checks, mutex rules, NUMA or topology constraints, or vendor-specific `Fit` behavior. These constraints keep their existing precedence; when topology candidates are otherwise tied, their utilization-score ordering can act as the tie-breaker. + #### Binpack -Binpack mainly focuses on the computing power and video memory usage of each card. The more it is used, the higher the score. +Binpack prefers the card with the higher device-utilization score. The following default-weight example assumes each card has ten virtual-device slots and no slot is currently in use: ```text -score: ((request.core + used.core) / allocatable.core + (request.mem + used.mem) / allocatable.mem)) * 10 +score: ((request.slot + used.slot) / allocatable.slot + + (request.core + used.core) / allocatable.core + + (request.mem + used.mem) / allocatable.mem) * 10 ``` 1. Binpack scoring information for GPU 1 is as follows ```text -GPU1 Score: ((20+10)/100 + (1000+2000)/8000)) * 10 = 6.75 +GPU1 Score: ((1+0)/10 + (20+10)/100 + (1000+2000)/8000) * 10 = 7.75 ``` 1. Binpack scoring information for GPU 2 is as follows ```text -GPU2 Score: ((20+70)/100 + (1000+6000)/8000)) * 10 = 17.75 +GPU2 Score: ((1+0)/10 + (20+70)/100 + (1000+6000)/8000) * 10 = 18.75 ``` In `Binpack` policy, `GPU2` is selected. #### Spread -Spread mainly focuses on the computing power and video memory usage of each card. The less it is used, the higher the score. +Spread prefers the card with the lower device-utilization score. Using the same default-weight example: ```text -score: ((request.core + used.core) / allocatable.core + (request.mem + used.mem) / allocatable.mem)) * 10 +score: ((request.slot + used.slot) / allocatable.slot + + (request.core + used.core) / allocatable.core + + (request.mem + used.mem) / allocatable.mem) * 10 ``` 1. Spread scoring information for GPU 1 is as follows ```text -GPU1 Score: ((20+10)/100 + (1000+2000)/8000)) * 10 = 6.75 +GPU1 Score: ((1+0)/10 + (20+10)/100 + (1000+2000)/8000) * 10 = 7.75 ``` 1. Spread scoring information for GPU 2 is as follows ```text -GPU2 Score: ((20+70)/100 + (1000+6000)/8000)) * 10 = 17.75 +GPU2 Score: ((1+0)/10 + (20+70)/100 + (1000+6000)/8000) * 10 = 18.75 ``` In `Spread` policy, `GPU1` is selected. From 9715d6e3dc987831886a88cbb28987038313a29e Mon Sep 17 00:00:00 2001 From: blackdragoon26 Date: Thu, 20 Aug 2026 22:45:08 +0530 Subject: [PATCH 2/4] docs(scheduler): add scoring weights references Signed-off-by: blackdragoon26 --- docs/userguide/configure.md | 1 + .../current/developers/scheduling.md | 65 ++++++++++++++++--- .../current/userguide/configure.md | 1 + 3 files changed, 59 insertions(+), 8 deletions(-) diff --git a/docs/userguide/configure.md b/docs/userguide/configure.md index e3ec92e6f..681f8c85b 100644 --- a/docs/userguide/configure.md +++ b/docs/userguide/configure.md @@ -81,6 +81,7 @@ helm install hami hami-charts/hami --set devicePlugin.deviceMemoryScaling=5 -n k | `nvidia.com/use-gputype` | String | If set, devices allocated by this pod MUST be one of the types defined in this string. | `"Tesla V100-PCIE-32GB, NVIDIA A10"` | | `hami.io/node-scheduler-policy` | String | GPU node scheduling policy: `"binpack"` allocates the pod to used GPU nodes for execution. `"spread"` allocates the pod to different GPU nodes for execution. | `"binpack"` or `"spread"` | | `hami.io/gpu-scheduler-policy` | String | GPU scheduling policy: `"binpack"` allocates the pod to the same GPU card for execution. `"spread"` allocates the pod to different GPU cards for execution. `"mutex"` allocates the pod only to a GPU card with no other workloads, giving it exclusive use of that card. | `"binpack"`, `"spread"` or `"mutex"` | +| `hami.io/device-scoring-weights` | String | Relative weights of virtual-device slot, device-core, and device-memory utilization in physical-device scoring. All three weights are required, must be non-negative integers, and at least one must be positive. | `"slot=1,core=1,memory=3"` | | `nvidia.com/vgpu-mode` | String | The type of vGPU instance this pod wishes to use. | `"hami-core"` or `"mig"` | ## Container Configs: Env diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/developers/scheduling.md b/i18n/zh/docusaurus-plugin-content-docs/current/developers/scheduling.md index 8beb35c06..0659e2f0b 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/developers/scheduling.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/developers/scheduling.md @@ -133,46 +133,95 @@ Node2 score: ((1+2)/4) * 10= 7.5 ![HAMi GPU 调度策略示意图，展示在单卡上的 Binpack 与 Spread 评分对比](/img/docs/common/developers/scheduling/gpu-scheduler-policy-demo.png) +#### 每个 Pod 的设备评分权重 + +默认情况下，HAMi 在计算物理设备得分时，会让预测的虚拟设备槽位、设备核心和设备显存利用率具有相同的影响。若要为某个工作负载调整它们的相对影响，可为 Pod 添加 `hami.io/device-scoring-weights` 注解： + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: memory-weighted-gpu-pod + annotations: + hami.io/device-scoring-weights: "slot=1,core=1,memory=3" +spec: + containers: + - name: workload + image: ubuntu:22.04 + command: ["bash", "-c", "sleep 86400"] + resources: + limits: + nvidia.com/gpu: 1 + nvidia.com/gpumem-percentage: 40 +``` + +HAMi 会预测在放置请求后每个候选设备的利用率，然后按以下公式计算设备得分： + +```text +score = 10 * ( + slotWeight * predictedSlotUtilization + + coreWeight * predictedCoreUtilization + + memoryWeight * predictedMemoryUtilization +) +``` + +注解必须包含 `slot`、`core` 和 `memory` 三个键。每个值都必须是非负整数，并且至少有一个值大于零。键的顺序以及两侧的空格不会影响解析。如果未设置该注解，HAMi 将使用 `slot=1,core=1,memory=1`，从而保持默认评分行为。无效的注解会使 Pod 无法被调度，直到该注解被修正。 + +例如，假设一个 Pod 请求 1 个 vGPU 和 40% 的设备显存，并且在计入该请求后有两个候选 GPU： + +| 设备 | 预测槽位利用率 | 预测核心利用率 | 预测显存利用率 | +| ----- | -------------: | -------------: | -------------: | +| GPU A | 0.2 | 0.9 | 0.5 | +| GPU B | 0.8 | 0.1 | 0.6 | + +使用默认的 `1:1:1` 权重时，GPU A 得分为 `16`，GPU B 得分为 `15`，因此 `binpack` 会优先选择 GPU A。使用 `slot=1,core=1,memory=3` 时，GPU A 得分为 `26`，GPU B 得分为 `27`，因此 `binpack` 会优先选择 GPU B。在 `spread` 策略下，则会优先选择得分较低的设备。 + +该注解只会改变用于排序候选设备的利用率得分。它不会绕过设备适配或容量检查、mutex 规则、NUMA 或拓扑约束，也不会改变厂商特定的 `Fit` 行为。这些约束仍保留现有的优先级；当拓扑候选项在其他方面相同时，可以使用利用率得分顺序作为决胜条件。 + #### Binpack -Binpack 主要关注每张卡的计算能力和显存使用情况。使用越多，得分越高。 +Binpack 优先选择设备利用率得分较高的卡。以下默认权重示例假设每张卡有 10 个虚拟设备槽位，并且当前没有槽位被使用： ```text -score: ((request.core + used.core) / allocatable.core + (request.mem + used.mem) / allocatable.mem)) * 10 +score: ((request.slot + used.slot) / allocatable.slot + + (request.core + used.core) / allocatable.core + + (request.mem + used.mem) / allocatable.mem) * 10 ``` 1. GPU1 的 Binpack 评分信息如下 ```text -GPU1 Score: ((20+10)/100 + (1000+2000)/8000)) * 10 = 6.75 +GPU1 Score: ((1+0)/10 + (20+10)/100 + (1000+2000)/8000) * 10 = 7.75 ``` 1. GPU2 的 Binpack 评分信息如下 ```text -GPU2 Score: ((20+70)/100 + (1000+6000)/8000)) * 10 = 17.75 +GPU2 Score: ((1+0)/10 + (20+70)/100 + (1000+6000)/8000) * 10 = 18.75 ``` 因此，在 `Binpack` 策略中我们可以选择 `GPU2`。 #### Spread -Spread 主要关注每张卡的计算能力和显存使用情况。使用越少，得分越高。 +Spread 优先选择设备利用率得分较低的卡。使用相同的默认权重示例： ```text -score: ((request.core + used.core) / allocatable.core + (request.mem + used.mem) / allocatable.mem)) * 10 +score: ((request.slot + used.slot) / allocatable.slot + + (request.core + used.core) / allocatable.core + + (request.mem + used.mem) / allocatable.mem) * 10 ``` 1. GPU1 的 Spread 评分信息如下 ```text -GPU1 Score: ((20+10)/100 + (1000+2000)/8000)) * 10 = 6.75 +GPU1 Score: ((1+0)/10 + (20+10)/100 + (1000+2000)/8000) * 10 = 7.75 ``` 1. GPU2 的 Spread 评分信息如下 ```text -GPU2 Score: ((20+70)/100 + (1000+6000)/8000)) * 10 = 17.75 +GPU2 Score: ((1+0)/10 + (20+70)/100 + (1000+6000)/8000) * 10 = 18.75 ``` 因此，在 `Spread` 策略中我们可以选择 `GPU1`。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/configure.md b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/configure.md index 3c8816922..8df4a874a 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/configure.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/configure.md @@ -83,6 +83,7 @@ helm install hami hami-charts/hami --set devicePlugin.deviceMemoryScaling=5 -n k | `nvidia.com/use-gputype` | 字符串 | 如果设置了此字段，则该 Pod 分配的设备**必须**是此字符串中定义的 GPU 类型之一。 | `"Tesla V100-PCIE-32GB, NVIDIA A10"` | | `hami.io/node-scheduler-policy` | 字符串 | GPU 节点调度策略：`"binpack"` 表示将 Pod 分配到已有负载的 GPU 节点上执行，`"spread"` 表示分配到不同的 GPU 节点上执行。 | `"binpack"` 或 `"spread"` | | `hami.io/gpu-scheduler-policy` | 字符串 | GPU 卡调度策略：`"binpack"` 表示将 Pod 分配到同一块 GPU 卡上执行，`"spread"` 表示分配到不同的 GPU 卡上执行。 | `"binpack"` 或 `"spread"` | +| `hami.io/device-scoring-weights` | 字符串 | 物理设备评分中虚拟设备槽位、设备核心和设备显存利用率的相对权重。必须提供全部三个权重，值必须为非负整数，并且至少有一个权重大于零。 | `"slot=1,core=1,memory=3"` | | `nvidia.com/vgpu-mode` | 字符串 | 指定该 Pod 希望使用的 vGPU 实例类型。 | `"hami-core"` 或 `"mig"` | ## 容器配置：环境变量 From b592b4e79932253a42566d178021f6fba8c5c52a Mon Sep 17 00:00:00 2001 From: blackdragoon26 Date: Thu, 3 Sep 2026 19:00:19 +0530 Subject: [PATCH 3/4] docs(scheduler): clarify scoring weight validation Signed-off-by: blackdragoon26 --- docs/developers/scheduling.md | 2 +- .../current/developers/scheduling.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/developers/scheduling.md b/docs/developers/scheduling.md index 62982c106..24ec94b29 100644 --- a/docs/developers/scheduling.md +++ b/docs/developers/scheduling.md @@ -164,7 +164,7 @@ score = 10 * ( ) ``` -The annotation must contain the `slot`, `core`, and `memory` keys. Each value must be a non-negative integer, and at least one value must be greater than zero. Key order and surrounding whitespace do not matter. If the annotation is absent, HAMi uses `slot=1,core=1,memory=1`, which preserves the default scoring behavior. An invalid annotation prevents the Pod from being scheduled until the annotation is corrected. +The annotation must contain the `slot`, `core`, and `memory` keys. Each value must be a non-negative integer, and at least one value must be greater than zero. Key order and surrounding whitespace do not matter. If the annotation is absent, HAMi uses `slot=1,core=1,memory=1`, which preserves the default scoring behavior. When the admission webhook is enabled, invalid annotations are rejected when a Pod requesting a HAMi-managed resource is created. The scheduler also validates the annotation so invalid values cannot be used when admission validation is unavailable. For example, consider two candidate GPUs after accounting for a Pod that requests one vGPU and 40% device memory: diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/developers/scheduling.md b/i18n/zh/docusaurus-plugin-content-docs/current/developers/scheduling.md index 0659e2f0b..a00ee2480 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/developers/scheduling.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/developers/scheduling.md @@ -165,7 +165,7 @@ score = 10 * ( ) ``` -注解必须包含 `slot`、`core` 和 `memory` 三个键。每个值都必须是非负整数，并且至少有一个值大于零。键的顺序以及两侧的空格不会影响解析。如果未设置该注解，HAMi 将使用 `slot=1,core=1,memory=1`，从而保持默认评分行为。无效的注解会使 Pod 无法被调度，直到该注解被修正。 +注解必须包含 `slot`、`core` 和 `memory` 三个键。每个值都必须是非负整数，并且至少有一个值大于零。键的顺序以及两侧的空格不会影响解析。如果未设置该注解，HAMi 将使用 `slot=1,core=1,memory=1`，从而保持默认评分行为。启用准入 Webhook 后，如果请求 HAMi 管理资源的 Pod 包含无效注解，则会在创建时被拒绝。调度器也会验证该注解，因此即使准入验证不可用，无效值也无法用于调度。 例如，假设一个 Pod 请求 1 个 vGPU 和 40% 的设备显存，并且在计入该请求后有两个候选 GPU： From 3ba709f16e0137ca21e0391f6ddc60bdc310aa90 Mon Sep 17 00:00:00 2001 From: blackdragoon26 Date: Thu, 3 Sep 2026 21:28:38 +0530 Subject: [PATCH 4/4] docs(scheduler): clarify invalid annotation fallback Signed-off-by: blackdragoon26 --- docs/developers/scheduling.md | 2 +- .../current/developers/scheduling.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/developers/scheduling.md b/docs/developers/scheduling.md index 24ec94b29..cb1e9a6dc 100644 --- a/docs/developers/scheduling.md +++ b/docs/developers/scheduling.md @@ -164,7 +164,7 @@ score = 10 * ( ) ``` -The annotation must contain the `slot`, `core`, and `memory` keys. Each value must be a non-negative integer, and at least one value must be greater than zero. Key order and surrounding whitespace do not matter. If the annotation is absent, HAMi uses `slot=1,core=1,memory=1`, which preserves the default scoring behavior. When the admission webhook is enabled, invalid annotations are rejected when a Pod requesting a HAMi-managed resource is created. The scheduler also validates the annotation so invalid values cannot be used when admission validation is unavailable. +The annotation must contain the `slot`, `core`, and `memory` keys. Each value must be a non-negative integer, and at least one value must be greater than zero. Key order and surrounding whitespace do not matter. If the annotation is absent, HAMi uses `slot=1,core=1,memory=1`, which preserves the default scoring behavior. When the admission webhook is enabled, invalid annotations are rejected when a Pod requesting a HAMi-managed resource is created. If admission validation is unavailable, the scheduler rejects the scheduling attempt and the Pod remains unschedulable until the annotation is corrected. For example, consider two candidate GPUs after accounting for a Pod that requests one vGPU and 40% device memory: diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/developers/scheduling.md b/i18n/zh/docusaurus-plugin-content-docs/current/developers/scheduling.md index a00ee2480..cbbba85d0 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/developers/scheduling.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/developers/scheduling.md @@ -165,7 +165,7 @@ score = 10 * ( ) ``` -注解必须包含 `slot`、`core` 和 `memory` 三个键。每个值都必须是非负整数，并且至少有一个值大于零。键的顺序以及两侧的空格不会影响解析。如果未设置该注解，HAMi 将使用 `slot=1,core=1,memory=1`，从而保持默认评分行为。启用准入 Webhook 后，如果请求 HAMi 管理资源的 Pod 包含无效注解，则会在创建时被拒绝。调度器也会验证该注解，因此即使准入验证不可用，无效值也无法用于调度。 +注解必须包含 `slot`、`core` 和 `memory` 三个键。每个值都必须是非负整数，并且至少有一个值大于零。键的顺序以及两侧的空格不会影响解析。如果未设置该注解，HAMi 将使用 `slot=1,core=1,memory=1`，从而保持默认评分行为。启用准入 Webhook 后，如果请求 HAMi 管理资源的 Pod 包含无效注解，则会在创建时被拒绝。如果准入验证不可用，调度器会拒绝此次调度尝试，Pod 将保持不可调度状态，直到该注解被修正。 例如，假设一个 Pod 请求 1 个 vGPU 和 40% 的设备显存，并且在计入该请求后有两个候选 GPU：