From 7a3eac727fdb2cfd35a094530d97d8e0bc6e4e0b Mon Sep 17 00:00:00 2001 From: Prabal Poddar Date: Fri, 21 Aug 2026 11:26:26 +0530 Subject: [PATCH 1/2] docs(developers): add missing task dispatch section and diagram to protocol.md Signed-off-by: Prabal Poddar --- docs/developers/protocol.md | 33 +++++++++++++++++++ .../version-v2.9.0/developers/protocol.md | 33 +++++++++++++++++++ 2 files changed, 66 insertions(+) diff --git a/docs/developers/protocol.md b/docs/developers/protocol.md index beb91d090..7ff81725c 100644 --- a/docs/developers/protocol.md +++ b/docs/developers/protocol.md @@ -38,3 +38,36 @@ Since system clock on scheduler node and 'device' node may not align properly, s ```text hami.io/node-handshake-{device-type}: Requesting_{scheduler_node_current_timestamp} ``` + +## Task Dispatch & Scheduling Decisions + +During the `bind` process, `kube-scheduler` binds the Pod to the target node. During container creation, `kubelet` invokes the device plugin's `Allocate` method to mount the device, but only provides the device `UUID`. In GPU sharing scenarios, the device plugin cannot natively obtain the workload's requested device specifications, such as GPU memory and compute core limits. + +Therefore, HAMi uses a protocol for the scheduler to communicate task allocation metadata to the device plugin. The scheduler passes this information by patching allocation annotations onto the Pod, which the device plugin reads during container setup, as shown below: + +HAMi task dispatch protocol diagram showing scheduler and device-plugin interaction + +During this process, the following annotations are managed on the Pod: + +- `hami.io/bind-phase`: Tracks allocation progress. The scheduler sets this to `allocating` during bind, and the device plugin updates it to `success` (or `failed`) once allocation finishes. This serves as the completion signal. +- `hami.io/bind-time`: Timestamp when the scheduler initiated the binding process. +- `hami.io/vgpu-node`: The target node assigned by the scheduler, used by the device plugin to identify the pending Pod on the node. +- `hami.io/vgpu-devices-allocated`: The devices and specifications allocated by the scheduler. +- `hami.io/vgpu-devices-to-allocate`: The devices pending allocation. When the scheduler prepares the Pod for binding, this annotation contains the target devices. During container setup, the device plugin allocates devices and incrementally removes them. Once all devices are allocated, this annotation becomes empty. + +For example, when a GPU task requesting 3000 MiB of device memory is dispatched, the scheduler sets the annotations to: + +```yaml +hami.io/bind-phase: "allocating" +hami.io/bind-time: "1716199325" +hami.io/vgpu-node: "node-1" +hami.io/vgpu-devices-allocated: GPU-0fc3eda5-e98b-a25b-5b0d-cf5c855d1448,NVIDIA,3000,0:; +hami.io/vgpu-devices-to-allocate: GPU-0fc3eda5-e98b-a25b-5b0d-cf5c855d1448,NVIDIA,3000,0:; +``` + +Once the device plugin completes allocation, `hami.io/bind-phase` transitions to `success` and `hami.io/vgpu-devices-to-allocate` is cleared: + +```yaml +hami.io/bind-phase: "success" +hami.io/vgpu-devices-to-allocate: ; +``` diff --git a/versioned_docs/version-v2.9.0/developers/protocol.md b/versioned_docs/version-v2.9.0/developers/protocol.md index beb91d090..7ff81725c 100644 --- a/versioned_docs/version-v2.9.0/developers/protocol.md +++ b/versioned_docs/version-v2.9.0/developers/protocol.md @@ -38,3 +38,36 @@ Since system clock on scheduler node and 'device' node may not align properly, s ```text hami.io/node-handshake-{device-type}: Requesting_{scheduler_node_current_timestamp} ``` + +## Task Dispatch & Scheduling Decisions + +During the `bind` process, `kube-scheduler` binds the Pod to the target node. During container creation, `kubelet` invokes the device plugin's `Allocate` method to mount the device, but only provides the device `UUID`. In GPU sharing scenarios, the device plugin cannot natively obtain the workload's requested device specifications, such as GPU memory and compute core limits. + +Therefore, HAMi uses a protocol for the scheduler to communicate task allocation metadata to the device plugin. The scheduler passes this information by patching allocation annotations onto the Pod, which the device plugin reads during container setup, as shown below: + +HAMi task dispatch protocol diagram showing scheduler and device-plugin interaction + +During this process, the following annotations are managed on the Pod: + +- `hami.io/bind-phase`: Tracks allocation progress. The scheduler sets this to `allocating` during bind, and the device plugin updates it to `success` (or `failed`) once allocation finishes. This serves as the completion signal. +- `hami.io/bind-time`: Timestamp when the scheduler initiated the binding process. +- `hami.io/vgpu-node`: The target node assigned by the scheduler, used by the device plugin to identify the pending Pod on the node. +- `hami.io/vgpu-devices-allocated`: The devices and specifications allocated by the scheduler. +- `hami.io/vgpu-devices-to-allocate`: The devices pending allocation. When the scheduler prepares the Pod for binding, this annotation contains the target devices. During container setup, the device plugin allocates devices and incrementally removes them. Once all devices are allocated, this annotation becomes empty. + +For example, when a GPU task requesting 3000 MiB of device memory is dispatched, the scheduler sets the annotations to: + +```yaml +hami.io/bind-phase: "allocating" +hami.io/bind-time: "1716199325" +hami.io/vgpu-node: "node-1" +hami.io/vgpu-devices-allocated: GPU-0fc3eda5-e98b-a25b-5b0d-cf5c855d1448,NVIDIA,3000,0:; +hami.io/vgpu-devices-to-allocate: GPU-0fc3eda5-e98b-a25b-5b0d-cf5c855d1448,NVIDIA,3000,0:; +``` + +Once the device plugin completes allocation, `hami.io/bind-phase` transitions to `success` and `hami.io/vgpu-devices-to-allocate` is cleared: + +```yaml +hami.io/bind-phase: "success" +hami.io/vgpu-devices-to-allocate: ; +``` From c1b5c8f8abc9157dd3d0a430c4ae4da84392caea Mon Sep 17 00:00:00 2001 From: Prabal Poddar Date: Wed, 9 Sep 2026 21:51:57 +0530 Subject: [PATCH 2/2] docs(inventory): remove obsolete references to pending #811 Signed-off-by: Prabal Poddar --- docs/diagrams-inventory.md | 10 +++++----- .../current/diagrams-inventory.md | 10 +++++----- .../version-v2.9.0/diagrams-inventory.md | 10 +++++----- versioned_docs/version-v2.9.0/diagrams-inventory.md | 10 +++++----- 4 files changed, 20 insertions(+), 20 deletions(-) diff --git a/docs/diagrams-inventory.md b/docs/diagrams-inventory.md index 0cc92fd2b..eec29d2f4 100644 --- a/docs/diagrams-inventory.md +++ b/docs/diagrams-inventory.md @@ -71,10 +71,10 @@ This page catalogs every diagram across the repository: `docs/`, `blog/`, `i18n/ | Image path | Format | Referenced in | Source file | Status | Control/data plane | | --- | --- | --- | --- | --- | --- | | `static/img/docs/common/developers/protocol/protocol-register.png` | PNG | `docs/developers/protocol.md` (EN), `i18n/zh/.../developers/protocol.md` (ZH) | none | unknown | not separated | -| `static/img/docs/common/developers/protocol/task-dispatch.png` | PNG | historical v1.3.0 and v2.4.1–v2.5.1 snapshots; current/v2.9.0 ZH; EN pending [#811](https://github.com/Project-HAMi/website/pull/811) | none | unknown | not separated | +| `static/img/docs/common/developers/protocol/task-dispatch.png` | PNG | historical v1.3.0 and v2.4.1–v2.5.1 snapshots; `docs/developers/protocol.md` (EN), `i18n/zh/.../developers/protocol.md` (ZH) | none | unknown | not separated | | `static/img/docs/common/developers/protocol/device-registration.png` | PNG | historical v1.3.0 and v2.4.1–v2.5.1 protocol snapshots | none | unknown | not separated | -**Note:** Current/v2.9.0 English and Chinese protocol pages use `protocol-register.png`; historical snapshots retain the older `device-registration.png` and `task-dispatch.png` references. Current/v2.9.0 Chinese pages use `task-dispatch.png`, and [#811](https://github.com/Project-HAMi/website/pull/811) proposes adding it to English. +**Note:** Current/v2.9.0 English and Chinese protocol pages use `protocol-register.png` and `task-dispatch.png`; historical snapshots retain the older `device-registration.png` and legacy `task-dispatch.png` references. --- @@ -241,9 +241,9 @@ The directory `static/img/docs/common/architecture/` does not exist. The file at This is a broken link that will cause a missing image in the rendered page. -### Protocol diagram inconsistency (EN vs. ZH) — partially resolved +### Protocol diagram inconsistency (EN vs. ZH) — resolved -Current/v2.9.0 English and Chinese protocol pages use `protocol-register.png`; historical snapshots retain the older `device-registration.png` and `task-dispatch.png` references. Current/v2.9.0 Chinese pages use `task-dispatch.png`, and [#811](https://github.com/Project-HAMi/website/pull/811) proposes adding it to English. +Current/v2.9.0 English and Chinese protocol pages use `protocol-register.png` and `task-dispatch.png`; historical snapshots retain the older `device-registration.png` and legacy `task-dispatch.png` references. --- @@ -262,7 +262,7 @@ Priority list for the redraw work in [#421](https://github.com/Project-HAMi/webs | Medium | `static/img/docs/common/developers/scheduling/node-scheduler-policy-demo.png` | PNG | No source | | Medium | `static/img/docs/common/developers/scheduling/gpu-scheduler-policy-demo.png` | PNG | No source | | Medium | `static/img/docs/common/developers/protocol/protocol-register.png` | PNG | No source; shared by EN and ZH | -| Medium | `static/img/docs/common/developers/protocol/task-dispatch.png` | PNG | No source; historical EN/ZH and current ZH, current EN pending [#811](https://github.com/Project-HAMi/website/pull/811) | +| Medium | `static/img/docs/common/developers/protocol/task-dispatch.png` | PNG | No source; historical EN/ZH, shared by current EN and ZH | | Medium | `static/img/docs/common/developers/protocol/device-registration.png` | PNG | No source; retained by historical protocol snapshots | | Medium | `static/img/docs/en/dynamic-mig/hami-dynamic-mig-structure.png` | PNG | No source | | Medium | `static/img/docs/en/dynamic-mig/hami-dynamic-mig-procedure.png` | PNG | No source | diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/diagrams-inventory.md b/i18n/zh/docusaurus-plugin-content-docs/current/diagrams-inventory.md index aefcba54a..9c551ff59 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/diagrams-inventory.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/diagrams-inventory.md @@ -72,10 +72,10 @@ translated: true | 图片路径 | Format | 引用位置 | Source file | Status | Control/data plane | | --- | --- | --- | --- | --- | --- | | `static/img/docs/common/developers/protocol/protocol-register.png` | PNG | `docs/developers/protocol.md`（英文）、`i18n/zh/.../developers/protocol.md`（中文） | none | unknown | not separated | -| `static/img/docs/common/developers/protocol/task-dispatch.png` | PNG | 历史 v1.3.0 和 v2.4.1–v2.5.1 快照；当前/v2.9.0 中文版；英文版待 [#811](https://github.com/Project-HAMi/website/pull/811) 合入 | none | unknown | not separated | +| `static/img/docs/common/developers/protocol/task-dispatch.png` | PNG | 历史 v1.3.0 和 v2.4.1–v2.5.1 快照；`docs/developers/protocol.md`（英文）、`i18n/zh/.../developers/protocol.md`（中文） | none | unknown | not separated | | `static/img/docs/common/developers/protocol/device-registration.png` | PNG | 历史 v1.3.0 和 v2.4.1–v2.5.1 协议快照 | none | unknown | not separated | -**说明：** 当前/v2.9.0 英文和中文协议页面使用 `protocol-register.png`；历史快照仍保留 `device-registration.png` 和 `task-dispatch.png` 的引用。当前/v2.9.0 中文页面使用 `task-dispatch.png`，[#811](https://github.com/Project-HAMi/website/pull/811) 提议将其添加到英文页面。 +**说明：** 当前/v2.9.0 英文和中文协议页面使用 `protocol-register.png` 与 `task-dispatch.png`；历史快照仍保留 `device-registration.png` 及旧版 `task-dispatch.png` 的引用。 --- @@ -242,9 +242,9 @@ translated: true 这是一个损坏的链接，会导致渲染页面中图片缺失。 -### 协议图不一致（英文 vs. 中文）— 已部分解决 +### 协议图不一致（英文 vs. 中文）— 已解决 -当前/v2.9.0 英文和中文协议页面使用 `protocol-register.png`；历史快照仍保留 `device-registration.png` 和 `task-dispatch.png` 的引用。当前/v2.9.0 中文页面使用 `task-dispatch.png`，[#811](https://github.com/Project-HAMi/website/pull/811) 提议将其添加到英文页面。 +当前/v2.9.0 英文和中文协议页面使用 `protocol-register.png` 与 `task-dispatch.png`；历史快照仍保留 `device-registration.png` 及旧版 `task-dispatch.png` 的引用。 --- @@ -263,7 +263,7 @@ translated: true | Medium | `static/img/docs/common/developers/scheduling/node-scheduler-policy-demo.png` | PNG | 无源文件 | | Medium | `static/img/docs/common/developers/scheduling/gpu-scheduler-policy-demo.png` | PNG | 无源文件 | | Medium | `static/img/docs/common/developers/protocol/protocol-register.png` | PNG | 无源文件；英文和中文共用 | -| Medium | `static/img/docs/common/developers/protocol/task-dispatch.png` | PNG | 无源文件；历史中英文版及当前中文版引用，当前英文版待 [#811](https://github.com/Project-HAMi/website/pull/811) 合入 | +| Medium | `static/img/docs/common/developers/protocol/task-dispatch.png` | PNG | 无源文件；历史中英文版引用，当前英文和中文共用 | | Medium | `static/img/docs/common/developers/protocol/device-registration.png` | PNG | 无源文件；由历史协议快照保留 | | Medium | `static/img/docs/en/dynamic-mig/hami-dynamic-mig-structure.png` | PNG | 无源文件 | | Medium | `static/img/docs/en/dynamic-mig/hami-dynamic-mig-procedure.png` | PNG | 无源文件 | diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/diagrams-inventory.md b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/diagrams-inventory.md index aefcba54a..9c551ff59 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/diagrams-inventory.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/diagrams-inventory.md @@ -72,10 +72,10 @@ translated: true | 图片路径 | Format | 引用位置 | Source file | Status | Control/data plane | | --- | --- | --- | --- | --- | --- | | `static/img/docs/common/developers/protocol/protocol-register.png` | PNG | `docs/developers/protocol.md`（英文）、`i18n/zh/.../developers/protocol.md`（中文） | none | unknown | not separated | -| `static/img/docs/common/developers/protocol/task-dispatch.png` | PNG | 历史 v1.3.0 和 v2.4.1–v2.5.1 快照；当前/v2.9.0 中文版；英文版待 [#811](https://github.com/Project-HAMi/website/pull/811) 合入 | none | unknown | not separated | +| `static/img/docs/common/developers/protocol/task-dispatch.png` | PNG | 历史 v1.3.0 和 v2.4.1–v2.5.1 快照；`docs/developers/protocol.md`（英文）、`i18n/zh/.../developers/protocol.md`（中文） | none | unknown | not separated | | `static/img/docs/common/developers/protocol/device-registration.png` | PNG | 历史 v1.3.0 和 v2.4.1–v2.5.1 协议快照 | none | unknown | not separated | -**说明：** 当前/v2.9.0 英文和中文协议页面使用 `protocol-register.png`；历史快照仍保留 `device-registration.png` 和 `task-dispatch.png` 的引用。当前/v2.9.0 中文页面使用 `task-dispatch.png`，[#811](https://github.com/Project-HAMi/website/pull/811) 提议将其添加到英文页面。 +**说明：** 当前/v2.9.0 英文和中文协议页面使用 `protocol-register.png` 与 `task-dispatch.png`；历史快照仍保留 `device-registration.png` 及旧版 `task-dispatch.png` 的引用。 --- @@ -242,9 +242,9 @@ translated: true 这是一个损坏的链接，会导致渲染页面中图片缺失。 -### 协议图不一致（英文 vs. 中文）— 已部分解决 +### 协议图不一致（英文 vs. 中文）— 已解决 -当前/v2.9.0 英文和中文协议页面使用 `protocol-register.png`；历史快照仍保留 `device-registration.png` 和 `task-dispatch.png` 的引用。当前/v2.9.0 中文页面使用 `task-dispatch.png`，[#811](https://github.com/Project-HAMi/website/pull/811) 提议将其添加到英文页面。 +当前/v2.9.0 英文和中文协议页面使用 `protocol-register.png` 与 `task-dispatch.png`；历史快照仍保留 `device-registration.png` 及旧版 `task-dispatch.png` 的引用。 --- @@ -263,7 +263,7 @@ translated: true | Medium | `static/img/docs/common/developers/scheduling/node-scheduler-policy-demo.png` | PNG | 无源文件 | | Medium | `static/img/docs/common/developers/scheduling/gpu-scheduler-policy-demo.png` | PNG | 无源文件 | | Medium | `static/img/docs/common/developers/protocol/protocol-register.png` | PNG | 无源文件；英文和中文共用 | -| Medium | `static/img/docs/common/developers/protocol/task-dispatch.png` | PNG | 无源文件；历史中英文版及当前中文版引用，当前英文版待 [#811](https://github.com/Project-HAMi/website/pull/811) 合入 | +| Medium | `static/img/docs/common/developers/protocol/task-dispatch.png` | PNG | 无源文件；历史中英文版引用，当前英文和中文共用 | | Medium | `static/img/docs/common/developers/protocol/device-registration.png` | PNG | 无源文件；由历史协议快照保留 | | Medium | `static/img/docs/en/dynamic-mig/hami-dynamic-mig-structure.png` | PNG | 无源文件 | | Medium | `static/img/docs/en/dynamic-mig/hami-dynamic-mig-procedure.png` | PNG | 无源文件 | diff --git a/versioned_docs/version-v2.9.0/diagrams-inventory.md b/versioned_docs/version-v2.9.0/diagrams-inventory.md index 0cc92fd2b..eec29d2f4 100644 --- a/versioned_docs/version-v2.9.0/diagrams-inventory.md +++ b/versioned_docs/version-v2.9.0/diagrams-inventory.md @@ -71,10 +71,10 @@ This page catalogs every diagram across the repository: `docs/`, `blog/`, `i18n/ | Image path | Format | Referenced in | Source file | Status | Control/data plane | | --- | --- | --- | --- | --- | --- | | `static/img/docs/common/developers/protocol/protocol-register.png` | PNG | `docs/developers/protocol.md` (EN), `i18n/zh/.../developers/protocol.md` (ZH) | none | unknown | not separated | -| `static/img/docs/common/developers/protocol/task-dispatch.png` | PNG | historical v1.3.0 and v2.4.1–v2.5.1 snapshots; current/v2.9.0 ZH; EN pending [#811](https://github.com/Project-HAMi/website/pull/811) | none | unknown | not separated | +| `static/img/docs/common/developers/protocol/task-dispatch.png` | PNG | historical v1.3.0 and v2.4.1–v2.5.1 snapshots; `docs/developers/protocol.md` (EN), `i18n/zh/.../developers/protocol.md` (ZH) | none | unknown | not separated | | `static/img/docs/common/developers/protocol/device-registration.png` | PNG | historical v1.3.0 and v2.4.1–v2.5.1 protocol snapshots | none | unknown | not separated | -**Note:** Current/v2.9.0 English and Chinese protocol pages use `protocol-register.png`; historical snapshots retain the older `device-registration.png` and `task-dispatch.png` references. Current/v2.9.0 Chinese pages use `task-dispatch.png`, and [#811](https://github.com/Project-HAMi/website/pull/811) proposes adding it to English. +**Note:** Current/v2.9.0 English and Chinese protocol pages use `protocol-register.png` and `task-dispatch.png`; historical snapshots retain the older `device-registration.png` and legacy `task-dispatch.png` references. --- @@ -241,9 +241,9 @@ The directory `static/img/docs/common/architecture/` does not exist. The file at This is a broken link that will cause a missing image in the rendered page. -### Protocol diagram inconsistency (EN vs. ZH) — partially resolved +### Protocol diagram inconsistency (EN vs. ZH) — resolved -Current/v2.9.0 English and Chinese protocol pages use `protocol-register.png`; historical snapshots retain the older `device-registration.png` and `task-dispatch.png` references. Current/v2.9.0 Chinese pages use `task-dispatch.png`, and [#811](https://github.com/Project-HAMi/website/pull/811) proposes adding it to English. +Current/v2.9.0 English and Chinese protocol pages use `protocol-register.png` and `task-dispatch.png`; historical snapshots retain the older `device-registration.png` and legacy `task-dispatch.png` references. --- @@ -262,7 +262,7 @@ Priority list for the redraw work in [#421](https://github.com/Project-HAMi/webs | Medium | `static/img/docs/common/developers/scheduling/node-scheduler-policy-demo.png` | PNG | No source | | Medium | `static/img/docs/common/developers/scheduling/gpu-scheduler-policy-demo.png` | PNG | No source | | Medium | `static/img/docs/common/developers/protocol/protocol-register.png` | PNG | No source; shared by EN and ZH | -| Medium | `static/img/docs/common/developers/protocol/task-dispatch.png` | PNG | No source; historical EN/ZH and current ZH, current EN pending [#811](https://github.com/Project-HAMi/website/pull/811) | +| Medium | `static/img/docs/common/developers/protocol/task-dispatch.png` | PNG | No source; historical EN/ZH, shared by current EN and ZH | | Medium | `static/img/docs/common/developers/protocol/device-registration.png` | PNG | No source; retained by historical protocol snapshots | | Medium | `static/img/docs/en/dynamic-mig/hami-dynamic-mig-structure.png` | PNG | No source | | Medium | `static/img/docs/en/dynamic-mig/hami-dynamic-mig-procedure.png` | PNG | No source |