-
Notifications
You must be signed in to change notification settings - Fork 100
docs(device): warn against Kubernetes quantity suffixes for device memory requests #714
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -121,6 +121,12 @@ Each unit of `iluvatar.ai/<card-type>.vMem` represents 256 MB of device memory. | |
|
|
||
| ::: | ||
|
|
||
| :::warning | ||
|
|
||
| Set `iluvatar.ai/<card-type>.vMem` as a **plain integer** (a count of 256 MB units), not a Kubernetes quantity. A suffixed value such as `16Gi` is parsed as its byte count (about 17 billion), which overflows HAMi's 32-bit memory field and is silently truncated, often to `0`. A zero memory request passes the scheduler's memory check unconditionally, so the pod can be placed on a GPU that is already full and then fails with out-of-memory errors at runtime. Use the integer form shown above (for example `64`). | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. hami #2285 is titled reject int32-overflowing memory requests in iluvatar and mthreads, so on iluvatar this now errors instead of truncating. should the iluvatar wording differ from the nvidia, hygon and cambricon ones? |
||
|
|
||
| ::: | ||
|
|
||
| ## Device UUID Selection | ||
|
|
||
| You can specify which GPU devices to use or exclude using annotations: | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -12,6 +12,12 @@ resources: | |
| nvidia.com/gpumem: 3000 # Each GPU contains 3000 MiB device memory | ||
| ``` | ||
|
|
||
| :::warning | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. your own hami pr #2285 makes the plugin reject an overflowing value instead of truncating it. from which hami release does this page describe the old behaviour? |
||
|
|
||
| Set `nvidia.com/gpumem` as a **plain integer** (a count of MiB), not a Kubernetes quantity. A suffixed value such as `16Gi` is parsed as its byte count (about 17 billion), which overflows HAMi's 32-bit memory field and is silently truncated, often to `0`. A zero memory request passes the scheduler's memory check unconditionally, so the pod can be placed on a GPU that is already full and then fails with out-of-memory errors at runtime. Use the integer form shown above (for example `3000`). | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. confirmed the mechanism: with Memreq 0 and MemPercentagereq 101, memreq stays 0 at nvidia/device.go:918 and the check at 927 then passes on any card. the source even carries a |
||
|
|
||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nvidia.com/gpumem-percentage and nvidia.com/gpucores just below take a plain integer too. do they hit the same field, and should the warning cover them? |
||
| ::: | ||
|
|
||
| Allocate a percentage of device memory by specifying resource `nvidia.com/gpumem-percentage`. Optional, each unit of `nvidia.com/gpumem-percentage` equals 1% of device memory. | ||
|
|
||
| ```yaml | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -30,7 +30,7 @@ title: 启用天数智芯 GPU 共享 | |
| - 安装 HAMi 时设置 `devices.iluvatar.enabled=true` | ||
|
|
||
| ```bash | ||
| helm install hami hami-charts/hami --set scheduler.kubeScheduler.imageTag={your kubernetes version} --set devices.iluvatar.enabled=true -n kube-system | ||
| helm install hami hami-charts/hami --set scheduler.kubeScheduler.image.tag={your kubernetes version} --set devices.iluvatar.enabled=true -n kube-system | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. same question as in #708: does the v2.9 chart use image.tag? if it still uses imageTag this breaks the snapshot. |
||
| ``` | ||
|
|
||
| **说明:** 当前支持的 GPU 型号及资源名称定义如下(位于 [https://github.com/Project-HAMi/HAMi/blob/master/charts/hami/templates/scheduler/device-configmap.yaml](https://github.com/Project-HAMi/HAMi/blob/master/charts/hami/templates/scheduler/device-configmap.yaml)): | ||
|
|
@@ -117,6 +117,12 @@ spec: | |
|
|
||
| ::: | ||
|
|
||
| :::warning | ||
|
|
||
| 请将 `iluvatar.ai/<card-type>.vMem` 设置为**纯整数**(以 256MB 为单位计数),不要使用 Kubernetes 数量单位。带单位的值(例如 `16Gi`)会被解析为字节数(约 170 亿),从而超出 HAMi 的 32 位显存字段范围并被静默截断,通常截断为 `0`。显存请求为 0 时会无条件通过调度器的显存检查,导致 Pod 被调度到显存已满的设备上,并在运行时发生显存不足(OOM)。请使用上面示例中的整数形式(例如 `64`)。 | ||
|
|
||
| ::: | ||
|
|
||
| ## 设备 UUID 选择 | ||
|
|
||
| 你可以通过 annotations 指定使用或排除哪些 GPU 设备: | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -10,9 +10,15 @@ translated: true | |
| resources: | ||
| limits: | ||
| nvidia.com/gpu: 1 # 请求 1 个 GPU | ||
| nvidia.com/gpumem: 3000 # 每个 GPU 包含 3000m 设备显存 | ||
| nvidia.com/gpumem: 3000 # 每个 GPU 包含 3000 MiB 设备显存 | ||
| ``` | ||
|
|
||
| :::warning | ||
|
|
||
| 请将 `nvidia.com/gpumem` 设置为**纯整数**(以 MiB 为单位计数),不要使用 Kubernetes 数量单位。带单位的值(例如 `16Gi`)会被解析为字节数(约 170 亿),从而超出 HAMi 的 32 位显存字段范围并被静默截断,通常截断为 `0`。显存请求为 0 时会无条件通过调度器的显存检查,导致 Pod 被调度到显存已满的设备上,并在运行时发生显存不足(OOM)。请使用上面示例中的整数形式(例如 `3000`)。 | ||
|
|
||
| ::: | ||
|
Comment on lines
+13
to
+20
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win Use Line 7 still says that each Proposed correction-每个 `nvidia.com/gpumem` 单位等于 1M。
+每个 `nvidia.com/gpumem` 单位等于 1 MiB。🤖 Prompt for AI Agents |
||
|
|
||
| 通过指定资源 `nvidia.com/gpumem-percentage` 来分配设备显存的百分比。可选项,每个 `nvidia.com/gpumem-percentage` 单位等于设备显存的 1% 百分比。 | ||
|
|
||
| ```yaml | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Apply the warning to plugin-specific Cambricon resource names.
The note at Line 24 says that the resource name can be
cambricon.com/mlu370.smlu.vmemoryor another type. The warning names onlycambricon.com/mlu.smlu.vmemory. Refer to the configured*.smlu.vmemoryresource so users of plugin-specific names receive the same integer-only guidance.docs/userguide/cambricon-device/specify-device-memory-usage.md#L16-L20: Generalize the warning to the configured Cambricon memory resource.i18n/zh/docusaurus-plugin-content-docs/current/userguide/cambricon-device/specify-device-memory-usage.md#L16-L20: Apply the same scope correction in the Chinese translation.📍 Affects 2 files
docs/userguide/cambricon-device/specify-device-memory-usage.md#L16-L20(this comment)i18n/zh/docusaurus-plugin-content-docs/current/userguide/cambricon-device/specify-device-memory-usage.md#L16-L20🤖 Prompt for AI Agents