docs(tutorials): add RTX PRO 6000 Dynamic MIG lab

i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/dynamic-mig-rtx-pro.md

@@ -0,0 +1,43 @@
+---
+title: "实验 17: RTX PRO 6000 动态 MIG 生命周期"
+description: "构建固定版本的 HAMi，并验证按 Pod 创建 MIG、混合规格、选择性回收、重启恢复和跨 GPU 调度。"
+sidebar_label: "实验 17: 动态 MIG 生命周期"
+lab:
+  level: Advanced
+  duration: 约 90 分钟
+  environment: 配备 8 张 NVIDIA RTX PRO 6000 Blackwell GPU 的单节点 Kubernetes 服务器
+  cost: 需要可计费的多 GPU 硬件
+  authors:
+    - shkatara
+    - saiyam1814
+  verified: "2026-08-11"
+tags:
+  - gpu-partitioning
+  - nvidia
+  - hami
+toc_max_heading_level: 2
+---
+
+:::caution[翻译进行中]
+
+本实验的完整中文翻译尚未完成。为避免命令、固定提交版本、安全警告和实测输出在翻译期间产生偏差，请暂时使用[英文版实验](/tutorials/labs/dynamic-mig-rtx-pro)。
+
+英文版包含完整的端到端操作：构建 HAMi 提交 `634bf2b32e68`、备份与受控交接、MIG Manager 所有权警告、`operatingmode` 与 `migStrategy` 的区别、单个 `1g.24gb` 请求、四个位置饱和、`1g.24gb` 与 `2g.48gb` 混合部署、相邻 CUDA 工作负载持续运行时的选择性回收、设备插件重启后的 UUID 稳定性、第五个 Pod 溢出到第二张 GPU，以及清理和运维陷阱。
+
+:::
+
+## 验证环境
+
+| 组件        | 实测值                                           |
+| ----------- | ------------------------------------------------ |
+| GPU         | 8 × NVIDIA RTX PRO 6000 Blackwell Server Edition |
+| NVIDIA 驱动 | `610.43.02`                                      |
+| Kubernetes  | `v1.35.6`                                        |
+| 操作系统    | Ubuntu 24.04.4 LTS                               |
+| HAMi 源码   | `634bf2b32e68e07d3fbcbd6da1ee079392fc07c1`       |
+
+:::danger[只能有一个 MIG 硬件状态管理者]
+
+NVIDIA GPU Operator MIG Manager 与 HAMi Dynamic MIG 都会创建和销毁 GI/CI。两者不得同时管理同一张物理 GPU。迁移现有节点前，请遵循固定版本的 [Dynamic MIG 迁移指南](https://github.com/Project-HAMi/HAMi/blob/634bf2b32e68e07d3fbcbd6da1ee079392fc07c1/docs/develop/dynamic-mig-migration.md)。
+
+:::

i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md

@@ -30,3 +30,4 @@ import LabCardGridAuto from '@site/src/components/labs/LabCardGridAuto';
 - **实验 12** 在 GKE 1.35/COS/CDI 上部署 KAI Scheduler 与 HAMi-core，并通过 CUDA 分配验证显存上限。
 - **实验 13** 在昇腾 310P3 ARM 服务器上源码编译 Volcano 与 ascend-device-plugin，验证 hami-vnpu-core 软切分、binpack 共卡与容器级监控指标。
 - **实验 14** 在挂载四块 T4 的 GKE 节点上安装 HAMi v2.10.0，并通过分配注解与调度器日志观察可组合的 `gpu-scheduler-policy` 策略链（`spread`、`binpack`、`mutex`、`mutex,binpack`）。
+- **实验 17** 在配备 8 张 RTX PRO 6000 的服务器上构建固定提交版本的 HAMi，并验证完整的 Dynamic MIG 生命周期：按 Pod 创建 MIG、混合规格、选择性回收、device plugin 重启恢复以及跨 GPU 调度。

sidebars-tutorials.js

@@ -86,6 +86,11 @@ module.exports = {
           id: "labs/hami-sglang",
           customProps: { level: "Intermediate", duration: "about 45 minutes" },
         },
+        {
+          type: "doc",
+          id: "labs/dynamic-mig-rtx-pro",
+          customProps: { level: "Advanced", duration: "about 90 minutes" },
+        },
       ],
     },
   ],

tutorials/labs/examples/17-dynamic-mig-rtx-pro/create-mixed-pods.sh

@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+: "${NODE:?Set NODE to the Kubernetes node name}"
+: "${GPU_UUID:?Set GPU_UUID to the physical GPU UUID used for the mixed-profile test}"
+
+NAMESPACE=${NAMESPACE:-hami-mig-retest}
+
+create_mig_pod() {
+  local name=$1
+  local memory=$2
+
+  kubectl apply -f - <<EOF
+apiVersion: v1
+kind: Pod
+metadata:
+  name: ${name}
+  namespace: ${NAMESPACE}
+  annotations:
+    nvidia.com/vgpu-mode: "mig"
+    hami.io/gpu-scheduler-policy: "binpack"
+    nvidia.com/use-gpuuuid: "${GPU_UUID}"
+spec:
+  schedulerName: hami-scheduler
+  nodeSelector:
+    kubernetes.io/hostname: ${NODE}
+  containers:
+    - name: cuda
+      image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04
+      imagePullPolicy: IfNotPresent
+      command:
+        - bash
+        - -lc
+        - |
+          set -euo pipefail
+          n=0
+          echo 0 > /tmp/gpu-progress
+          while true; do
+            /cuda-samples/vectorAdd > /tmp/vectoradd.last 2>&1
+            n=\$((n + 1))
+            echo "\$n" > /tmp/gpu-progress.next
+            mv /tmp/gpu-progress.next /tmp/gpu-progress
+          done
+      resources:
+        limits:
+          nvidia.com/gpu: 1
+          nvidia.com/gpumem: ${memory}
+EOF
+}
+
+create_mig_pod mixed-small 8000
+create_mig_pod mixed-large 30000
+
+kubectl wait -n "${NAMESPACE}" --for=condition=Ready \
+  pod/mixed-small pod/mixed-large --timeout=180s

tutorials/labs/examples/17-dynamic-mig-rtx-pro/hami-values.yaml

@@ -0,0 +1,60 @@
+global:
+  imageTag: master-634bf2b32e68
+
+scheduler:
+  defaultSchedulerPolicy:
+    nodeSchedulerPolicy: binpack
+    gpuSchedulerPolicy: binpack
+  extender:
+    image:
+      registry: localhost
+      repository: hami-dynamic-mig
+      tag: master-634bf2b32e68
+      pullPolicy: Never
+
+devicePlugin:
+  image:
+    registry: localhost
+    repository: hami-dynamic-mig
+    tag: master-634bf2b32e68
+    pullPolicy: Never
+  monitor:
+    image:
+      registry: localhost
+      repository: hami-dynamic-mig
+      tag: master-634bf2b32e68
+      pullPolicy: Never
+
+  # NVIDIA's static MIG resource exposure strategy. This is not HAMi's
+  # per-node Dynamic MIG operating mode below.
+  migStrategy: none
+
+  # Steps 3 and 8 of the lab fill in the node name and the excluded GPU index
+  # list below before rendering. The verified run excluded [0, 1, 2, 3, 5, 6, 7],
+  # which registers only GPU 4.
+  nodeConfiguration:
+    config: |
+      {
+        "nodeconfig": [
+          {
+            "name": "__NODE_NAME__",
+            "operatingmode": "mig",
+            "devicememoryscaling": 1,
+            "devicecorescaling": 1,
+            "devicesplitcount": 10,
+            "preconfigureddevicememory": 0,
+            "enablenumatopology": false,
+            "migstrategy": "none",
+            "filterdevices": {
+              "uuid": [],
+              "index": [__EXCLUDED_GPU_INDICES__]
+            },
+            "enablegetpreferredallocation": false
+          }
+        ]
+      }
+
+# Use the device-config.yaml bundled with the pinned source chart. It contains
+# the RTX PRO 6000 profile allowlist tested by this lab.
+device-config:
+  content: ""

tutorials/labs/examples/17-dynamic-mig-rtx-pro/mig-small-pack.yaml

@@ -0,0 +1,42 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: mig-small-pack
+  namespace: hami-mig-retest
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: mig-small-pack
+  template:
+    metadata:
+      labels:
+        app: mig-small-pack
+      annotations:
+        nvidia.com/vgpu-mode: "mig"
+        hami.io/gpu-scheduler-policy: "binpack"
+    spec:
+      schedulerName: hami-scheduler
+      nodeSelector:
+        kubernetes.io/hostname: __NODE_NAME__
+      containers:
+        - name: cuda
+          image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04
+          imagePullPolicy: IfNotPresent
+          command:
+            - bash
+            - -lc
+            - |
+              set -euo pipefail
+              n=0
+              echo 0 > /tmp/gpu-progress
+              while true; do
+                /cuda-samples/vectorAdd > /tmp/vectoradd.last 2>&1
+                n=$((n + 1))
+                echo "$n" > /tmp/gpu-progress.next
+                mv /tmp/gpu-progress.next /tmp/gpu-progress
+              done
+          resources:
+            limits:
+              nvidia.com/gpu: 1
+              nvidia.com/gpumem: 8000

tutorials/overview.md

@@ -30,3 +30,4 @@ Each lab lists its own prerequisites.
 - **Lab 12** deploys KAI Scheduler and HAMi-core on GKE 1.35/COS/CDI and proves the memory ceiling with CUDA allocations.
 - **Lab 13** builds Volcano and the ascend-device-plugin from source on an Ascend 310P3 ARM server and verifies hami-vnpu-core soft slicing, binpack card sharing, and per-container metrics.
 - **Lab 14** installs HAMi v2.10.0 on a four-T4 GKE node and observes the composable `gpu-scheduler-policy` chains (`spread`, `binpack`, `mutex`, `mutex,binpack`) through allocation annotations and scheduler logs.
+- **Lab 17** builds a pinned HAMi snapshot on an eight-GPU RTX PRO 6000 server and verifies the complete Dynamic MIG lifecycle: per-Pod placement, mixed profiles, selective reclamation, device-plugin restart recovery, and spillover to a second GPU.