From d0d10ecf03655ed6f48a209a7c69800656598511 Mon Sep 17 00:00:00 2001 From: Saiyam Pathak Date: Thu, 27 Aug 2026 20:20:15 +0530 Subject: [PATCH 1/6] docs(tutorials): add RTX PRO 6000 Dynamic MIG lab Adds Lab 15, which builds a pinned HAMi snapshot and walks the full Dynamic MIG lifecycle on RTX PRO 6000 hardware: per-Pod MIG placement, mixed profiles, selective reclamation, restart recovery, and multi-GPU spillover. English and Chinese, with the example manifests. Signed-off-by: Saiyam Pathak --- .../current/labs/dynamic-mig-rtx-pro.md | 43 + .../current/overview.md | 2 +- sidebars-tutorials.js | 5 + tutorials/labs/dynamic-mig-rtx-pro.md | 747 ++++++++++++++++++ .../create-mixed-pods.sh | 55 ++ .../15-dynamic-mig-rtx-pro/hami-values.yaml | 57 ++ .../mig-small-pack.yaml | 42 + tutorials/overview.md | 2 +- 8 files changed, 951 insertions(+), 2 deletions(-) create mode 100644 i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/dynamic-mig-rtx-pro.md create mode 100644 tutorials/labs/dynamic-mig-rtx-pro.md create mode 100755 tutorials/labs/examples/15-dynamic-mig-rtx-pro/create-mixed-pods.sh create mode 100644 tutorials/labs/examples/15-dynamic-mig-rtx-pro/hami-values.yaml create mode 100644 tutorials/labs/examples/15-dynamic-mig-rtx-pro/mig-small-pack.yaml diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/dynamic-mig-rtx-pro.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/dynamic-mig-rtx-pro.md new file mode 100644 index 000000000..5453caf75 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/dynamic-mig-rtx-pro.md @@ -0,0 +1,43 @@ +--- +title: "实验 15: RTX PRO 6000 动态 MIG 生命周期" +description: "构建固定版本的 HAMi，并验证按 Pod 创建 MIG、混合规格、选择性回收、重启恢复和跨 GPU 调度。" +sidebar_label: "实验 15: 动态 MIG 生命周期" +lab: + level: Advanced + duration: 约 90 分钟 + environment: 配备 8 张 NVIDIA RTX PRO 6000 Blackwell GPU 的单节点 Kubernetes 服务器 + cost: 需要可计费的多 GPU 硬件 + authors: + - shkatara + - saiyam1814 + verified: "2026-08-11" +tags: + - GPU 分区 + - nvidia + - hami +toc_max_heading_level: 2 +--- + +:::caution[翻译进行中] + +本实验的完整中文翻译尚未完成。为避免命令、固定提交版本、安全警告和实测输出在翻译期间产生偏差，请暂时使用[英文版实验](/tutorials/labs/dynamic-mig-rtx-pro)。 + +英文版包含完整的端到端操作：构建 HAMi 提交 `634bf2b32e68`、备份与受控交接、MIG Manager 所有权警告、`operatingmode` 与 `migStrategy` 的区别、单个 `1g.24gb` 请求、四个位置饱和、`1g.24gb` 与 `2g.48gb` 混合部署、相邻 CUDA 工作负载持续运行时的选择性回收、设备插件重启后的 UUID 稳定性、第五个 Pod 溢出到第二张 GPU，以及清理和运维陷阱。 + +::: + +## 验证环境 + +| 组件 | 实测值 | +| ----------- | ------------------------------------------------ | +| GPU | 8 × NVIDIA RTX PRO 6000 Blackwell Server Edition | +| NVIDIA 驱动 | `610.43.02` | +| Kubernetes | `v1.35.6` | +| 操作系统 | Ubuntu 24.04.4 LTS | +| HAMi 源码 | `634bf2b32e68e07d3fbcbd6da1ee079392fc07c1` | + +:::danger[只能有一个 MIG 硬件状态管理者] + +NVIDIA GPU Operator MIG Manager 与 HAMi Dynamic MIG 都会创建和销毁 GI/CI。两者不得同时管理同一张物理 GPU。迁移现有节点前，请遵循固定版本的 [Dynamic MIG 迁移指南](https://github.com/Project-HAMi/HAMi/blob/634bf2b32e68e07d3fbcbd6da1ee079392fc07c1/docs/develop/dynamic-mig-migration.md)。 + +::: diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md index 8b49dc7a0..5d0fbfacf 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md @@ -17,7 +17,7 @@ import LabCardGridAuto from '@site/src/components/labs/LabCardGridAuto'; ## 实验 - +实验 15 构建固定提交版本的 HAMi，并在 RTX PRO 6000 硬件上验证完整的 Dynamic MIG 生命周期。 每个实验都列出了各自的前提条件。 diff --git a/sidebars-tutorials.js b/sidebars-tutorials.js index 2d4b01385..d7953c00c 100644 --- a/sidebars-tutorials.js +++ b/sidebars-tutorials.js @@ -86,6 +86,11 @@ module.exports = { id: "labs/hami-sglang", customProps: { level: "Intermediate", duration: "about 45 minutes" }, }, + { + type: "doc", + id: "labs/dynamic-mig-rtx-pro", + customProps: { level: "Advanced", duration: "about 90 minutes" }, + }, ], }, ], diff --git a/tutorials/labs/dynamic-mig-rtx-pro.md b/tutorials/labs/dynamic-mig-rtx-pro.md new file mode 100644 index 000000000..ba44da86d --- /dev/null +++ b/tutorials/labs/dynamic-mig-rtx-pro.md @@ -0,0 +1,747 @@ +--- +title: "Lab 15: Dynamic MIG Lifecycle on RTX PRO 6000" +description: "Build a pinned HAMi snapshot and verify per-Pod MIG placement, mixed profiles, selective reclamation, restart recovery, and multi-GPU spillover." +sidebar_label: "Lab 15: Dynamic MIG Lifecycle" +lab: + level: Advanced + duration: about 90 minutes + environment: single-node Kubernetes server with 8 NVIDIA RTX PRO 6000 Blackwell GPUs + cost: requires access to billable multi-GPU hardware + authors: + - shkatara + - saiyam1814 + verified: "2026-08-11" +tags: + - gpu-partitioning + - nvidia + - hami +toc_max_heading_level: 2 +--- + +This lab builds HAMi from the source snapshot that first contained the merged per-Pod Dynamic MIG implementation, then follows one MIG allocation through creation, saturation, mixed-profile placement, selective reclamation, device-plugin adoption, and spillover to a second GPU. A Pod asks for memory through HAMi's usual resource API; HAMi chooses the smallest allowed NVIDIA MIG profile with enough memory and a legal free placement, then creates and later reclaims that Pod's GPU Instance (GI) and Compute Instance (CI). + +The commands and outputs were captured on 2026-08-11 from the [original verified test](https://blog.kubesimplify.com/dynamic-mig-in-kubernetes-with-hami). This tutorial removes the surrounding narrative and retains the reproducible procedure and evidence. + +:::caution[Experimental source snapshot] + +[HAMi PR #2378](https://github.com/Project-HAMi/HAMi/pull/2378) was merged at the tested commit, but HAMi v2.9.0 predates this implementation. This lab therefore builds commit [`634bf2b32e68`](https://github.com/Project-HAMi/HAMi/commit/634bf2b32e68e07d3fbcbd6da1ee079392fc07c1). When an official HAMi release includes PR #2378, prefer that release's matching chart and images. + +::: + +## What You'll Learn + +- Pin the HAMi chart and all three HAMi runtime containers to one source commit. +- Distinguish HAMi's per-node `operatingmode: "mig"` from NVIDIA's static `migStrategy`. +- Verify one 8,000 MiB request, four-placement saturation, and mixed `1g.24gb` plus `2g.48gb` placement. +- Prove that deleting one Pod reclaims only its GI/CI while a neighboring CUDA loop progresses. +- Prove that a complete live allocation survives a device-plugin restart with the same MIG UUID. +- Expose a second GPU and verify that a fifth small Pod spills over instead of overcommitting the first GPU. + +## Lab Overview + +```mermaid +%% title: Dynamic MIG Lifecycle Lab Flow +flowchart LR + S1["Step 1
Inventory and handover"] --> S2["Step 2
Build pinned source"] + S2 --> S3["Step 3
Render and install"] + S3 --> S4["Step 4
Create one 1g"] + S4 --> S5["Step 5
Saturate four placements"] + S5 --> S6["Step 6
Mix and reclaim"] + S6 --> S7["Step 7
Restart and adopt"] + S7 --> S8["Step 8
Spill to GPU 5"] +``` + +## Prerequisites + +The verified environment was: + +| Component | Tested value | +| ----------------- | ------------------------------------------------ | +| GPUs | 8 × NVIDIA RTX PRO 6000 Blackwell Server Edition | +| GPU memory | 97,887 MiB per physical GPU | +| NVIDIA driver | `610.43.02` | +| Kubernetes | `v1.35.6` | +| Operating system | Ubuntu 24.04.4 LTS, kernel `6.8.0-100-generic` | +| Container runtime | containerd `2.2.1` | +| HAMi source | `634bf2b32e68e07d3fbcbd6da1ee079392fc07c1` | + +You also need: + +- root access to the GPU node, working `nvidia-smi`, MIG mode enabled, and no unmanaged CUDA processes; +- `git`, GNU Make, Docker, `ctr`, Helm, `kubectl`, and `jq`; +- cluster-admin access and permission to replace the existing HAMi installation; +- a local checkout of this website repository for the files under [`tutorials/labs/examples/15-dynamic-mig-rtx-pro/`](https://github.com/Project-HAMi/website/tree/master/tutorials/labs/examples/15-dynamic-mig-rtx-pro); and +- an explicit maintenance window for the **whole GPU node**, not only the GPUs that HAMi will register. + +The supplied values target the verified eight-GPU node and initially register only GPU index 4. Adapt both `filterdevices.index` lists before starting if your topology differs. You need at least two compatible GPUs to reproduce Step 8. + +:::danger[Assign one MIG hardware owner] + +NVIDIA GPU Operator MIG Manager and HAMi Dynamic MIG both create and destroy GI/CI state. They **must not control the same physical GPU at the same time**. GPU Operator may continue providing the driver, Container Toolkit, and monitoring, but stop MIG Manager reconciliation on the target node before this handover. Deleting one MIG Manager Pod is insufficient if its controller recreates it. HAMi must also be the only device plugin registering the parent `nvidia.com/gpu` resource on the target node. + +Existing MIG Manager or legacy `knownMigGeometries` users must follow the pinned [Dynamic MIG migration guide](https://github.com/Project-HAMi/HAMi/blob/634bf2b32e68e07d3fbcbd6da1ee079392fc07c1/docs/develop/dynamic-mig-migration.md): inventory, cordon, drain legacy GPU Pods, transfer mutation ownership, then validate one node at a time. + +::: + +Host-level `nvidia-smi`, Docker, and `ctr` commands run on the GPU node. `kubectl` and Helm may run anywhere with the intended kubeconfig; the verified single-node run executed everything on that node. + +## Step 1: Back Up and Establish an Idle Handover + +Select the single Kubernetes node and set a durable working directory. If your cluster has other nodes, set `NODE` explicitly to the eight-GPU node instead. + +```bash +export NODE=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}') +export LAB=/root/hami-dynamic-mig-rerun-2026-08-11 +export EXAMPLES=tutorials/labs/examples/15-dynamic-mig-rtx-pro + +mkdir -p "$LAB" +``` + +If a release named `hami` already exists in `hami-system`, save both Helm's stored state and the live objects; they can differ. + +```bash +if helm status hami -n hami-system >/dev/null 2>&1; then + helm get values hami -n hami-system --all -o yaml \ + > "$LAB/helm-values-before.yaml" + helm get manifest hami -n hami-system \ + > "$LAB/helm-manifest-before.yaml" + kubectl get configmaps -n hami-system -o yaml \ + > "$LAB/live-configmaps-before.yaml" +fi +kubectl get node "$NODE" -o yaml > "$LAB/node-before.yaml" +kubectl get pods -A --field-selector spec.nodeName="$NODE" -o wide +nvidia-smi -L > "$LAB/nvidia-smi-L-before.txt" +``` + +Inventory MIG mode and active compute processes: + +```bash +nvidia-smi \ + --query-gpu=index,name,uuid,driver_version,memory.total,mig.mode.current \ + --format=csv + +nvidia-smi \ + --query-compute-apps=gpu_uuid,pid,process_name,used_gpu_memory \ + --format=csv +``` + +The eight cards reported MIG mode enabled. These are the two rows used later: + +```plaintext +4, NVIDIA RTX PRO 6000 Blackwell Server Edition, GPU-4c395b7a-a7e6-d90f-1ced-d96e8dd68288, 610.43.02, 97887 MiB, Enabled +5, NVIDIA RTX PRO 6000 Blackwell Server Edition, GPU-04dc48d7-7048-aef5-ad36-f5db716e7668, 610.43.02, 97887 MiB, Enabled +``` + +The process query returned only its header: + +```plaintext +gpu_uuid, pid, process_name, used_gpu_memory [MiB] +``` + +Stop or migrate every GPU workload that the handover must not disrupt, disable MIG Manager reconciliation, and repeat the process query. Do not continue until the node has an explicitly empty baseline. Enabling MIG mode, clearing old layouts, and the initial plugin startup can reset GPUs. + +## Step 2: Build the Pinned HAMi Source + +Clone the repository, detach at the full tested SHA, and build one immutable image for all HAMi components: + +```bash +export HAMI_SHA=634bf2b32e68e07d3fbcbd6da1ee079392fc07c1 +export HAMI_TAG=master-634bf2b32e68 +export HAMI_IMAGE=localhost/hami-dynamic-mig:$HAMI_TAG + +git clone --recurse-submodules \ + https://github.com/Project-HAMi/HAMi.git "$LAB/HAMi" +git -C "$LAB/HAMi" checkout --detach "$HAMI_SHA" +git -C "$LAB/HAMi" submodule update --init --recursive + +make -C "$LAB/HAMi" docker \ + IMG_NAME=localhost/hami-dynamic-mig \ + IMG_TAG="$HAMI_TAG" \ + VERSION="$HAMI_TAG" \ + TARGET_PLATFORMS=linux/amd64 + +docker image inspect "$HAMI_IMAGE" \ + --format='ID={{.Id}} Architecture={{.Architecture}} SizeBytes={{.Size}}' +``` + +:::note + +HAMi v2.10.0 (released 2026-08-21) includes commit `634bf2b`, so the released chart and images ship everything this lab builds from source. The lab keeps the pinned build it was verified with; a future revision will re-verify against the v2.10.0 release artifacts and drop the source build. + +::: + +The pinned build produced: + +```plaintext +ID=sha256:0ddda56e333ff74e52d9908e00b85e7860cf4694fc09951aaa178e8c8e6dde76 Architecture=amd64 SizeBytes=411671341 +``` + +The verified cluster was single-node, so it imported the local image into Kubernetes' containerd namespace: + +```bash +docker save --output "$LAB/hami-$HAMI_TAG.tar" "$HAMI_IMAGE" +sudo ctr --namespace k8s.io images import "$LAB/hami-$HAMI_TAG.tar" +sudo ctr --namespace k8s.io images list | grep -F "$HAMI_IMAGE" +``` + +`localhost/...` works only when every HAMi component runs on a node containing that imported image. In a multi-node cluster, push the immutable tag to a registry reachable by every scheduler and GPU node, then update the registry and pull policy in the supplied values. + +## Step 3: Render and Perform the Controlled Install + +Create node-specific copies of the supplied values and workload manifest: + +```bash +sed "s/__NODE_NAME__/${NODE}/g" "$EXAMPLES/hami-values.yaml" \ + > "$LAB/hami-values-one-gpu.yaml" +sed "s/__NODE_NAME__/${NODE}/g" "$EXAMPLES/mig-small-pack.yaml" \ + > "$LAB/mig-small-pack.yaml" +``` + +Two similarly named settings have separate responsibilities: + +- `devicePlugin.nodeConfiguration.config` sets `operatingmode: "mig"`, activating HAMi Dynamic MIG for this node. +- Top-level `devicePlugin.migStrategy: none` prevents the NVIDIA device-plugin path from publishing pre-created MIG resources such as `nvidia.com/mig-1g.24gb`. Workloads still request `nvidia.com/gpu`; HAMi creates their MIG instances dynamically. + +The `filterdevices.index` field is an **exclusion** list. `[0, 1, 2, 3, 5, 6, 7]` registers only GPU 4. It is not a startup safety boundary; Step 7 demonstrates that the plugin still reconciles filtered GPUs. + +Render before changing the cluster: + +```bash +helm lint "$LAB/HAMi/charts/hami" \ + -f "$LAB/hami-values-one-gpu.yaml" + +helm template hami "$LAB/HAMi/charts/hami" \ + --namespace hami-system \ + --kube-version 1.35.6 \ + -f "$LAB/hami-values-one-gpu.yaml" \ + > "$LAB/rendered-current-hami.yaml" + +grep -n -A 25 'migProfileAllowlist' \ + "$LAB/rendered-current-hami.yaml" +grep -n -E 'image:|imagePullPolicy:' \ + "$LAB/rendered-current-hami.yaml" +! grep -q 'projecthami/hami:v2.9.0' \ + "$LAB/rendered-current-hami.yaml" +``` + +Confirm the rendered allowlist includes `1g.24gb`, `2g.48gb`, and `4g.96gb` for `RTX PRO 6000 Blackwell Server Edition`, and that the scheduler extender, device plugin, and monitor all use `localhost/hami-dynamic-mig:master-634bf2b32e68`. The chart's `appVersion` still says `2.9.0` at this commit; live image references, not chart metadata, prove the runtime version. + +:::warning[Destructive handover] + +The verified run used a fresh reinstall only after all GPU Pods and processes were gone. This is not a general in-place upgrade procedure. Migrate an existing deployment with the pinned migration guide linked above. + +::: + +```bash +if helm status hami -n hami-system >/dev/null 2>&1; then + helm uninstall hami -n hami-system --wait --timeout 5m +fi + +helm upgrade --install hami "$LAB/HAMi/charts/hami" \ + -n hami-system \ + --create-namespace \ + --reset-values \ + -f "$LAB/hami-values-one-gpu.yaml" \ + --wait \ + --timeout 10m + +kubectl get pods -n hami-system \ + -o custom-columns='POD:.metadata.name,CONTAINERS:.spec.containers[*].name,IMAGES:.spec.containers[*].image' +``` + +Verify that all three HAMi containers show: + +```plaintext +localhost/hami-dynamic-mig:master-634bf2b32e68 +``` + +A stale MIG UUID caused one transient monitor CDI `StartError` during the verified handover. Kubernetes retried it and both plugin containers became ready. If you see a restart, inspect the previous state instead of assuming it was harmless: + +```bash +kubectl get pods -n hami-system +kubectl logs -n hami-system \ + -l app.kubernetes.io/component=hami-device-plugin \ + --all-containers=true --previous --tail=100 +``` + +## Step 4: Discover Placements and Create One `1g.24gb` + +Inspect what the plugin learned from NVML: + +```bash +kubectl get node "$NODE" -o json | +jq ' + .metadata.annotations["hami.io/node-nvidia-register"] + | fromjson + | .[] + | {id, index, type, mode, count, migProfiles} +' +``` + +GPU 4 registered these capabilities: + +| Profile | `memoryMB` | Core | `sliceCount` | Legal NVML placements (`start`, `size`) | +| --------- | ---------: | ---: | -----------: | --------------------------------------- | +| `1g.24gb` | 24,192 | 25 | 1 | `(0,3)`, `(3,3)`, `(6,3)`, `(9,3)` | +| `2g.48gb` | 48,512 | 50 | 2 | `(0,6)`, `(6,6)` | +| `4g.96gb` | 97,408 | 100 | 4 | `(0,12)` | + +`start` and `size` describe a half-open slice interval `[start, start + size)`; they are not GiB. The registered `count: 4` is only a coarse maximum. Actual capacity depends on non-overlapping legal placements. + +Create the namespace and one repeatable CUDA workload. It runs NVIDIA's `vectorAdd` sample continuously and increments `/tmp/gpu-progress` after every successful iteration. + +```bash +kubectl create namespace hami-mig-retest +kubectl apply -f "$LAB/mig-small-pack.yaml" +kubectl rollout status deployment/mig-small-pack \ + -n hami-mig-retest --timeout=180s + +POD=$(kubectl get pods -n hami-mig-retest \ + -l app=mig-small-pack \ + -o jsonpath='{.items[0].metadata.name}') +``` + +Inspect HAMi's controller-owned allocation identity. Users may read this annotation but must never create or edit it. + +```bash +kubectl get pod "$POD" -n hami-mig-retest -o json | +jq '.metadata.annotations["hami.io/vgpu-mig-allocations"] | fromjson' +``` + +The 8,000 MiB request selected the smallest fitting allowed profile: + +```json +[ + { + "containerIndex": 0, + "deviceIndex": 0, + "gpuUUID": "GPU-4c395b7a-a7e6-d90f-1ced-d96e8dd68288", + "profile": "1g.24gb", + "placement": { "start": 9, "size": 3 }, + "migUUID": "MIG-a5fa6120-f6fa-51b6-9820-a42112640629", + "gpuInstanceID": 6, + "computeInstanceID": 0 + } +] +``` + +On a Dynamic MIG node, `nvidia.com/gpumem: 8000` is a **minimum profile requirement**, not an 8,000 MiB software cap. This GPU has no 8 GiB profile, so the container receives the complete 24,192 MiB instance. `nvidia.com/gpucores` does not choose a MIG profile; the hardware profile fixes the compute fraction. + +Confirm that the host and container expose the same MIG UUID, then prove the workload advances: + +```bash +nvidia-smi -L +kubectl exec -n hami-mig-retest "$POD" -- nvidia-smi -L + +before=$(kubectl exec -n hami-mig-retest "$POD" -- cat /tmp/gpu-progress) +sleep 3 +after=$(kubectl exec -n hami-mig-retest "$POD" -- cat /tmp/gpu-progress) +printf 'before=%s after=%s\n' "$before" "$after" +test "$after" -gt "$before" +``` + +```plaintext +before=75 after=77 +``` + +The first placement need not start at 0; the verified first allocation legally started at 9. + +## Step 5: Fill All Four Legal Placements + +Scale the same Deployment to four Pods: + +```bash +kubectl scale deployment/mig-small-pack \ + -n hami-mig-retest --replicas=4 +kubectl rollout status deployment/mig-small-pack \ + -n hami-mig-retest --timeout=180s +nvidia-smi -L + +kubectl get pods -n hami-mig-retest -l app=mig-small-pack -o json | +jq -r ' + ["PARENT_GPU", "PROFILE", "START", "SIZE"], + ( + .items[] + | (.metadata.annotations["hami.io/vgpu-mig-allocations"] | fromjson | .[0]) as $a + | [$a.gpuUUID, $a.profile, ($a.placement.start | tostring), ($a.placement.size | tostring)] + ) + | @tsv +' +``` + +All four legal `1g.24gb` starts were occupied: + +```plaintext +PARENT_GPU PROFILE START SIZE +GPU-4c395b7a-a7e6-d90f-1ced-d96e8dd68288 1g.24gb 0 3 +GPU-4c395b7a-a7e6-d90f-1ced-d96e8dd68288 1g.24gb 3 3 +GPU-4c395b7a-a7e6-d90f-1ced-d96e8dd68288 1g.24gb 6 3 +GPU-4c395b7a-a7e6-d90f-1ced-d96e8dd68288 1g.24gb 9 3 +``` + +With only GPU 4 registered, a fifth replica remained unbound instead of overcommitting the card: + +```bash +kubectl scale deployment/mig-small-pack \ + -n hami-mig-retest --replicas=5 +sleep 15 +kubectl get pods -n hami-mig-retest -o wide + +PENDING_POD=$(kubectl get pods -n hami-mig-retest \ + -l app=mig-small-pack --field-selector=status.phase=Pending \ + -o jsonpath='{.items[0].metadata.name}') +kubectl describe pod "$PENDING_POD" -n hami-mig-retest | \ + grep 'CardTimeSlicingExhausted' +``` + +Its scheduling event included: + +```plaintext +0/1 nodes are available: 1 1/1 CardTimeSlicingExhausted. +``` + +The inherited event name is misleading here: this test did not use time slicing. It means no legal Dynamic MIG placement remained on a registered GPU. Return to four replicas before continuing: + +```bash +kubectl scale deployment/mig-small-pack \ + -n hami-mig-retest --replicas=4 +``` + +## Step 6: Mix Profiles and Reclaim Only One Instance + +Remove the packing Pods, derive GPU 4's UUID on this host, and run the supplied script. It creates an 8,000 MiB Pod and a 30,000 MiB Pod with the same CUDA progress loop and pins both to the same physical card. + +```bash +kubectl scale deployment/mig-small-pack \ + -n hami-mig-retest --replicas=0 +kubectl wait -n hami-mig-retest \ + --for=delete pod -l app=mig-small-pack --timeout=180s + +export GPU_UUID=$(nvidia-smi -i 4 --query-gpu=uuid --format=csv,noheader) +"$EXAMPLES/create-mixed-pods.sh" +``` + +Inspect both allocation records: + +```bash +kubectl get pods mixed-small mixed-large -n hami-mig-retest -o json | +jq -r ' + ["POD", "PROFILE", "START", "SIZE", "MIG_UUID", "GI", "CI"], + ( + .items + | sort_by(.metadata.name)[] + | . as $pod + | ($pod.metadata.annotations["hami.io/vgpu-mig-allocations"] | fromjson | .[0]) as $a + | [ + $pod.metadata.name, + $a.profile, + ($a.placement.start | tostring), + ($a.placement.size | tostring), + $a.migUUID, + ($a.gpuInstanceID | tostring), + ($a.computeInstanceID | tostring) + ] + ) + | @tsv +' +``` + +The live allocation table was: + +```plaintext +POD PROFILE START SIZE MIG_UUID GI CI +mixed-large 2g.48gb 0 6 MIG-b23491d8-d784-58d9-bcfa-3c171ead22da 1 0 +mixed-small 1g.24gb 9 3 MIG-a5fa6120-f6fa-51b6-9820-a42112640629 6 0 +``` + +Intervals `[0,6)` and `[9,12)` do not overlap, so both profiles fit. During the same three-second window both loops progressed: + +```bash +small_before=$(kubectl exec -n hami-mig-retest mixed-small -- cat /tmp/gpu-progress) +large_before=$(kubectl exec -n hami-mig-retest mixed-large -- cat /tmp/gpu-progress) +sleep 3 +small_after=$(kubectl exec -n hami-mig-retest mixed-small -- cat /tmp/gpu-progress) +large_after=$(kubectl exec -n hami-mig-retest mixed-large -- cat /tmp/gpu-progress) +printf 'small: %s -> %s\nlarge: %s -> %s\n' \ + "$small_before" "$small_after" "$large_before" "$large_after" +test "$small_after" -gt "$small_before" +test "$large_after" -gt "$large_before" +``` + +```plaintext +small: 64 -> 67 +large: 37 -> 39 +``` + +Now capture the small instance identity, delete only its Pod, and poll the host because reclamation is asynchronous: + +```bash +small_mig_uuid=$(kubectl get pod mixed-small -n hami-mig-retest -o json | + jq -r '.metadata.annotations["hami.io/vgpu-mig-allocations"] | fromjson | .[0].migUUID') +large_before=$(kubectl exec -n hami-mig-retest mixed-large -- \ + cat /tmp/gpu-progress) + +kubectl delete pod mixed-small -n hami-mig-retest + +until ! nvidia-smi -L | grep -Fq "$small_mig_uuid"; do + sleep 1 +done +nvidia-smi -L | grep '^ MIG ' +``` + +Only the large instance remained: + +```plaintext +MIG 2g.48gb Device 0: (UUID: MIG-b23491d8-d784-58d9-bcfa-3c171ead22da) +``` + +Verify the neighbor continued computing throughout the reclaim: + +```bash +large_after=$(kubectl exec -n hami-mig-retest mixed-large -- \ + cat /tmp/gpu-progress) +printf 'large: %s -> %s\n' "$large_before" "$large_after" +test "$large_after" -gt "$large_before" \ + && echo 'PASS: 2g workload survived 1g reclamation' +``` + +```plaintext +large: 61 -> 94 +PASS: 2g workload survived 1g reclamation +``` + +On this GPU and driver, recreating the freed placement later produced the same `MIG-a5fa...` UUID. A MIG UUID is not a generation counter: observed disappearance proves reclamation, while a different UUID is not required for recreation. + +## Step 7: Restart the Device Plugin and Verify UUID Stability + +This is a disruptive controller test. Keep only the valid, HAMi-managed `mixed-large` allocation active. Every other GPU on the node must remain free of unmanaged work, because plugin startup has node-wide hardware scope at this commit. + +Record the allocation's UUID and progress, replace the device-plugin Pod, and wait for the DaemonSet: + +```bash +LARGE_MIG_UUID=$(kubectl get pod mixed-large -n hami-mig-retest -o json | + jq -r '.metadata.annotations["hami.io/vgpu-mig-allocations"] | fromjson | .[0].migUUID') +OLD_DP_POD=$(kubectl get pods -n hami-system \ + -l app.kubernetes.io/component=hami-device-plugin \ + -o jsonpath='{.items[0].metadata.name}') +progress_before=$(kubectl exec -n hami-mig-retest mixed-large -- \ + cat /tmp/gpu-progress) + +kubectl delete pod "$OLD_DP_POD" -n hami-system +kubectl rollout status daemonset/hami-device-plugin \ + -n hami-system --timeout=180s + +NEW_DP_POD=$(kubectl get pods -n hami-system \ + -l app.kubernetes.io/component=hami-device-plugin \ + -o jsonpath='{.items[0].metadata.name}') +kubectl logs "$NEW_DP_POD" -n hami-system --all-containers=true | + grep 'mig init: resolved startup layout' +``` + +The replacement plugin classified GPU 4 as in use and all other GPUs as reset candidates: + +```plaintext +mig init: resolved startup layout inUseGPUs=[4] resetGPUs=[0,1,2,3,5,6,7] +``` + +It verified the complete Pod annotation against NVML and adopted the live allocation. Confirm the exact UUID still exists and the CUDA loop advanced: + +```bash +nvidia-smi -L | grep -F "$LARGE_MIG_UUID" + +progress_after=$(kubectl exec -n hami-mig-retest mixed-large -- \ + cat /tmp/gpu-progress) +printf 'progress: %s -> %s\n' "$progress_before" "$progress_after" +test "$progress_after" -gt "$progress_before" \ + && echo 'PASS: MIG UUID and CUDA workload survived device-plugin restart' +``` + +```plaintext +progress: 115 -> 187 +PASS: MIG UUID and CUDA workload survived device-plugin restart +``` + +:::danger[Filtering does not constrain startup mutation] + +The log proves that `filterdevices` limits registration and scheduling but not Dynamic MIG startup cleanup at this commit. The plugin reconciled all eight physical GPUs, including filtered ones. Treat the first install and every plugin restart as whole-node maintenance. This happy-path recovery also assumes a complete, valid allocation annotation; it does not promise adoption of malformed state. + +::: + +## Step 8: Expose GPU 5 and Verify Fifth-Pod Spillover + +Delete the mixed-profile workload and wait until no test MIG instance remains: + +```bash +kubectl delete pod mixed-large -n hami-mig-retest + +until ! nvidia-smi -L | grep -q '^ MIG '; do + sleep 2 +done +``` + +Create a second values file that removes GPU 5 from the exclusion list. This changes the list from `[0, 1, 2, 3, 5, 6, 7]` to `[0, 1, 2, 3, 6, 7]`, registering GPUs 4 and 5. + +```bash +sed 's/"index": \[0, 1, 2, 3, 5, 6, 7\]/"index": [0, 1, 2, 3, 6, 7]/' \ + "$LAB/hami-values-one-gpu.yaml" > "$LAB/hami-values-two-gpus.yaml" + +helm upgrade hami "$LAB/HAMi/charts/hami" \ + -n hami-system \ + --reset-values \ + -f "$LAB/hami-values-two-gpus.yaml" \ + --wait \ + --timeout 10m + +# This ConfigMap change did not trigger a plugin rollout in the verified chart. +kubectl rollout restart daemonset/hami-device-plugin -n hami-system +kubectl rollout status daemonset/hami-device-plugin \ + -n hami-system --timeout=180s +``` + +Do not trust Helm success alone. Verify the live node registration: + +```bash +kubectl get node "$NODE" -o json | +jq -r ' + .metadata.annotations["hami.io/node-nvidia-register"] + | fromjson + | map(.index) + | sort + | join(",") +' +``` + +```plaintext +4,5 +``` + +Scale the existing Deployment from zero to five and inspect each parent GPU: + +```bash +kubectl scale deployment/mig-small-pack \ + -n hami-mig-retest --replicas=5 +kubectl rollout status deployment/mig-small-pack \ + -n hami-mig-retest --timeout=180s + +kubectl get pods -n hami-mig-retest -l app=mig-small-pack -o json | +jq -r ' + ["POD", "PARENT_GPU", "PROFILE", "START"], + ( + .items + | sort_by(.metadata.name)[] + | . as $pod + | ($pod.metadata.annotations["hami.io/vgpu-mig-allocations"] | fromjson | .[0]) as $a + | [ + $pod.metadata.name, + $a.gpuUUID, + $a.profile, + ($a.placement.start | tostring) + ] + ) + | @tsv +' +``` + +The verified bin-packing result filled all four placements on GPU 4, then placed the fifth Pod on GPU 5: + +```plaintext +POD PARENT_GPU PROFILE START +mig-small-pack-6f5b7bd7b-dwld2 GPU-4c395b7a-a7e6-d90f-1ced-d96e8dd68288 1g.24gb 3 +mig-small-pack-6f5b7bd7b-g72fd GPU-4c395b7a-a7e6-d90f-1ced-d96e8dd68288 1g.24gb 0 +mig-small-pack-6f5b7bd7b-jgql2 GPU-04dc48d7-7048-aef5-ad36-f5db716e7668 1g.24gb 9 +mig-small-pack-6f5b7bd7b-rlxbn GPU-4c395b7a-a7e6-d90f-1ced-d96e8dd68288 1g.24gb 9 +mig-small-pack-6f5b7bd7b-vjfv9 GPU-4c395b7a-a7e6-d90f-1ced-d96e8dd68288 1g.24gb 6 +``` + +Placement starts are legal choices, not an allocation sequence; GPU 5's first allocation may start at 9. + +## Cleanup + +Delete the test namespace and verify that all per-Pod instances are gone before another plugin restart: + +```bash +kubectl delete namespace hami-mig-retest \ + --wait=true --timeout=180s + +if nvidia-smi -L | grep -q '^ MIG '; then + echo 'FAIL: MIG instances remain' + nvidia-smi -L +else + echo 'PASS: no MIG instances remain' +fi +``` + +```plaintext +PASS: no MIG instances remain +``` + +Restore the original exclusion list, then deliberately restart the plugin while the whole node is idle: + +```bash +helm upgrade hami "$LAB/HAMi/charts/hami" \ + -n hami-system \ + --reset-values \ + -f "$LAB/hami-values-one-gpu.yaml" \ + --wait \ + --timeout 10m +kubectl rollout restart daemonset/hami-device-plugin -n hami-system +kubectl rollout status daemonset/hami-device-plugin \ + -n hami-system --timeout=180s + +printf 'Registered GPU indices: ' +kubectl get node "$NODE" -o json | +jq -r ' + .metadata.annotations["hami.io/node-nvidia-register"] + | fromjson + | map(.index) + | join(",") +' + +if nvidia-smi -L | grep -q '^ MIG '; then + echo 'MIG state: FAIL - instances remain' +else + echo 'MIG state: PASS - no instances remain' +fi +kubectl get pods -n hami-system +``` + +The verified final state was: + +```plaintext +Registered GPU indices: 4 +MIG state: PASS - no instances remain +NAME READY STATUS RESTARTS +hami-device-plugin-6snlc 2/2 Running 0 +hami-scheduler-74fbfcfbb5-qxftm 2/2 Running 0 +``` + +This leaves the pinned HAMi installation running with only GPU 4 registered. Preserve the Step 1 backups until you have either accepted this installation or restored the previous deployment through its documented migration or rollback procedure. Do not directly roll binaries back to a legacy Dynamic MIG implementation while new-format allocations are active. + +## Operational Traps + +- **Chart metadata is not runtime identity.** At this commit the chart defaults to v2.9.0. Pin and inspect the extender, plugin, and monitor images; do not use `latest` or rely on `appVersion`. +- **`operatingmode` is not `migStrategy`.** The node JSON selects HAMi Dynamic MIG; the top-level Helm value controls NVIDIA's static resource exposure path. +- **MIG Manager and HAMi cannot share mutation ownership.** Stop reconciliation, not just one Pod, before HAMi starts managing GI/CI state. +- **`filterdevices` is not a hardware protection boundary.** It excludes registration, while startup reconciliation can still touch every GPU on the node. +- **A Helm upgrade may not restart the plugin.** The tested DaemonSet had no checksum for the node-configuration ConfigMap. Restart only in a safe window, then inspect the live registration annotation. +- **Scheduler reasons can use inherited language.** `CardTimeSlicingExhausted` represented exhausted MIG placements here, not a switch to time slicing. +- **Reclamation is eventual and UUIDs may be reused.** Poll host state after deletion. Disappearance and later reappearance is stronger evidence than expecting a new UUID. +- **Dynamic placement is still constrained.** Profiles coexist only when NVML reports non-overlapping legal intervals; HAMi does not move or destroy a live neighbor to satisfy a new request. +- **A homogeneous test is not a heterogeneous-node guarantee.** The verified node had eight identical supported GPUs. Validate mixed-model nodes separately. + +## What This Lab Proved + +| Claim | Evidence | +| --- | --- | +| 8,000 MiB selects real hardware isolation | The Pod received one `1g.24gb` GI/CI and the same MIG UUID appeared on host and in container | +| One RTX PRO 6000 has four small-profile placements | Starts 0, 3, 6, and 9 were occupied; a fifth Pod stayed `Pending` while only GPU 4 was registered | +| Different profiles can coexist | `2g.48gb` at `[0,6)` and `1g.24gb` at `[9,12)` ran CUDA together | +| Reclamation is selective | Deleting `mixed-small` removed only its GI/CI while `mixed-large` advanced from 61 to 94 | +| Valid allocation state is recoverable | Plugin restart retained the `2g.48gb` UUID and CUDA progress advanced from 115 to 187 | +| Capacity spills across GPUs | With GPUs 4 and 5 registered, four Pods packed onto GPU 4 and the fifth used GPU 5 | + +## Next Steps + +- Read the pinned [migration guide](https://github.com/Project-HAMi/HAMi/blob/634bf2b32e68e07d3fbcbd6da1ee079392fc07c1/docs/develop/dynamic-mig-migration.md) before moving a production node from fixed geometries or MIG Manager. +- Compare this hardware-isolated path with [Lab 7: GPU Isolation on k3s Without the GPU Operator](./hami-isolation-k3s.md), which verifies HAMi-core software isolation. +- Validate the [NVIDIA supported MIG profiles](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/latest/supported-mig-profiles.html) for every GPU model and driver in your fleet. diff --git a/tutorials/labs/examples/15-dynamic-mig-rtx-pro/create-mixed-pods.sh b/tutorials/labs/examples/15-dynamic-mig-rtx-pro/create-mixed-pods.sh new file mode 100755 index 000000000..ab1b33c91 --- /dev/null +++ b/tutorials/labs/examples/15-dynamic-mig-rtx-pro/create-mixed-pods.sh @@ -0,0 +1,55 @@ +#!/usr/bin/env bash +set -euo pipefail + +: "${NODE:?Set NODE to the Kubernetes node name}" +: "${GPU_UUID:?Set GPU_UUID to the physical GPU UUID used for the mixed-profile test}" + +NAMESPACE=${NAMESPACE:-hami-mig-retest} + +create_mig_pod() { + local name=$1 + local memory=$2 + + kubectl apply -f - < /tmp/gpu-progress + while true; do + /cuda-samples/vectorAdd > /tmp/vectoradd.last 2>&1 + n=\$((n + 1)) + echo "\$n" > /tmp/gpu-progress.next + mv /tmp/gpu-progress.next /tmp/gpu-progress + done + resources: + limits: + nvidia.com/gpu: 1 + nvidia.com/gpumem: ${memory} +EOF +} + +create_mig_pod mixed-small 8000 +create_mig_pod mixed-large 30000 + +kubectl wait -n "${NAMESPACE}" --for=condition=Ready \ + pod/mixed-small pod/mixed-large --timeout=180s diff --git a/tutorials/labs/examples/15-dynamic-mig-rtx-pro/hami-values.yaml b/tutorials/labs/examples/15-dynamic-mig-rtx-pro/hami-values.yaml new file mode 100644 index 000000000..9d2efe1ad --- /dev/null +++ b/tutorials/labs/examples/15-dynamic-mig-rtx-pro/hami-values.yaml @@ -0,0 +1,57 @@ +global: + imageTag: master-634bf2b32e68 + +scheduler: + defaultSchedulerPolicy: + nodeSchedulerPolicy: binpack + gpuSchedulerPolicy: binpack + extender: + image: + registry: localhost + repository: hami-dynamic-mig + tag: master-634bf2b32e68 + pullPolicy: Never + +devicePlugin: + image: + registry: localhost + repository: hami-dynamic-mig + tag: master-634bf2b32e68 + pullPolicy: Never + monitor: + image: + registry: localhost + repository: hami-dynamic-mig + tag: master-634bf2b32e68 + pullPolicy: Never + + # NVIDIA's static MIG resource exposure strategy. This is not HAMi's + # per-node Dynamic MIG operating mode below. + migStrategy: none + + nodeConfiguration: + config: | + { + "nodeconfig": [ + { + "name": "__NODE_NAME__", + "operatingmode": "mig", + "devicememoryscaling": 1, + "devicecorescaling": 1, + "devicesplitcount": 10, + "preconfigureddevicememory": 0, + "enablenumatopology": false, + "migstrategy": "none", + "filterdevices": { + "uuid": [], + "index": [0, 1, 2, 3, 5, 6, 7] + }, + "enablegetpreferredallocation": false + } + ] + } + +# Use the device-config.yaml bundled with the pinned source chart. It contains +# the RTX PRO 6000 profile allowlist tested by this lab. +device-config: + content: "" diff --git a/tutorials/labs/examples/15-dynamic-mig-rtx-pro/mig-small-pack.yaml b/tutorials/labs/examples/15-dynamic-mig-rtx-pro/mig-small-pack.yaml new file mode 100644 index 000000000..92efc5a45 --- /dev/null +++ b/tutorials/labs/examples/15-dynamic-mig-rtx-pro/mig-small-pack.yaml @@ -0,0 +1,42 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: mig-small-pack + namespace: hami-mig-retest +spec: + replicas: 1 + selector: + matchLabels: + app: mig-small-pack + template: + metadata: + labels: + app: mig-small-pack + annotations: + nvidia.com/vgpu-mode: "mig" + hami.io/gpu-scheduler-policy: "binpack" + spec: + schedulerName: hami-scheduler + nodeSelector: + kubernetes.io/hostname: __NODE_NAME__ + containers: + - name: cuda + image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04 + imagePullPolicy: IfNotPresent + command: + - bash + - -lc + - | + set -euo pipefail + n=0 + echo 0 > /tmp/gpu-progress + while true; do + /cuda-samples/vectorAdd > /tmp/vectoradd.last 2>&1 + n=$((n + 1)) + echo "$n" > /tmp/gpu-progress.next + mv /tmp/gpu-progress.next /tmp/gpu-progress + done + resources: + limits: + nvidia.com/gpu: 1 + nvidia.com/gpumem: 8000 diff --git a/tutorials/overview.md b/tutorials/overview.md index 3294c270a..555dcf410 100644 --- a/tutorials/overview.md +++ b/tutorials/overview.md @@ -17,7 +17,7 @@ Background knowledge that the labs build on. ## Labs - + Lab 15 builds a pinned HAMi snapshot and verifies the complete Dynamic MIG lifecycle on RTX PRO 6000 hardware. Each lab lists its own prerequisites. From 19fba88ebd725913f05f85891f39a3a2c0523379 Mon Sep 17 00:00:00 2001 From: Saiyam Pathak Date: Wed, 2 Sep 2026 17:24:53 +0530 Subject: [PATCH 2/6] docs(tutorials): address review on the RTX PRO 6000 Dynamic MIG lab - Align the zh tag with the en slug (gpu-partitioning) so tag pages match across locales. - Parameterize the GPU choice: Step 1 defines PRIMARY_GPU and SECONDARY_GPU, Step 3 checks the one-GPU exclusion list, Step 6 derives the UUID from PRIMARY_GPU, and Step 8 removes SECONDARY_GPU from the list with a variable-driven sed and prints the result. - Scope the device-plugin Pod selection in Step 7 to $NODE and add a Step 1 check that $NODE is the only gpu=on node before the DaemonSet restarts. Signed-off-by: Saiyam Pathak --- .../current/labs/dynamic-mig-rtx-pro.md | 2 +- tutorials/labs/dynamic-mig-rtx-pro.md | 31 ++++++++++++++----- 2 files changed, 25 insertions(+), 8 deletions(-) diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/dynamic-mig-rtx-pro.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/dynamic-mig-rtx-pro.md index 5453caf75..a10bdacac 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/dynamic-mig-rtx-pro.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/dynamic-mig-rtx-pro.md @@ -12,7 +12,7 @@ lab: - saiyam1814 verified: "2026-08-11" tags: - - GPU 分区 + - gpu-partitioning - nvidia - hami toc_max_heading_level: 2 diff --git a/tutorials/labs/dynamic-mig-rtx-pro.md b/tutorials/labs/dynamic-mig-rtx-pro.md index ba44da86d..ee1f68fac 100644 --- a/tutorials/labs/dynamic-mig-rtx-pro.md +++ b/tutorials/labs/dynamic-mig-rtx-pro.md @@ -73,7 +73,7 @@ You also need: - a local checkout of this website repository for the files under [`tutorials/labs/examples/15-dynamic-mig-rtx-pro/`](https://github.com/Project-HAMi/website/tree/master/tutorials/labs/examples/15-dynamic-mig-rtx-pro); and - an explicit maintenance window for the **whole GPU node**, not only the GPUs that HAMi will register. -The supplied values target the verified eight-GPU node and initially register only GPU index 4. Adapt both `filterdevices.index` lists before starting if your topology differs. You need at least two compatible GPUs to reproduce Step 8. +The supplied values target the verified eight-GPU node and initially register only GPU index 4. If your topology differs, choose your own primary and spillover GPU indices in Step 1 and edit the `filterdevices.index` list in the supplied values so that it excludes every index except the primary GPU. You need at least two compatible GPUs to reproduce Step 8. :::danger[Assign one MIG hardware owner] @@ -87,16 +87,26 @@ Host-level `nvidia-smi`, Docker, and `ctr` commands run on the GPU node. `kubect ## Step 1: Back Up and Establish an Idle Handover -Select the single Kubernetes node and set a durable working directory. If your cluster has other nodes, set `NODE` explicitly to the eight-GPU node instead. +Select the single Kubernetes node, choose the two GPU indices this lab uses, and set a durable working directory. If your cluster has other nodes, set `NODE` explicitly to the eight-GPU node instead. ```bash export NODE=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}') +export PRIMARY_GPU=4 # the only GPU registered with HAMi until Step 8 +export SECONDARY_GPU=5 # the spillover GPU added in Step 8 export LAB=/root/hami-dynamic-mig-rerun-2026-08-11 export EXAMPLES=tutorials/labs/examples/15-dynamic-mig-rtx-pro mkdir -p "$LAB" ``` +The verified run used GPU 4 and GPU 5. Every later command that touches those GPUs reads these two variables. The supplied `hami-values.yaml` still lists the excluded indices literally; Step 3 shows where to check it. + +Steps 7 and 8 restart the `hami-device-plugin` DaemonSet. The chart schedules it on every node labeled `gpu=on`, so confirm that `$NODE` is the only such node before continuing: + +```bash +kubectl get nodes -l gpu=on -o name +``` + If a release named `hami` already exists in `hami-system`, save both Helm's stored state and the live objects; they can differ. ```bash @@ -195,8 +205,12 @@ sed "s/__NODE_NAME__/${NODE}/g" "$EXAMPLES/hami-values.yaml" \ > "$LAB/hami-values-one-gpu.yaml" sed "s/__NODE_NAME__/${NODE}/g" "$EXAMPLES/mig-small-pack.yaml" \ > "$LAB/mig-small-pack.yaml" + +grep -n '"index"' "$LAB/hami-values-one-gpu.yaml" ``` +The `grep` output must list every GPU index on the node except `$PRIMARY_GPU`. If you chose a different primary GPU in Step 1, edit the list in `$LAB/hami-values-one-gpu.yaml` before continuing. + Two similarly named settings have separate responsibilities: - `devicePlugin.nodeConfiguration.config` sets `operatingmode: "mig"`, activating HAMi Dynamic MIG for this node. @@ -409,7 +423,7 @@ kubectl scale deployment/mig-small-pack \ ## Step 6: Mix Profiles and Reclaim Only One Instance -Remove the packing Pods, derive GPU 4's UUID on this host, and run the supplied script. It creates an 8,000 MiB Pod and a 30,000 MiB Pod with the same CUDA progress loop and pins both to the same physical card. +Remove the packing Pods, derive the primary GPU's UUID on this host, and run the supplied script. It creates an 8,000 MiB Pod and a 30,000 MiB Pod with the same CUDA progress loop and pins both to the same physical card. ```bash kubectl scale deployment/mig-small-pack \ @@ -417,7 +431,7 @@ kubectl scale deployment/mig-small-pack \ kubectl wait -n hami-mig-retest \ --for=delete pod -l app=mig-small-pack --timeout=180s -export GPU_UUID=$(nvidia-smi -i 4 --query-gpu=uuid --format=csv,noheader) +export GPU_UUID=$(nvidia-smi -i "$PRIMARY_GPU" --query-gpu=uuid --format=csv,noheader) "$EXAMPLES/create-mixed-pods.sh" ``` @@ -516,13 +530,14 @@ On this GPU and driver, recreating the freed placement later produced the same ` This is a disruptive controller test. Keep only the valid, HAMi-managed `mixed-large` allocation active. Every other GPU on the node must remain free of unmanaged work, because plugin startup has node-wide hardware scope at this commit. -Record the allocation's UUID and progress, replace the device-plugin Pod, and wait for the DaemonSet: +Record the allocation's UUID and progress, replace the device-plugin Pod running on `$NODE`, and wait for the DaemonSet: ```bash LARGE_MIG_UUID=$(kubectl get pod mixed-large -n hami-mig-retest -o json | jq -r '.metadata.annotations["hami.io/vgpu-mig-allocations"] | fromjson | .[0].migUUID') OLD_DP_POD=$(kubectl get pods -n hami-system \ -l app.kubernetes.io/component=hami-device-plugin \ + --field-selector spec.nodeName="$NODE" \ -o jsonpath='{.items[0].metadata.name}') progress_before=$(kubectl exec -n hami-mig-retest mixed-large -- \ cat /tmp/gpu-progress) @@ -533,6 +548,7 @@ kubectl rollout status daemonset/hami-device-plugin \ NEW_DP_POD=$(kubectl get pods -n hami-system \ -l app.kubernetes.io/component=hami-device-plugin \ + --field-selector spec.nodeName="$NODE" \ -o jsonpath='{.items[0].metadata.name}') kubectl logs "$NEW_DP_POD" -n hami-system --all-containers=true | grep 'mig init: resolved startup layout' @@ -579,11 +595,12 @@ until ! nvidia-smi -L | grep -q '^ MIG '; do done ``` -Create a second values file that removes GPU 5 from the exclusion list. This changes the list from `[0, 1, 2, 3, 5, 6, 7]` to `[0, 1, 2, 3, 6, 7]`, registering GPUs 4 and 5. +Create a second values file that removes `$SECONDARY_GPU` from the exclusion list. In the verified run this changed the list from `[0, 1, 2, 3, 5, 6, 7]` to `[0, 1, 2, 3, 6, 7]`, registering GPUs 4 and 5. The `grep` output must show the shorter list; if it still matches the one-GPU file, `SECONDARY_GPU` was not in the list. ```bash -sed 's/"index": \[0, 1, 2, 3, 5, 6, 7\]/"index": [0, 1, 2, 3, 6, 7]/' \ +sed -E "/\"index\":/ { s/(\[|, )${SECONDARY_GPU}, /\1/; s/, ${SECONDARY_GPU}\]/]/; }" \ "$LAB/hami-values-one-gpu.yaml" > "$LAB/hami-values-two-gpus.yaml" +grep -n '"index"' "$LAB/hami-values-two-gpus.yaml" helm upgrade hami "$LAB/HAMi/charts/hami" \ -n hami-system \ From ddfebfa9518b196c8bc2422746db1fd3bfbca6f9 Mon Sep 17 00:00:00 2001 From: Saiyam Pathak Date: Wed, 2 Sep 2026 17:38:06 +0530 Subject: [PATCH 3/6] docs(tutorials): credit both kubesimplify authors on the RTX PRO 6000 lab The lab is derived from the Dynamic MIG post that Shubham Katara and Saiyam Pathak wrote together on the kubesimplify blog. Name both authors in the intro next to the existing frontmatter credit and record the co-authorship in git. Co-authored-by: Shubham Katara <13725551+shkatara@users.noreply.github.com> Signed-off-by: Saiyam Pathak --- tutorials/labs/dynamic-mig-rtx-pro.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tutorials/labs/dynamic-mig-rtx-pro.md b/tutorials/labs/dynamic-mig-rtx-pro.md index ee1f68fac..2601ca9b8 100644 --- a/tutorials/labs/dynamic-mig-rtx-pro.md +++ b/tutorials/labs/dynamic-mig-rtx-pro.md @@ -20,7 +20,7 @@ toc_max_heading_level: 2 This lab builds HAMi from the source snapshot that first contained the merged per-Pod Dynamic MIG implementation, then follows one MIG allocation through creation, saturation, mixed-profile placement, selective reclamation, device-plugin adoption, and spillover to a second GPU. A Pod asks for memory through HAMi's usual resource API; HAMi chooses the smallest allowed NVIDIA MIG profile with enough memory and a legal free placement, then creates and later reclaims that Pod's GPU Instance (GI) and Compute Instance (CI). -The commands and outputs were captured on 2026-08-11 from the [original verified test](https://blog.kubesimplify.com/dynamic-mig-in-kubernetes-with-hami). This tutorial removes the surrounding narrative and retains the reproducible procedure and evidence. +The commands and outputs were captured on 2026-08-11 from the [original verified test](https://blog.kubesimplify.com/dynamic-mig-in-kubernetes-with-hami), which [Shubham Katara](https://github.com/shkatara) and [Saiyam Pathak](https://github.com/saiyam1814) wrote together on the kubesimplify blog. This tutorial removes the surrounding narrative and retains the reproducible procedure and evidence. :::caution[Experimental source snapshot] From b0efd5f1e327ad4555ffad5060fa10389ac5ddef Mon Sep 17 00:00:00 2001 From: Saiyam Pathak Date: Wed, 2 Sep 2026 17:52:20 +0530 Subject: [PATCH 4/6] docs(tutorials): derive both MIG exclusion lists from the chosen GPUs Replace the literal filterdevices.index list in the supplied values with a placeholder. Step 3 renders the one-GPU list from the node's nvidia-smi inventory minus PRIMARY_GPU, and Step 8 renders the two-GPU list from the same inventory minus PRIMARY_GPU and SECONDARY_GPU, so no manual edit of the exclusion list is needed. Default values reproduce the verified run. Signed-off-by: Saiyam Pathak --- tutorials/labs/dynamic-mig-rtx-pro.md | 28 ++++++++++++------- .../15-dynamic-mig-rtx-pro/hami-values.yaml | 5 +++- 2 files changed, 22 insertions(+), 11 deletions(-) diff --git a/tutorials/labs/dynamic-mig-rtx-pro.md b/tutorials/labs/dynamic-mig-rtx-pro.md index 2601ca9b8..2dee178ef 100644 --- a/tutorials/labs/dynamic-mig-rtx-pro.md +++ b/tutorials/labs/dynamic-mig-rtx-pro.md @@ -73,7 +73,7 @@ You also need: - a local checkout of this website repository for the files under [`tutorials/labs/examples/15-dynamic-mig-rtx-pro/`](https://github.com/Project-HAMi/website/tree/master/tutorials/labs/examples/15-dynamic-mig-rtx-pro); and - an explicit maintenance window for the **whole GPU node**, not only the GPUs that HAMi will register. -The supplied values target the verified eight-GPU node and initially register only GPU index 4. If your topology differs, choose your own primary and spillover GPU indices in Step 1 and edit the `filterdevices.index` list in the supplied values so that it excludes every index except the primary GPU. You need at least two compatible GPUs to reproduce Step 8. +The supplied values target the verified eight-GPU node and initially register only GPU index 4. If your topology differs, choose your own primary and spillover GPU indices in Step 1; Steps 3 and 8 derive the `filterdevices.index` exclusion lists from those choices and the node's GPU inventory. You need at least two compatible GPUs to reproduce Step 8. :::danger[Assign one MIG hardware owner] @@ -99,7 +99,7 @@ export EXAMPLES=tutorials/labs/examples/15-dynamic-mig-rtx-pro mkdir -p "$LAB" ``` -The verified run used GPU 4 and GPU 5. Every later command that touches those GPUs reads these two variables. The supplied `hami-values.yaml` still lists the excluded indices literally; Step 3 shows where to check it. +The verified run used GPU 4 and GPU 5. Every later command that touches those GPUs reads these two variables, and Steps 3 and 8 derive the `filterdevices.index` exclusion lists from them. Steps 7 and 8 restart the `hami-device-plugin` DaemonSet. The chart schedules it on every node labeled `gpu=on`, so confirm that `$NODE` is the only such node before continuing: @@ -198,25 +198,29 @@ sudo ctr --namespace k8s.io images list | grep -F "$HAMI_IMAGE" ## Step 3: Render and Perform the Controlled Install -Create node-specific copies of the supplied values and workload manifest: +Create node-specific copies of the supplied values and workload manifest. The exclusion list is every GPU index that `nvidia-smi` reports except `$PRIMARY_GPU`, so run this on the GPU node: ```bash -sed "s/__NODE_NAME__/${NODE}/g" "$EXAMPLES/hami-values.yaml" \ - > "$LAB/hami-values-one-gpu.yaml" +ONE_GPU_EXCLUDES=$(nvidia-smi --query-gpu=index --format=csv,noheader | tr -d ' ' | + grep -vx "$PRIMARY_GPU" | paste -sd ',' - | sed 's/,/, /g') + +sed -e "s/__NODE_NAME__/${NODE}/g" \ + -e "s/__EXCLUDED_GPU_INDICES__/${ONE_GPU_EXCLUDES}/" \ + "$EXAMPLES/hami-values.yaml" > "$LAB/hami-values-one-gpu.yaml" sed "s/__NODE_NAME__/${NODE}/g" "$EXAMPLES/mig-small-pack.yaml" \ > "$LAB/mig-small-pack.yaml" grep -n '"index"' "$LAB/hami-values-one-gpu.yaml" ``` -The `grep` output must list every GPU index on the node except `$PRIMARY_GPU`. If you chose a different primary GPU in Step 1, edit the list in `$LAB/hami-values-one-gpu.yaml` before continuing. +In the verified run the `grep` output showed `[0, 1, 2, 3, 5, 6, 7]`, which registers only GPU 4. Two similarly named settings have separate responsibilities: - `devicePlugin.nodeConfiguration.config` sets `operatingmode: "mig"`, activating HAMi Dynamic MIG for this node. - Top-level `devicePlugin.migStrategy: none` prevents the NVIDIA device-plugin path from publishing pre-created MIG resources such as `nvidia.com/mig-1g.24gb`. Workloads still request `nvidia.com/gpu`; HAMi creates their MIG instances dynamically. -The `filterdevices.index` field is an **exclusion** list. `[0, 1, 2, 3, 5, 6, 7]` registers only GPU 4. It is not a startup safety boundary; Step 7 demonstrates that the plugin still reconciles filtered GPUs. +The `filterdevices.index` field is an **exclusion** list; the rendered `[0, 1, 2, 3, 5, 6, 7]` registers only GPU 4. It is not a startup safety boundary; Step 7 demonstrates that the plugin still reconciles filtered GPUs. Render before changing the cluster: @@ -595,11 +599,15 @@ until ! nvidia-smi -L | grep -q '^ MIG '; do done ``` -Create a second values file that removes `$SECONDARY_GPU` from the exclusion list. In the verified run this changed the list from `[0, 1, 2, 3, 5, 6, 7]` to `[0, 1, 2, 3, 6, 7]`, registering GPUs 4 and 5. The `grep` output must show the shorter list; if it still matches the one-GPU file, `SECONDARY_GPU` was not in the list. +Render a second values file whose exclusion list omits both `$PRIMARY_GPU` and `$SECONDARY_GPU`. In the verified run this changed the list from `[0, 1, 2, 3, 5, 6, 7]` to `[0, 1, 2, 3, 6, 7]`, registering GPUs 4 and 5. ```bash -sed -E "/\"index\":/ { s/(\[|, )${SECONDARY_GPU}, /\1/; s/, ${SECONDARY_GPU}\]/]/; }" \ - "$LAB/hami-values-one-gpu.yaml" > "$LAB/hami-values-two-gpus.yaml" +TWO_GPU_EXCLUDES=$(nvidia-smi --query-gpu=index --format=csv,noheader | tr -d ' ' | + grep -vx -e "$PRIMARY_GPU" -e "$SECONDARY_GPU" | paste -sd ',' - | sed 's/,/, /g') + +sed -e "s/__NODE_NAME__/${NODE}/g" \ + -e "s/__EXCLUDED_GPU_INDICES__/${TWO_GPU_EXCLUDES}/" \ + "$EXAMPLES/hami-values.yaml" > "$LAB/hami-values-two-gpus.yaml" grep -n '"index"' "$LAB/hami-values-two-gpus.yaml" helm upgrade hami "$LAB/HAMi/charts/hami" \ diff --git a/tutorials/labs/examples/15-dynamic-mig-rtx-pro/hami-values.yaml b/tutorials/labs/examples/15-dynamic-mig-rtx-pro/hami-values.yaml index 9d2efe1ad..017041cdf 100644 --- a/tutorials/labs/examples/15-dynamic-mig-rtx-pro/hami-values.yaml +++ b/tutorials/labs/examples/15-dynamic-mig-rtx-pro/hami-values.yaml @@ -29,6 +29,9 @@ devicePlugin: # per-node Dynamic MIG operating mode below. migStrategy: none + # Steps 3 and 8 of the lab fill in the node name and the excluded GPU index + # list below before rendering. The verified run excluded [0, 1, 2, 3, 5, 6, 7], + # which registers only GPU 4. nodeConfiguration: config: | { @@ -44,7 +47,7 @@ devicePlugin: "migstrategy": "none", "filterdevices": { "uuid": [], - "index": [0, 1, 2, 3, 5, 6, 7] + "index": [__EXCLUDED_GPU_INDICES__] }, "enablegetpreferredallocation": false } From 47ff08bdadd060038c7fede8c7357a6e7c698f00 Mon Sep 17 00:00:00 2001 From: Saiyam Pathak Date: Thu, 3 Sep 2026 17:56:19 +0530 Subject: [PATCH 5/6] docs(tutorials): renumber the RTX PRO 6000 Dynamic MIG lab to Lab 17 #654 merged as Lab 15 and #655 was queued first for Lab 16, so this lab takes 17. Rename the examples directory, update the titles, sidebar labels, and paths in both locales, and list the lab in the overview notes instead of appending text to the card grid line. Signed-off-by: Saiyam Pathak --- .../current/labs/dynamic-mig-rtx-pro.md | 4 ++-- .../current/overview.md | 3 ++- tutorials/labs/dynamic-mig-rtx-pro.md | 8 ++++---- .../create-mixed-pods.sh | 0 .../hami-values.yaml | 0 .../mig-small-pack.yaml | 0 tutorials/overview.md | 3 ++- 7 files changed, 10 insertions(+), 8 deletions(-) rename tutorials/labs/examples/{15-dynamic-mig-rtx-pro => 17-dynamic-mig-rtx-pro}/create-mixed-pods.sh (100%) rename tutorials/labs/examples/{15-dynamic-mig-rtx-pro => 17-dynamic-mig-rtx-pro}/hami-values.yaml (100%) rename tutorials/labs/examples/{15-dynamic-mig-rtx-pro => 17-dynamic-mig-rtx-pro}/mig-small-pack.yaml (100%) diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/dynamic-mig-rtx-pro.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/dynamic-mig-rtx-pro.md index a10bdacac..567ce28cc 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/dynamic-mig-rtx-pro.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/dynamic-mig-rtx-pro.md @@ -1,7 +1,7 @@ --- -title: "实验 15: RTX PRO 6000 动态 MIG 生命周期" +title: "实验 17: RTX PRO 6000 动态 MIG 生命周期" description: "构建固定版本的 HAMi，并验证按 Pod 创建 MIG、混合规格、选择性回收、重启恢复和跨 GPU 调度。" -sidebar_label: "实验 15: 动态 MIG 生命周期" +sidebar_label: "实验 17: 动态 MIG 生命周期" lab: level: Advanced duration: 约 90 分钟 diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md index 5d0fbfacf..2dee09a09 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md @@ -17,7 +17,7 @@ import LabCardGridAuto from '@site/src/components/labs/LabCardGridAuto'; ## 实验 -实验 15 构建固定提交版本的 HAMi，并在 RTX PRO 6000 硬件上验证完整的 Dynamic MIG 生命周期。 + 每个实验都列出了各自的前提条件。 @@ -30,3 +30,4 @@ import LabCardGridAuto from '@site/src/components/labs/LabCardGridAuto'; - **实验 12** 在 GKE 1.35/COS/CDI 上部署 KAI Scheduler 与 HAMi-core，并通过 CUDA 分配验证显存上限。 - **实验 13** 在昇腾 310P3 ARM 服务器上源码编译 Volcano 与 ascend-device-plugin，验证 hami-vnpu-core 软切分、binpack 共卡与容器级监控指标。 - **实验 14** 在挂载四块 T4 的 GKE 节点上安装 HAMi v2.10.0，并通过分配注解与调度器日志观察可组合的 `gpu-scheduler-policy` 策略链（`spread`、`binpack`、`mutex`、`mutex,binpack`）。 +- **实验 17** 在配备 8 张 RTX PRO 6000 的服务器上构建固定提交版本的 HAMi，并验证完整的 Dynamic MIG 生命周期：按 Pod 创建 MIG、混合规格、选择性回收、device plugin 重启恢复以及跨 GPU 调度。 diff --git a/tutorials/labs/dynamic-mig-rtx-pro.md b/tutorials/labs/dynamic-mig-rtx-pro.md index 2dee178ef..27313173b 100644 --- a/tutorials/labs/dynamic-mig-rtx-pro.md +++ b/tutorials/labs/dynamic-mig-rtx-pro.md @@ -1,7 +1,7 @@ --- -title: "Lab 15: Dynamic MIG Lifecycle on RTX PRO 6000" +title: "Lab 17: Dynamic MIG Lifecycle on RTX PRO 6000" description: "Build a pinned HAMi snapshot and verify per-Pod MIG placement, mixed profiles, selective reclamation, restart recovery, and multi-GPU spillover." -sidebar_label: "Lab 15: Dynamic MIG Lifecycle" +sidebar_label: "Lab 17: Dynamic MIG Lifecycle" lab: level: Advanced duration: about 90 minutes @@ -70,7 +70,7 @@ You also need: - root access to the GPU node, working `nvidia-smi`, MIG mode enabled, and no unmanaged CUDA processes; - `git`, GNU Make, Docker, `ctr`, Helm, `kubectl`, and `jq`; - cluster-admin access and permission to replace the existing HAMi installation; -- a local checkout of this website repository for the files under [`tutorials/labs/examples/15-dynamic-mig-rtx-pro/`](https://github.com/Project-HAMi/website/tree/master/tutorials/labs/examples/15-dynamic-mig-rtx-pro); and +- a local checkout of this website repository for the files under [`tutorials/labs/examples/17-dynamic-mig-rtx-pro/`](https://github.com/Project-HAMi/website/tree/master/tutorials/labs/examples/17-dynamic-mig-rtx-pro); and - an explicit maintenance window for the **whole GPU node**, not only the GPUs that HAMi will register. The supplied values target the verified eight-GPU node and initially register only GPU index 4. If your topology differs, choose your own primary and spillover GPU indices in Step 1; Steps 3 and 8 derive the `filterdevices.index` exclusion lists from those choices and the node's GPU inventory. You need at least two compatible GPUs to reproduce Step 8. @@ -94,7 +94,7 @@ export NODE=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}') export PRIMARY_GPU=4 # the only GPU registered with HAMi until Step 8 export SECONDARY_GPU=5 # the spillover GPU added in Step 8 export LAB=/root/hami-dynamic-mig-rerun-2026-08-11 -export EXAMPLES=tutorials/labs/examples/15-dynamic-mig-rtx-pro +export EXAMPLES=tutorials/labs/examples/17-dynamic-mig-rtx-pro mkdir -p "$LAB" ``` diff --git a/tutorials/labs/examples/15-dynamic-mig-rtx-pro/create-mixed-pods.sh b/tutorials/labs/examples/17-dynamic-mig-rtx-pro/create-mixed-pods.sh similarity index 100% rename from tutorials/labs/examples/15-dynamic-mig-rtx-pro/create-mixed-pods.sh rename to tutorials/labs/examples/17-dynamic-mig-rtx-pro/create-mixed-pods.sh diff --git a/tutorials/labs/examples/15-dynamic-mig-rtx-pro/hami-values.yaml b/tutorials/labs/examples/17-dynamic-mig-rtx-pro/hami-values.yaml similarity index 100% rename from tutorials/labs/examples/15-dynamic-mig-rtx-pro/hami-values.yaml rename to tutorials/labs/examples/17-dynamic-mig-rtx-pro/hami-values.yaml diff --git a/tutorials/labs/examples/15-dynamic-mig-rtx-pro/mig-small-pack.yaml b/tutorials/labs/examples/17-dynamic-mig-rtx-pro/mig-small-pack.yaml similarity index 100% rename from tutorials/labs/examples/15-dynamic-mig-rtx-pro/mig-small-pack.yaml rename to tutorials/labs/examples/17-dynamic-mig-rtx-pro/mig-small-pack.yaml diff --git a/tutorials/overview.md b/tutorials/overview.md index 555dcf410..906ff98f8 100644 --- a/tutorials/overview.md +++ b/tutorials/overview.md @@ -17,7 +17,7 @@ Background knowledge that the labs build on. ## Labs - Lab 15 builds a pinned HAMi snapshot and verifies the complete Dynamic MIG lifecycle on RTX PRO 6000 hardware. + Each lab lists its own prerequisites. @@ -30,3 +30,4 @@ Each lab lists its own prerequisites. - **Lab 12** deploys KAI Scheduler and HAMi-core on GKE 1.35/COS/CDI and proves the memory ceiling with CUDA allocations. - **Lab 13** builds Volcano and the ascend-device-plugin from source on an Ascend 310P3 ARM server and verifies hami-vnpu-core soft slicing, binpack card sharing, and per-container metrics. - **Lab 14** installs HAMi v2.10.0 on a four-T4 GKE node and observes the composable `gpu-scheduler-policy` chains (`spread`, `binpack`, `mutex`, `mutex,binpack`) through allocation annotations and scheduler logs. +- **Lab 17** builds a pinned HAMi snapshot on an eight-GPU RTX PRO 6000 server and verifies the complete Dynamic MIG lifecycle: per-Pod placement, mixed profiles, selective reclamation, device-plugin restart recovery, and spillover to a second GPU. From e2519a3e62669f225c221389eea5ed93767f56f7 Mon Sep 17 00:00:00 2001 From: Saiyam Pathak Date: Mon, 7 Sep 2026 15:00:02 +0530 Subject: [PATCH 6/6] docs(tutorials): group the RTX PRO 6000 lab overview into three phases Stack the eight steps into three labeled phases (prepare, allocate, recover and spill) instead of one long row, so the step boxes are readable at normal page width. Signed-off-by: Saiyam Pathak --- tutorials/labs/dynamic-mig-rtx-pro.md | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/tutorials/labs/dynamic-mig-rtx-pro.md b/tutorials/labs/dynamic-mig-rtx-pro.md index 27313173b..e6b3c729d 100644 --- a/tutorials/labs/dynamic-mig-rtx-pro.md +++ b/tutorials/labs/dynamic-mig-rtx-pro.md @@ -41,14 +41,20 @@ The commands and outputs were captured on 2026-08-11 from the [original verified ```mermaid %% title: Dynamic MIG Lifecycle Lab Flow -flowchart LR - S1["Step 1
Inventory and handover"] --> S2["Step 2
Build pinned source"] - S2 --> S3["Step 3
Render and install"] - S3 --> S4["Step 4
Create one 1g"] - S4 --> S5["Step 5
Saturate four placements"] - S5 --> S6["Step 6
Mix and reclaim"] - S6 --> S7["Step 7
Restart and adopt"] - S7 --> S8["Step 8
Spill to GPU 5"] +flowchart TB + subgraph P1["Phase 1: Prepare the node"] + direction LR + S1["Step 1
Inventory and handover"] --> S2["Step 2
Build pinned source"] --> S3["Step 3
Render and install"] + end + subgraph P2["Phase 2: Allocate MIG per Pod"] + direction LR + S4["Step 4
Create one 1g instance"] --> S5["Step 5
Saturate four placements"] --> S6["Step 6
Mix profiles and reclaim one"] + end + subgraph P3["Phase 3: Prove recovery and spillover"] + direction LR + S7["Step 7
Restart plugin, adopt live instance"] --> S8["Step 8
Register GPU 5, spill fifth Pod"] + end + P1 --> P2 --> P3 ``` ## Prerequisites