Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/drivers/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -281,7 +281,7 @@ To check the full spec of `DeviceConfig` definition run `kubectl get crds device
|---------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|
| `enable` | Set to true for installing out-of-tree driver.<br>Set to false to skip driver install and use inbox/pre-installed driver. | `true` |
| `blacklist` | Set to true to have the operator init the node labeller DaemonSet and add `amdgpu` to the selected worker nodes' modprobe blacklist.<br>Set to false to remove `amdgpu` from the selected nodes' modprobe blacklist. | `false` |
| `version` | ROCm driver version (e.g., "6.2.2").<br>See ROCm Versions: https://rocm.docs.amd.com/en/latest/release/versions.html | Ubuntu: `6.1.3`<br>CoreOS: `6.2.2` |
| `version` | amdgpu driver version (e.g., "6.2.2").<br>See amdgpu Versions: https://instinct.docs.amd.com/projects/amdgpu-docs/en/latest/release/versions.html | Ubuntu: `6.1.3`<br>CoreOS: `6.2.2` |
| `image` | Registry URL and repository (without tag).<br>Note: Operator manages tags automatically. | Vanilla k8s: `image-registry:5000/$MOD_NAMESPACE/amdgpu_kmod`<br>OpenShift: `image-registry.openshift-image-registry.svc:5000/$MOD_NAMESPACE/amdgpu_kmod` |
| `imageRegistrySecret.name` | Name of registry credentials secret to pull/push driver image. | |
| `imageRegistryTLS.insecure` | If true, check if the container image already exists using plain HTTP. | `false` |
Expand Down
2 changes: 1 addition & 1 deletion docs/drivers/precompiled-driver.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Users could prepare pre-compiled driver images in advance and import them into t
KMM determines the appropriate driver image based on the combination of:

1. Worker node OS information
2. Requested ROCm driver version
2. Requested amdgpu driver version

### Image Tag Format

Expand Down
2 changes: 1 addition & 1 deletion docs/drivers/upgrading.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ To check the full spec of upgrade configuration run kubectl get crds deviceconfi
| `maxUnavailableNodes` | Maximum number (or Percentage) of nodes which can be unavailable (cordoned) in the cluster | `25%` |
| `rebootRequired` | Reboot the node after driver upgrade is done. Waits for 60 mins post reboot before declaring as failed | `true` |

**Warning**: When using ROCm drivers version 6.3 and below, a known issue may prevent the driver upgrade from fully completing unless the node is rebooted. As a workaround, we strongly recommend setting the `rebootRequired` field to `true` in your upgrade policy. This ensures that a reboot is triggered after the driver upgrade, allowing the new driver to be fully loaded. This workaround should be applied until a permanent fix is provided in a future release.
**Warning**: When using amdgpu driver versions 6.3 and below, a known issue may prevent the driver upgrade from fully completing unless the node is rebooted. As a workaround, we strongly recommend setting the `rebootRequired` field to `true` in your upgrade policy. This ensures that a reboot is triggered after the driver upgrade, allowing the new driver to be fully loaded. This workaround should be applied until a permanent fix is provided in a future release.

#### `driver.upgradePolicy.nodeDrainPolicy` Parameters

Expand Down
4 changes: 2 additions & 2 deletions docs/knownlimitations.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,11 +99,11 @@
- *Root Cause:* DCM does not currently support standalone installation via a Debian package like other standalone components of the GPU Operator. We will be adding a Debian package to support standalone bare metal installations in the next release of DCM.
- *Recommendation:* Those wishing to use GPU partitioning in a bare metal environment should instead use the standalone docker image for DCM. Alternatively users can use amd-smi to change partitioning modes. See [amdgpu-docs documentation](https://instinct.docs.amd.com/projects/amdgpu-docs/en/latest/gpu-partitioning/mi300x/quick-start-guide.html) for how to do this.

16. **The GPU Operator will report an error when ROCm driver install version doesn't match the version string in the [Radeon Repo](https://repo.radeon.com/rocm/apt/).**
16. **The GPU Operator will report an error when amdgpu driver install version doesn't match the version string in the [Radeon Repo](https://repo.radeon.com/rocm/apt/).**

- *Impact:* The DeviceConfig will report an error if you specify `"6.4.0"` or `"6.3.0"` for the `spec.driver.version`.
- *Root Cause:* The version specified in the CR would still have to match the version string on Radeon repo.
- *Recommendation:* Although this will be fixed in a future version of the GPU Operator, for the time being you will instead need to specific `"6.4"` or `"6.3"` when installing those versions of the ROCm amdgpu driver.
- *Recommendation:* Although this will be fixed in a future version of the GPU Operator, for the time being you will instead need to specify `"6.4"` or `"6.3"` when installing those versions of the ROCm amdgpu driver.

17. **Device plugin crashes when unable to establish watcher on kubelet device plugin path**

Expand Down
4 changes: 2 additions & 2 deletions docs/releasenotes.md
Original file line number Diff line number Diff line change
Expand Up @@ -346,11 +346,11 @@ The AMD GPU Operator v1.3.0 release introduces new features, most notably of whi
- *Root Cause:* DCM does not currently support standalone installation via a Debian package like other standalone components of the GPU Operator. We will be adding a Debian package to support standalone bare metal installations in the next release of DCM.
- *Recommendation:* Those wishing to use GPU partitioning in a bare metal environment should instead use the standalone docker image for DCM. Alternatively users can use amd-smi to change partitioning modes. See [amdgpu-docs documentation](https://instinct.docs.amd.com/projects/amdgpu-docs/en/latest/gpu-partitioning/mi300x/quick-start-guide.html) for how to do this.

2. **The GPU Operator will report an error when ROCm driver install version doesn't match the version string in the [Radeon Repo](https://repo.radeon.com/rocm/apt/).**
2. **The GPU Operator will report an error when amdgpu driver install version doesn't match the version string in the [Radeon Repo](https://repo.radeon.com/rocm/apt/).**

- *Impact:* The DeviceConfig will report an error if you specify `"6.4.0"` or `"6.3.0"` for the `spec.driver.version`.
- *Root Cause:* The version specified in the CR would still have to match the version string on Radeon repo.
- *Recommendation:* Although this will be fixed in a future version of the GPU Operator, for the time being you will instead need to specific `"6.4"` or `"6.3"` when installing those versions of the ROCm amdgpu driver.
- *Recommendation:* Although this will be fixed in a future version of the GPU Operator, for the time being you will instead need to specify `"6.4"` or `"6.3"` when installing those versions of the ROCm amdgpu driver.

> **Note:** All current and historical limitations for the GPU Operator, including their latest statuses and any associated workarounds or fixes, are tracked in the following documentation page: [Known Issues and Limitations](https://instinct.docs.amd.com/projects/gpu-operator/en/latest/knownlimitations.html).
Please refer to this page regularly for the most up-to-date information.
Expand Down
Loading