diff --git a/README.md b/README.md index 037d62a..e95efdf 100644 --- a/README.md +++ b/README.md @@ -70,6 +70,14 @@ We aim to provide a dynamic resource where users can find the latest optimizatio - [PMU](hardware/PMU/README.md) - [Priority Core Turbo](hardware/priority_core_turbo/README.md) - [Scaling](hardware/scaling/README.md) +- Kubernetes + - [Kubernetes Optimization](kubernetes/README.md) + - [NRI Resource-Policy Plugins](kubernetes/nri-resource-policies/README.md) + - [Balloons Policy](kubernetes/nri-resource-policies/balloons.md) + - [Topology-Aware Policy](kubernetes/nri-resource-policies/topology-aware.md) + - [Choosing a Policy](kubernetes/nri-resource-policies/choosing-a-policy.md) + - [Choosing a Policy with the Kubernetes Managers](kubernetes/nri-resource-policies/choosing-a-policy-with-kubernetes-managers.md) + - [Quick Start](kubernetes/nri-resource-policies/quickstart.md) ## Contributing diff --git a/hardware/NUMA/README.md b/hardware/NUMA/README.md index 935630a..ab53867 100644 --- a/hardware/NUMA/README.md +++ b/hardware/NUMA/README.md @@ -4,6 +4,8 @@ Non-Uniform Memory Access (NUMA) is a memory architecture used in modern multi-s This directory collects best practices, case studies, and performance opportunities for running workloads in NUMA environments. The goal is to help you understand why hardware topology matters, identify configuration mismatches that leave performance on the table, and apply proven, hardware-aware patterns to your deployments. +> **Running in Kubernetes?** The Topology-aware policy applies NUMA-aligned placement to every pod automatically. See [Kubernetes Optimization](../../kubernetes/nri-resource-policies/topology-aware.md). + Actual improvements will vary depending on a given workload's characteristics. It is recommended to use a profiling solution such as [VTune Profiler](https://github.com/intel/optimization-zone/tree/main/tools/vtune/README.md) to more accurately gauge where an application's hotpaths are and which optimizations would have the greatest effect. ## What's Inside diff --git a/hardware/priority_core_turbo/README.md b/hardware/priority_core_turbo/README.md index 60f4f95..db18481 100644 --- a/hardware/priority_core_turbo/README.md +++ b/hardware/priority_core_turbo/README.md @@ -11,6 +11,8 @@ Running these threads on **High-Priority (HP) cores** improves GPU utilization, This guide provides a Docker-based workflow and supporting scripts to check PCT support, configure CLOS assignments, and validate the selected high-priority CPU list. The PCT setup and verification steps run from the container, so users do not need to install `intel-speed-select` or other extra host-side dependencies beyond Docker. +> **Running in Kubernetes?** To assign PCT high-priority cores to specific pods, use the Balloons policy. See [Kubernetes Optimization](../../kubernetes/nri-resource-policies/balloons.md). + Validated platforms: - **Intel® Xeon® 6776P** diff --git a/kubernetes/README.md b/kubernetes/README.md new file mode 100644 index 0000000..5604629 --- /dev/null +++ b/kubernetes/README.md @@ -0,0 +1,81 @@ +# Kubernetes Optimization + +Most tuning guides in the Optimization Zone assume you control the host: you +pin threads, set the CPU governor, disable deep C-states, or align a workload +to a NUMA node. In Kubernetes you usually cannot do that by hand. The scheduler +places pods on nodes, and the built-in CPU and Memory managers know little +about NUMA distance, cache domains, device locality, Priority Core Turbo (PCT), +uncore and C-states, or real-time scheduling. + +This section is about closing that gap. It shows how to apply the same +hardware-aware optimizations to containerized workloads, on both cloud and edge +nodes, without editing each node by hand. + +## The mechanism: NRI resource-policy plugins + +The [NRI (Node Resource Interface) resource-policy plugins](https://github.com/containers/nri-plugins) +run as a DaemonSet on each node. They hook into the container runtime +(containerd or CRI-O) and set CPU, memory, and device affinity for each +container as it starts. They can also tune CPU frequency, C-states, IRQ +affinity, and process scheduling per group of containers. + +Two policies cover most needs: + +- **[Topology-aware](nri-resource-policies/topology-aware.md)** — automatic + NUMA-, cache-, and device-aligned placement. No per-workload configuration + needed. Start here if you want the wins from the [NUMA guide](../hardware/NUMA/README.md) + without hand-pinning. +- **[Balloons](nri-resource-policies/balloons.md)** — group containers into CPU + pools ("balloons") and tune each pool: dedicated CPUs, frequency, C-states, + PCT high-priority cores, real-time scheduling, device locality. Choose this + when you need explicit control per group of workloads. + +Not sure which fits? See [choosing a policy](nri-resource-policies/choosing-a-policy.md), +or, if you already use the built-in Kubernetes managers, +[choosing a policy with the Kubernetes managers](nri-resource-policies/choosing-a-policy-with-kubernetes-managers.md). + +New here? Start with the [foundation page](nri-resource-policies/README.md), or +jump to the [quick start](nri-resource-policies/quickstart.md). + +## What maps to what + +If you came from a hardware or software guide, this is where its host-level +lever shows up in Kubernetes: + +| Optimization Zone topic | NRI policy feature | +| --- | --- | +| [NUMA alignment](../hardware/NUMA/README.md) | Topology-aware automatic alignment; Balloons topology balancing | +| Memory bandwidth vs. latency trade-off | Balloons spreading across NUMA nodes and sockets, or local-only | +| Multi-tier memory (DRAM, HBM, PMEM) | Topology-aware multi-tier allocation | +| [Priority Core Turbo](../hardware/priority_core_turbo/README.md) | Balloons CPU classes, any QoS; Topology-aware CPU classes, Guaranteed containers only | +| [CPU frequency, EPP, governor](../software/common/README.md) | Balloons CPU classes: min/max frequency, governor, EPP, uncore | +| C-states and wakeup latency | Balloons CPU classes: disabled C-states | +| [Hyper-threading choices](../software/common/README.md) | Topology-aware HT control; Balloons hyper-thread-aware sharing | +| Dedicated cores / isolation | Balloons dedicated pools; Topology-aware exclusive CPUs and isolcpus | +| Burst headroom without noisy neighbors | Balloons idle-CPU sharing | +| Real-time scheduling and I/O priority | Balloons scheduling classes (SCHED_FIFO, ioClass, ioPriority) | +| IRQ affinity | Balloons IRQ affinity, any QoS; Topology-aware IRQ affinity, Guaranteed containers only | +| PCI / GPU / NIC / accelerator locality | Balloons device locality; Topology-aware device alignment | + +Each feature is documented upstream. The pages in this section explain when to +reach for it and link to the details. + +## Who this is for + +- **AI and inference** — put host-side serial work (tokenization, dispatch) on + PCT high-priority cores and keep it local to the GPU or NIC. +- **Databases** — align CPU and memory to cut cross-NUMA traffic and tail + latency; isolate from noisy neighbors. +- **Web services and proxies** — let bursty handlers use idle CPUs without + hurting a latency-critical neighbor. +- **Real-time and industrial / edge** — dedicated cores, no deep C-states, + real-time scheduling, and IRQs steered off the control loop. + +## Measure your results + +Improvements depend on the workload. Use a profiler such as +[VTune Profiler](../tools/vtune/README.md) or [PerfSpect](../tools/perfspect/README.md) +to see where time goes and to compare before and after. + +> Performance varies by use, configuration, and other factors. See the +> [disclaimer](../README.md#disclaimer). diff --git a/kubernetes/nri-resource-policies/README.md b/kubernetes/nri-resource-policies/README.md new file mode 100644 index 0000000..e1c9d63 --- /dev/null +++ b/kubernetes/nri-resource-policies/README.md @@ -0,0 +1,81 @@ +# NRI Resource-Policy Plugins + +The NRI resource-policy plugins apply hardware-aware placement to containers in +Kubernetes. This page covers what they are and how to get one running. The +policy pages ([Balloons](balloons.md), [Topology-aware](topology-aware.md)) +cover when and why to use each. + +## What is NRI + +[NRI (Node Resource Interface)](https://containers.github.io/nri-plugins/stable/docs/resource-policy/introduction.html) +is a plugin interface in the container runtime. A resource-policy plugin +receives each container's lifecycle events and decides its CPU, memory, and +device affinity before it starts. This happens on the node, below the +Kubernetes scheduler. + +## How it runs + +- One plugin runs as a **DaemonSet**, one pod per node. +- It talks to the container runtime over NRI: **containerd 1.7+** or + **CRI-O 1.26+**, with NRI enabled (the default in current containerd). +- You run **one policy per node**. Balloons and Topology-aware are + alternatives, not layers. + +## Prerequisites + +- Kubernetes 1.24 or newer (1.27+ for the PCT flow). +- A container runtime with NRI enabled: containerd 1.7+ or CRI-O 1.26+. +- Helm 3+. + +See the upstream [setup guide](https://containers.github.io/nri-plugins/stable/docs/resource-policy/setup.html) +for enabling NRI and for the full install procedure. + +## Install + +Add the Helm repository: + +```bash +helm repo add nri-plugins https://containers.github.io/nri-plugins +helm repo update +``` + +Install one policy. For Topology-aware: + +```bash +helm install nri-resource-policy-topology-aware \ + nri-plugins/nri-resource-policy-topology-aware --namespace kube-system +``` + +For Balloons: + +```bash +helm install nri-resource-policy-balloons \ + nri-plugins/nri-resource-policy-balloons --namespace kube-system +``` + +Check the DaemonSet is ready: + +```bash +kubectl -n kube-system rollout status ds/nri-resource-policy-topology-aware +``` + +## Configuration + +You configure a policy with a custom resource, not Helm values, so changes +apply without reinstalling. Each policy has its own kind +(`BalloonsPolicy`, `TopologyAwarePolicy`). Configuration works at three scopes, +most specific wins: + +- **default** — all nodes without a more specific config. +- **group** — nodes labeled `config.nri/group=$NAME`. +- **node** — a single named node. + +This lets one cluster serve mixed hardware and mixed workloads. See the +upstream [configuration guide](https://containers.github.io/nri-plugins/stable/docs/resource-policy/configuration.html). + +## Scope of these pages + +This section orients you and points to the details. For install options, the +full configuration schema, and cookbooks, the upstream +[NRI plugins documentation](https://containers.github.io/nri-plugins/stable/) +is the source of truth. diff --git a/kubernetes/nri-resource-policies/balloons.md b/kubernetes/nri-resource-policies/balloons.md new file mode 100644 index 0000000..b1e7c9d --- /dev/null +++ b/kubernetes/nri-resource-policies/balloons.md @@ -0,0 +1,73 @@ +# Balloons Policy + +The Balloons policy groups containers into CPU pools called *balloons*. A +container runs only on the CPUs of its balloon. Each balloon can be tuned on +its own: how many CPUs, at what frequency, which C-states, how they are +scheduled, and which devices they sit close to. + +Reach for Balloons when you need explicit control over how groups of workloads +share a node. For automatic alignment with no per-workload setup, see +[Topology-aware](topology-aware.md) instead. + +A balloon can hold containers of any QoS class: BestEffort, Burstable, or +Guaranteed. Its tuning (frequency, C-states, PCT, IRQ affinity, scheduling) +applies to all of them. This is a key difference from Topology-aware, whose CPU +tuning, PCT, and IRQ affinity apply only to Guaranteed containers with exclusive +CPUs. + +## What it solves + +Each item below is a common host-level tuning task and where Balloons handles +it in Kubernetes. Follow the links for the configuration. + +- **Isolate a latency-critical workload.** Give it dedicated CPUs, stop it + sharing L2 cache with other workloads, and disable deep C-states so cores do + not sleep. See the upstream + [latency-critical recipe](https://containers.github.io/nri-plugins/stable/docs/resource-policy/policy/balloons.html#latency-critical-containers). + +- **Run hot serial threads at top turbo.** Put them on Priority Core Turbo + high-priority cores while the rest of the node stays at base frequency. See + the [PCT guide](../../hardware/priority_core_turbo/README.md) for the hardware + side and the upstream + [PCT quick start](https://containers.github.io/nri-plugins/stable/docs/resource-policy/policy/howto/balloons-pct-quickstart.html) + for the Kubernetes side. + +- **Get more memory bandwidth.** Spread a balloon's CPUs across NUMA nodes or + sockets to use more memory channels, or keep it local for lowest latency. See + the upstream + [memory bandwidth recipe](https://containers.github.io/nri-plugins/stable/docs/resource-policy/policy/balloons.html#maximum-memory-bandwidth-containers) + and the [NUMA guide](../../hardware/NUMA/README.md). + +- **Let bursty workloads use idle CPUs safely.** A balloon can borrow otherwise + idle CPUs within a core, L2 domain, NUMA node, or package, and you control + which workload types share a physical core. See the upstream + [hyper-thread sharing recipe](https://containers.github.io/nri-plugins/stable/docs/resource-policy/policy/balloons.html#workload-aware-hyperthread-sharing). + +- **Set frequency, governor, EPP, and C-states per group.** The host-wide knobs + in the [common recommendations](../../software/common/README.md) become + per-balloon CPU classes. See the upstream + [CPU tuning options](https://containers.github.io/nri-plugins/stable/docs/resource-policy/policy/balloons.html#cpu-tuning). + +- **Apply real-time scheduling and I/O priority.** Assign a balloon a scheduling + class such as `SCHED_FIFO` with a priority, plus an I/O class and priority. + Useful for control loops and other real-time work. See the upstream + [CPU tuning options](https://containers.github.io/nri-plugins/stable/docs/resource-policy/policy/balloons.html#cpu-tuning). + +- **Steer IRQs off critical cores.** Move device interrupts away from the CPUs + running a latency-critical balloon. + +- **Keep CPUs close to a device.** Place a balloon on the CPUs nearest a GPU, + NIC, or accelerator to cut access latency. Useful for AI inference and packet + processing. + +## Try it + +The upstream [PCT quick start](https://containers.github.io/nri-plugins/stable/docs/resource-policy/policy/howto/balloons-pct-quickstart.html) +is the fastest hands-on path and shows a measurable difference between tuned and +untuned pods. For a local isolation example, see the +[quick start](quickstart.md) in this section. + +## Reference + +Full configuration, all balloon and CPU-class fields, and more recipes are in +the upstream [Balloons documentation](https://containers.github.io/nri-plugins/stable/docs/resource-policy/policy/balloons.html). diff --git a/kubernetes/nri-resource-policies/choosing-a-policy-with-kubernetes-managers.md b/kubernetes/nri-resource-policies/choosing-a-policy-with-kubernetes-managers.md new file mode 100644 index 0000000..38826a3 --- /dev/null +++ b/kubernetes/nri-resource-policies/choosing-a-policy-with-kubernetes-managers.md @@ -0,0 +1,62 @@ +# Choosing a Policy, Including the Built-in Kubernetes Managers + +Kubernetes ships its own CPU manager, Memory manager, Device manager, and +Topology manager. They pin CPUs and memory and try to keep a pod's resources on +one NUMA node. If you already use them, this page shows where the NRI policies +fit and what they add. + +For the choice between the two NRI policies alone, see +[choosing a policy](choosing-a-policy.md). + +## Three options + +- **Built-in Kubernetes managers** — CPU manager (`static` policy), Memory + manager, Device manager, and Topology manager working together. +- **[Topology-aware](topology-aware.md)** — an NRI policy that aligns CPU and + memory automatically. +- **[Balloons](balloons.md)** — an NRI policy built around CPU pools you define. + +## Topology-aware vs. the built-in managers + +Topology-aware is close to a drop-in replacement for the built-in CPU and +Memory managers. The intent is the same: place a workload's CPU and memory +together on aligned hardware. The differences are in the allocation: + +- **Smarter allocation.** It builds a pool tree from the real topology and + scores candidates, rather than following the managers' fixed rules. +- **Scales further.** The built-in managers already struggle to align well + around 8 NUMA nodes. Topology-aware keeps working at 8 nodes and beyond. +- **More tuning for Guaranteed containers.** It can also set CPU frequency, + C-states, PCT, and IRQ affinity for Guaranteed containers with exclusive + CPUs. The built-in managers do not offer these. + +If your goal is aligned placement and you find the built-in managers limiting on +larger systems, Topology-aware is the natural step up. + +## Balloons vs. the built-in managers + +Balloons has different semantics. Instead of per-pod pinning rules, you +partition the node into CPU pools and assign containers and pods to them: + +- **Group and mix freely.** Put containers from the same or different pods, + applications, or namespaces into the same pool, or keep them apart. This + grouping is more flexible than the built-in managers allow. +- **More to tune, for any QoS class.** Each pool can set CPU frequency, + C-states, EPP, PCT high-priority cores, real-time scheduling, I/O priority, + IRQ affinity, and device locality. This tuning applies to any container in a + pool, not just Guaranteed ones. The built-in managers do not offer these, and + Topology-aware applies the CPU, PCT, and IRQ tuning only to Guaranteed + containers. + +Choose Balloons when you need that grouping flexibility or the extra tuning. + +## Which to use + +| Situation | Use | +| --- | --- | +| Aligned CPU/memory, small NUMA count, happy with built-ins | Built-in managers | +| Aligned CPU/memory, larger NUMA count or want smarter allocation | Topology-aware | +| Need CPU pools, per-group tuning, PCT, or real-time control | Balloons | + +The NRI policies are alternatives to the built-in managers for these tasks, not +additions on top. Run one approach per node. diff --git a/kubernetes/nri-resource-policies/choosing-a-policy.md b/kubernetes/nri-resource-policies/choosing-a-policy.md new file mode 100644 index 0000000..8ce6afe --- /dev/null +++ b/kubernetes/nri-resource-policies/choosing-a-policy.md @@ -0,0 +1,41 @@ +# Choosing a Policy + +Balloons and Topology-aware are alternatives. You run one policy per node, so +pick the one that matches how much control you need. If you already use the +built-in Kubernetes CPU and Memory managers, see +[choosing a policy with the Kubernetes managers](choosing-a-policy-with-kubernetes-managers.md). + +## Start here + +Use **[Topology-aware](topology-aware.md)** unless you have a specific reason +not to. It aligns CPU and memory to the hardware for every pod with no +per-workload configuration. + +Switch to **[Balloons](balloons.md)** when you need any of these: + +- dedicated CPU pools for groups of containers or pods, +- CPU frequency, C-states, EPP, PCT, or IRQ tuning for **any** QoS class, not + just Guaranteed containers, +- real-time scheduling or I/O priority, +- grouping containers from different pods or namespaces into one pool, +- explicit control over which workloads share physical cores. + +Topology-aware also supports CPU power, PCT, and IRQ tuning, but only for +Guaranteed containers with exclusive CPUs. Balloons applies the same tuning to +any container in a balloon. + +## Side by side + +| | Topology-aware | Balloons | +| --- | --- | --- | +| Main idea | Automatic hardware-aligned placement | CPU pools you define and tune | +| Setup effort | None per workload | You define balloon types | +| CPU/memory alignment | Yes, automatic | Yes, configurable | +| CPU power, PCT, IRQ tuning | Guaranteed (exclusive-CPU) containers only | Any container in a balloon | +| Grouping / mixing workloads | Per-pod alignment | Group any pods or containers into shared pools | +| Best for | Most workloads, the default | Explicit control and mixed workloads on a node | + +## Rule of thumb + +Do you need per-pool tuning for non-Guaranteed workloads, explicit pool shapes, +or grouping across pods? Use Balloons. Otherwise start with Topology-aware. diff --git a/kubernetes/nri-resource-policies/quickstart.md b/kubernetes/nri-resource-policies/quickstart.md new file mode 100644 index 0000000..afdf116 --- /dev/null +++ b/kubernetes/nri-resource-policies/quickstart.md @@ -0,0 +1,191 @@ +# Quick Start: Isolating a Latency-Critical Service with Balloons + +This is a short, reproducible example. Two identical Redis pods run on one node +under the same noisy neighbor. One is placed in a dedicated CPU pool; the other +shares CPUs with the noise. You measure the tail latency of each and see the +difference. + +> Numbers below are from a 2-socket, 4-NUMA-node test VM and vary by hardware. +> The comparison is what matters: the isolated pod's tail latency is lower and +> steadier. + +## Prerequisites + +- A single-node Kubernetes cluster with the Balloons policy installed. See the + [install steps](README.md). +- A node with at least 6 CPUs. +- `kubectl` pointing at the cluster. + +## 1. Apply the policy + +Three CPU pools: `dedicated` (selected by a pod label), `shared` (everything +else in the `default` namespace), and the implicit reserved pool for system +pods. Each work pool is fixed at 2 CPUs, so contention is deterministic. + +```bash +cat > balloons-quickstart.yaml <<'EOF' +apiVersion: config.nri/v1alpha1 +kind: BalloonsPolicy +metadata: + name: default + namespace: kube-system +spec: + pinCPU: true + reservedResources: + cpu: cpuset:0-1 + balloonTypes: + - name: dedicated + minCPUs: 2 + maxCPUs: 2 + preferNewBalloons: true + matchExpressions: + - key: pod/labels/balloon + operator: In + values: ["dedicated"] + - name: shared + minCPUs: 2 + maxCPUs: 2 + namespaces: ["default"] +EOF +kubectl apply -f balloons-quickstart.yaml +``` + +The Helm chart ships a default `BalloonsPolicy`, so `kubectl apply` updates it +and prints a one-line warning about a missing annotation. That is expected and +harmless; the config is applied. + +## 2. Deploy the workloads + +Two Redis pods and one noisy neighbor. `redis-dedicated` carries the +`balloon: dedicated` label and lands in its own pool. `redis-shared` has no +label and shares the `shared` pool with `noisy`, which keeps 8 CPU-burning +loops running. + +```bash +cat > quickstart-pods.yaml <<'EOF' +apiVersion: v1 +kind: Pod +metadata: + name: redis-dedicated + labels: + balloon: dedicated +spec: + containers: + - name: redis + image: docker.io/library/redis:7-alpine +--- +apiVersion: v1 +kind: Pod +metadata: + name: redis-shared +spec: + containers: + - name: redis + image: docker.io/library/redis:7-alpine +--- +apiVersion: v1 +kind: Pod +metadata: + name: noisy +spec: + containers: + - name: burn + image: docker.io/library/busybox:stable + command: ["sh", "-c", "for i in $(seq 1 8); do while true; do :; done & done; wait"] +EOF +kubectl apply -f quickstart-pods.yaml +kubectl wait --for=condition=Ready --timeout=120s pod/redis-dedicated pod/redis-shared pod/noisy +``` + +## 3. Confirm which containers run on which CPUs + +There are two simple ways to check. + +**Per container (most direct).** Each container reports the CPUs it may run on. +No extra tools needed: + +```bash +for p in redis-dedicated redis-shared noisy; do + echo -n "$p: " + kubectl exec "$p" -- grep Cpus_allowed_list /proc/1/status +done +``` + +Example output: + +```text +redis-dedicated: Cpus_allowed_list: 10-11 +redis-shared: Cpus_allowed_list: 4-5 +noisy: Cpus_allowed_list: 4-5 +``` + +`redis-dedicated` has its own CPUs. `redis-shared` and `noisy` share the same +pair. That shared pair is the contention this example creates. The exact CPU +numbers vary per run. + +**Per pool (from NodeResourceTopology).** The plugin publishes each balloon and +its CPUs as a NodeResourceTopology object, so you can see the pools cluster-wide +(requires `jq`): + +```bash +kubectl get noderesourcetopologies -o json \ + | jq -r '.items[].zones[] | select(.type=="balloon") + | .name + "\t" + ([.attributes[] | select(.name=="cpuset").value][0])' +``` + +Example output: + +```text +reserved[0] 0-1 +default[0] +dedicated[0] 10-11 +shared[0] 4-5 +``` + +`dedicated[0]` and `shared[0]` hold different CPUs; `reserved[0]` is the pool +for system pods. + +## 4. Measure tail latency + +Run the built-in `redis-benchmark` inside each Redis pod and read the latency +summary. The `p99` column is the 99th-percentile latency. + +```bash +echo "== dedicated ==" +kubectl exec redis-dedicated -- redis-benchmark -n 200000 -c 50 -t get 2>&1 \ + | grep -A2 "latency summary" +echo "== shared ==" +kubectl exec redis-shared -- redis-benchmark -n 200000 -c 50 -t get 2>&1 \ + | grep -A2 "latency summary" +``` + +Example output (2-socket, 4-NUMA-node test VM; numbers vary by hardware): + +```text +== dedicated == + latency summary (msec): + avg min p50 p95 p99 max + 0.539 0.192 0.535 0.599 0.767 18.607 +== shared == + latency summary (msec): + avg min p50 p95 p99 max + 0.837 0.136 0.775 1.695 1.927 2.207 +``` + +The dedicated pod holds a low, steady p99 (about 0.77 ms) across runs. The +shared pod, competing with the noisy neighbor for the same CPUs, has a higher +and more variable p99 (roughly 1.2 to 2 ms in repeated runs). Both pods are +identical; only the CPU pool differs. + +## 5. Clean up + +```bash +kubectl delete -f quickstart-pods.yaml +kubectl delete -f balloons-quickstart.yaml +rm -f quickstart-pods.yaml balloons-quickstart.yaml +``` + +## Next + +To tune frequency, C-states, scheduling, or device locality per pool, see the +[Balloons page](balloons.md). diff --git a/kubernetes/nri-resource-policies/topology-aware.md b/kubernetes/nri-resource-policies/topology-aware.md new file mode 100644 index 0000000..6daf941 --- /dev/null +++ b/kubernetes/nri-resource-policies/topology-aware.md @@ -0,0 +1,51 @@ +# Topology-Aware Policy + +The Topology-aware policy aligns each workload with the hardware. It reads the +node's topology (sockets, dies, NUMA nodes, L3 caches) and places a container's +CPU and memory together, as close as they fit. You get NUMA- and cache-aligned +placement without configuring anything per workload. + +This is the policy to start with. If you need explicit CPU pools or per-group +power and scheduling tuning, use [Balloons](balloons.md) instead. + +## What it solves + +- **NUMA and cache alignment, automatically.** The [NUMA guide](../../hardware/NUMA/README.md) + explains why aligning CPU and memory matters and how to do it by hand. + Topology-aware does the same alignment for every pod as it starts, and picks + the tightest fit that has room. Its + [case study](../../hardware/NUMA/case_studies/java_server_side_workload.md) + shows the kind of gain at stake. + +- **Scales to large NUMA counts.** It builds a pool tree from the real topology + and scores candidates, so it keeps working on systems with many NUMA nodes + where simpler pinning breaks down. + +- **Exclusive, shared, or mixed CPUs.** Give a workload its own cores, a shared + pool, or a mix. It can also use kernel-isolated (`isolcpus`) CPUs for the + exclusive part. + +- **Multi-tier memory.** Assign workloads to the memory type they prefer across + DRAM, HBM, and PMEM, with an optional cold-start phase pinned to PMEM. + +- **Device-aligned placement.** Pick the pool nearest the devices a workload + uses, so CPU, memory, and device stay local. + +- **CPU tuning, PCT, and IRQ affinity (Guaranteed containers only).** + Topology-aware can also set CPU frequency, C-states, Priority Core Turbo, and + IRQ affinity per CPU class. These apply only to **Guaranteed** containers that + hold exclusive CPUs. To tune Burstable or BestEffort workloads, or shared CPU + pools, use [Balloons](balloons.md), which applies per-pool tuning to any QoS + class. + +## How it decides + +For each container it filters out pools that lack free capacity, scores the +rest, and picks the best. Scoring prefers tighter alignment (lower latency), +more free capacity, and better device locality. Pools lower in the tree mean +stricter alignment; higher pools fit more but relax alignment. + +## Reference + +For installation, configuration, and cookbooks, see the upstream +[Topology-aware documentation](https://containers.github.io/nri-plugins/stable/docs/resource-policy/policy/topology-aware.html).