Add optimizing Kubernetes applications and services with NRI resource policies

README.md

@@ -70,6 +70,14 @@ We aim to provide a dynamic resource where users can find the latest optimizatio
   - [PMU](hardware/PMU/README.md)
   - [Priority Core Turbo](hardware/priority_core_turbo/README.md)
   - [Scaling](hardware/scaling/README.md)
+- Kubernetes
+  - [Kubernetes Optimization](kubernetes/README.md)
+    - [NRI Resource-Policy Plugins](kubernetes/nri-resource-policies/README.md)
+      - [Balloons Policy](kubernetes/nri-resource-policies/balloons.md)
+      - [Topology-Aware Policy](kubernetes/nri-resource-policies/topology-aware.md)
+      - [Choosing a Policy](kubernetes/nri-resource-policies/choosing-a-policy.md)
+      - [Choosing a Policy with the Kubernetes Managers](kubernetes/nri-resource-policies/choosing-a-policy-with-kubernetes-managers.md)
+      - [Quick Start](kubernetes/nri-resource-policies/quickstart.md)
 
 ## Contributing
 

hardware/NUMA/README.md

@@ -4,6 +4,8 @@ Non-Uniform Memory Access (NUMA) is a memory architecture used in modern multi-s
 
 This directory collects best practices, case studies, and performance opportunities for running workloads in NUMA environments. The goal is to help you understand why hardware topology matters, identify configuration mismatches that leave performance on the table, and apply proven, hardware-aware patterns to your deployments.
 
+> **Running in Kubernetes?** The Topology-aware policy applies NUMA-aligned placement to every pod automatically. See [Kubernetes Optimization](../../kubernetes/nri-resource-policies/topology-aware.md).
+
 Actual improvements will vary depending on a given workload's characteristics. It is recommended to use a profiling solution such as [VTune Profiler](https://github.com/intel/optimization-zone/tree/main/tools/vtune/README.md) to more accurately gauge where an application's hotpaths are and which optimizations would have the greatest effect.
 
 ## What's Inside

hardware/priority_core_turbo/README.md

@@ -11,6 +11,8 @@ Running these threads on **High-Priority (HP) cores** improves GPU utilization,
 
 This guide provides a Docker-based workflow and supporting scripts to check PCT support, configure CLOS assignments, and validate the selected high-priority CPU list. The PCT setup and verification steps run from the container, so users do not need to install `intel-speed-select` or other extra host-side dependencies beyond Docker.  
 
+> **Running in Kubernetes?** To assign PCT high-priority cores to specific pods, use the Balloons policy. See [Kubernetes Optimization](../../kubernetes/nri-resource-policies/balloons.md).
+
 Validated platforms:
 
 - **Intel® Xeon® 6776P**

kubernetes/README.md

@@ -0,0 +1,81 @@
+# Kubernetes Optimization
+
+Most tuning guides in the Optimization Zone assume you control the host: you
+pin threads, set the CPU governor, disable deep C-states, or align a workload
+to a NUMA node. In Kubernetes you usually cannot do that by hand. The scheduler
+places pods on nodes, and the built-in CPU and Memory managers know little
+about NUMA distance, cache domains, device locality, Priority Core Turbo (PCT),
+uncore and C-states, or real-time scheduling.
+
+This section is about closing that gap. It shows how to apply the same
+hardware-aware optimizations to containerized workloads, on both cloud and edge
+nodes, without editing each node by hand.
+
+## The mechanism: NRI resource-policy plugins
+
+The [NRI (Node Resource Interface) resource-policy plugins](https://github.com/containers/nri-plugins)
+run as a DaemonSet on each node. They hook into the container runtime
+(containerd or CRI-O) and set CPU, memory, and device affinity for each
+container as it starts. They can also tune CPU frequency, C-states, IRQ
+affinity, and process scheduling per group of containers.
+
+Two policies cover most needs:
+
+- **[Topology-aware](nri-resource-policies/topology-aware.md)** — automatic
+  NUMA-, cache-, and device-aligned placement. No per-workload configuration
+  needed. Start here if you want the wins from the [NUMA guide](../hardware/NUMA/README.md)
+  without hand-pinning.
+- **[Balloons](nri-resource-policies/balloons.md)** — group containers into CPU
+  pools ("balloons") and tune each pool: dedicated CPUs, frequency, C-states,
+  PCT high-priority cores, real-time scheduling, device locality. Choose this
+  when you need explicit control per group of workloads.
+
+Not sure which fits? See [choosing a policy](nri-resource-policies/choosing-a-policy.md),
+or, if you already use the built-in Kubernetes managers,
+[choosing a policy with the Kubernetes managers](nri-resource-policies/choosing-a-policy-with-kubernetes-managers.md).
+
+New here? Start with the [foundation page](nri-resource-policies/README.md), or
+jump to the [quick start](nri-resource-policies/quickstart.md).
+
+## What maps to what
+
+If you came from a hardware or software guide, this is where its host-level
+lever shows up in Kubernetes:
+
+| Optimization Zone topic | NRI policy feature |
+| --- | --- |
+| [NUMA alignment](../hardware/NUMA/README.md) | Topology-aware automatic alignment; Balloons topology balancing |
+| Memory bandwidth vs. latency trade-off | Balloons spreading across NUMA nodes and sockets, or local-only |
+| Multi-tier memory (DRAM, HBM, PMEM) | Topology-aware multi-tier allocation |
+| [Priority Core Turbo](../hardware/priority_core_turbo/README.md) | Balloons CPU classes, any QoS; Topology-aware CPU classes, Guaranteed containers only |
+| [CPU frequency, EPP, governor](../software/common/README.md) | Balloons CPU classes: min/max frequency, governor, EPP, uncore |
+| C-states and wakeup latency | Balloons CPU classes: disabled C-states |
+| [Hyper-threading choices](../software/common/README.md) | Topology-aware HT control; Balloons hyper-thread-aware sharing |
+| Dedicated cores / isolation | Balloons dedicated pools; Topology-aware exclusive CPUs and isolcpus |
+| Burst headroom without noisy neighbors | Balloons idle-CPU sharing |
+| Real-time scheduling and I/O priority | Balloons scheduling classes (SCHED_FIFO, ioClass, ioPriority) |
+| IRQ affinity | Balloons IRQ affinity, any QoS; Topology-aware IRQ affinity, Guaranteed containers only |
+| PCI / GPU / NIC / accelerator locality | Balloons device locality; Topology-aware device alignment |
+
+Each feature is documented upstream. The pages in this section explain when to
+reach for it and link to the details.
+
+## Who this is for
+
+- **AI and inference** — put host-side serial work (tokenization, dispatch) on
+  PCT high-priority cores and keep it local to the GPU or NIC.
+- **Databases** — align CPU and memory to cut cross-NUMA traffic and tail
+  latency; isolate from noisy neighbors.
+- **Web services and proxies** — let bursty handlers use idle CPUs without
+  hurting a latency-critical neighbor.
+- **Real-time and industrial / edge** — dedicated cores, no deep C-states,
+  real-time scheduling, and IRQs steered off the control loop.
+
+## Measure your results
+
+Improvements depend on the workload. Use a profiler such as
+[VTune Profiler](../tools/vtune/README.md) or [PerfSpect](../tools/perfspect/README.md)
+to see where time goes and to compare before and after.
+
+> Performance varies by use, configuration, and other factors. See the
+> [disclaimer](../README.md#disclaimer).

kubernetes/nri-resource-policies/README.md

@@ -0,0 +1,81 @@
+# NRI Resource-Policy Plugins
+
+The NRI resource-policy plugins apply hardware-aware placement to containers in
+Kubernetes. This page covers what they are and how to get one running. The
+policy pages ([Balloons](balloons.md), [Topology-aware](topology-aware.md))
+cover when and why to use each.
+
+## What is NRI
+
+[NRI (Node Resource Interface)](https://containers.github.io/nri-plugins/stable/docs/resource-policy/introduction.html)
+is a plugin interface in the container runtime. A resource-policy plugin
+receives each container's lifecycle events and decides its CPU, memory, and
+device affinity before it starts. This happens on the node, below the
+Kubernetes scheduler.
+
+## How it runs
+
+- One plugin runs as a **DaemonSet**, one pod per node.
+- It talks to the container runtime over NRI: **containerd 1.7+** or
+  **CRI-O 1.26+**, with NRI enabled (the default in current containerd).
+- You run **one policy per node**. Balloons and Topology-aware are
+  alternatives, not layers.
+
+## Prerequisites
+
+- Kubernetes 1.24 or newer (1.27+ for the PCT flow).
+- A container runtime with NRI enabled: containerd 1.7+ or CRI-O 1.26+.
+- Helm 3+.
+
+See the upstream [setup guide](https://containers.github.io/nri-plugins/stable/docs/resource-policy/setup.html)
+for enabling NRI and for the full install procedure.
+
+## Install
+
+Add the Helm repository:
+
+```bash
+helm repo add nri-plugins https://containers.github.io/nri-plugins
+helm repo update
+```
+
+Install one policy. For Topology-aware:
+
+```bash
+helm install nri-resource-policy-topology-aware \
+  nri-plugins/nri-resource-policy-topology-aware --namespace kube-system
+```
+
+For Balloons:
+
+```bash
+helm install nri-resource-policy-balloons \
+  nri-plugins/nri-resource-policy-balloons --namespace kube-system
+```
+
+Check the DaemonSet is ready:
+
+```bash
+kubectl -n kube-system rollout status ds/nri-resource-policy-topology-aware
+```
+
+## Configuration
+
+You configure a policy with a custom resource, not Helm values, so changes
+apply without reinstalling. Each policy has its own kind
+(`BalloonsPolicy`, `TopologyAwarePolicy`). Configuration works at three scopes,
+most specific wins:
+
+- **default** — all nodes without a more specific config.
+- **group** — nodes labeled `config.nri/group=$NAME`.
+- **node** — a single named node.
+
+This lets one cluster serve mixed hardware and mixed workloads. See the
+upstream [configuration guide](https://containers.github.io/nri-plugins/stable/docs/resource-policy/configuration.html).
+
+## Scope of these pages
+
+This section orients you and points to the details. For install options, the
+full configuration schema, and cookbooks, the upstream
+[NRI plugins documentation](https://containers.github.io/nri-plugins/stable/)
+is the source of truth.

kubernetes/nri-resource-policies/balloons.md

@@ -0,0 +1,73 @@
+# Balloons Policy
+
+The Balloons policy groups containers into CPU pools called *balloons*. A
+container runs only on the CPUs of its balloon. Each balloon can be tuned on
+its own: how many CPUs, at what frequency, which C-states, how they are
+scheduled, and which devices they sit close to.
+
+Reach for Balloons when you need explicit control over how groups of workloads
+share a node. For automatic alignment with no per-workload setup, see
+[Topology-aware](topology-aware.md) instead.
+
+A balloon can hold containers of any QoS class: BestEffort, Burstable, or
+Guaranteed. Its tuning (frequency, C-states, PCT, IRQ affinity, scheduling)
+applies to all of them. This is a key difference from Topology-aware, whose CPU
+tuning, PCT, and IRQ affinity apply only to Guaranteed containers with exclusive
+CPUs.
+
+## What it solves
+
+Each item below is a common host-level tuning task and where Balloons handles
+it in Kubernetes. Follow the links for the configuration.
+
+- **Isolate a latency-critical workload.** Give it dedicated CPUs, stop it
+  sharing L2 cache with other workloads, and disable deep C-states so cores do
+  not sleep. See the upstream
+  [latency-critical recipe](https://containers.github.io/nri-plugins/stable/docs/resource-policy/policy/balloons.html#latency-critical-containers).
+
+- **Run hot serial threads at top turbo.** Put them on Priority Core Turbo
+  high-priority cores while the rest of the node stays at base frequency. See
+  the [PCT guide](../../hardware/priority_core_turbo/README.md) for the hardware
+  side and the upstream
+  [PCT quick start](https://containers.github.io/nri-plugins/stable/docs/resource-policy/policy/howto/balloons-pct-quickstart.html)
+  for the Kubernetes side.
+
+- **Get more memory bandwidth.** Spread a balloon's CPUs across NUMA nodes or
+  sockets to use more memory channels, or keep it local for lowest latency. See
+  the upstream
+  [memory bandwidth recipe](https://containers.github.io/nri-plugins/stable/docs/resource-policy/policy/balloons.html#maximum-memory-bandwidth-containers)
+  and the [NUMA guide](../../hardware/NUMA/README.md).
+
+- **Let bursty workloads use idle CPUs safely.** A balloon can borrow otherwise
+  idle CPUs within a core, L2 domain, NUMA node, or package, and you control
+  which workload types share a physical core. See the upstream
+  [hyper-thread sharing recipe](https://containers.github.io/nri-plugins/stable/docs/resource-policy/policy/balloons.html#workload-aware-hyperthread-sharing).
+
+- **Set frequency, governor, EPP, and C-states per group.** The host-wide knobs
+  in the [common recommendations](../../software/common/README.md) become
+  per-balloon CPU classes. See the upstream
+  [CPU tuning options](https://containers.github.io/nri-plugins/stable/docs/resource-policy/policy/balloons.html#cpu-tuning).
+
+- **Apply real-time scheduling and I/O priority.** Assign a balloon a scheduling
+  class such as `SCHED_FIFO` with a priority, plus an I/O class and priority.
+  Useful for control loops and other real-time work. See the upstream
+  [CPU tuning options](https://containers.github.io/nri-plugins/stable/docs/resource-policy/policy/balloons.html#cpu-tuning).
+
+- **Steer IRQs off critical cores.** Move device interrupts away from the CPUs
+  running a latency-critical balloon.
+
+- **Keep CPUs close to a device.** Place a balloon on the CPUs nearest a GPU,
+  NIC, or accelerator to cut access latency. Useful for AI inference and packet
+  processing.
+
+## Try it
+
+The upstream [PCT quick start](https://containers.github.io/nri-plugins/stable/docs/resource-policy/policy/howto/balloons-pct-quickstart.html)
+is the fastest hands-on path and shows a measurable difference between tuned and
+untuned pods. For a local isolation example, see the
+[quick start](quickstart.md) in this section.
+
+## Reference
+
+Full configuration, all balloon and CPU-class fields, and more recipes are in
+the upstream [Balloons documentation](https://containers.github.io/nri-plugins/stable/docs/resource-policy/policy/balloons.html).

kubernetes/nri-resource-policies/choosing-a-policy-with-kubernetes-managers.md

@@ -0,0 +1,62 @@
+# Choosing a Policy, Including the Built-in Kubernetes Managers
+
+Kubernetes ships its own CPU manager, Memory manager, Device manager, and
+Topology manager. They pin CPUs and memory and try to keep a pod's resources on
+one NUMA node. If you already use them, this page shows where the NRI policies
+fit and what they add.
+
+For the choice between the two NRI policies alone, see
+[choosing a policy](choosing-a-policy.md).
+
+## Three options
+
+- **Built-in Kubernetes managers** — CPU manager (`static` policy), Memory
+  manager, Device manager, and Topology manager working together.
+- **[Topology-aware](topology-aware.md)** — an NRI policy that aligns CPU and
+  memory automatically.
+- **[Balloons](balloons.md)** — an NRI policy built around CPU pools you define.
+
+## Topology-aware vs. the built-in managers
+
+Topology-aware is close to a drop-in replacement for the built-in CPU and
+Memory managers. The intent is the same: place a workload's CPU and memory
+together on aligned hardware. The differences are in the allocation:
+
+- **Smarter allocation.** It builds a pool tree from the real topology and
+  scores candidates, rather than following the managers' fixed rules.
+- **Scales further.** The built-in managers already struggle to align well
+  around 8 NUMA nodes. Topology-aware keeps working at 8 nodes and beyond.
+- **More tuning for Guaranteed containers.** It can also set CPU frequency,
+  C-states, PCT, and IRQ affinity for Guaranteed containers with exclusive
+  CPUs. The built-in managers do not offer these.
+
+If your goal is aligned placement and you find the built-in managers limiting on
+larger systems, Topology-aware is the natural step up.
+
+## Balloons vs. the built-in managers
+
+Balloons has different semantics. Instead of per-pod pinning rules, you
+partition the node into CPU pools and assign containers and pods to them:
+
+- **Group and mix freely.** Put containers from the same or different pods,
+  applications, or namespaces into the same pool, or keep them apart. This
+  grouping is more flexible than the built-in managers allow.
+- **More to tune, for any QoS class.** Each pool can set CPU frequency,
+  C-states, EPP, PCT high-priority cores, real-time scheduling, I/O priority,
+  IRQ affinity, and device locality. This tuning applies to any container in a
+  pool, not just Guaranteed ones. The built-in managers do not offer these, and
+  Topology-aware applies the CPU, PCT, and IRQ tuning only to Guaranteed
+  containers.
+
+Choose Balloons when you need that grouping flexibility or the extra tuning.
+
+## Which to use
+
+| Situation | Use |
+| --- | --- |
+| Aligned CPU/memory, small NUMA count, happy with built-ins | Built-in managers |
+| Aligned CPU/memory, larger NUMA count or want smarter allocation | Topology-aware |
+| Need CPU pools, per-group tuning, PCT, or real-time control | Balloons |
+
+The NRI policies are alternatives to the built-in managers for these tasks, not
+additions on top. Run one approach per node.

kubernetes/nri-resource-policies/choosing-a-policy.md

@@ -0,0 +1,41 @@
+# Choosing a Policy
+
+Balloons and Topology-aware are alternatives. You run one policy per node, so
+pick the one that matches how much control you need. If you already use the
+built-in Kubernetes CPU and Memory managers, see
+[choosing a policy with the Kubernetes managers](choosing-a-policy-with-kubernetes-managers.md).
+
+## Start here
+
+Use **[Topology-aware](topology-aware.md)** unless you have a specific reason
+not to. It aligns CPU and memory to the hardware for every pod with no
+per-workload configuration.
+
+Switch to **[Balloons](balloons.md)** when you need any of these:
+
+- dedicated CPU pools for groups of containers or pods,
+- CPU frequency, C-states, EPP, PCT, or IRQ tuning for **any** QoS class, not
+  just Guaranteed containers,
+- real-time scheduling or I/O priority,
+- grouping containers from different pods or namespaces into one pool,
+- explicit control over which workloads share physical cores.
+
+Topology-aware also supports CPU power, PCT, and IRQ tuning, but only for
+Guaranteed containers with exclusive CPUs. Balloons applies the same tuning to
+any container in a balloon.
+
+## Side by side
+
+| | Topology-aware | Balloons |
+| --- | --- | --- |
+| Main idea | Automatic hardware-aligned placement | CPU pools you define and tune |
+| Setup effort | None per workload | You define balloon types |
+| CPU/memory alignment | Yes, automatic | Yes, configurable |
+| CPU power, PCT, IRQ tuning | Guaranteed (exclusive-CPU) containers only | Any container in a balloon |
+| Grouping / mixing workloads | Per-pod alignment | Group any pods or containers into shared pools |
+| Best for | Most workloads, the default | Explicit control and mixed workloads on a node |
+
+## Rule of thumb
+
+Do you need per-pool tuning for non-Guaranteed workloads, explicit pool shapes,
+or grouping across pods? Use Balloons. Otherwise start with Topology-aware.