From 0fdd7886488a12429ee71e5c6d2c8d852e400535 Mon Sep 17 00:00:00 2001 From: gthulhu-work Date: Sat, 5 Sep 2026 18:13:13 +0800 Subject: [PATCH 1/7] docs: align homepage with Claim2Core direction --- docs/index.md | 112 +++++++++++++++++++++++++++++++------------------- 1 file changed, 70 insertions(+), 42 deletions(-) diff --git a/docs/index.md b/docs/index.md index 1831a90..135aa69 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,94 +1,122 @@ cncf landscape - ebpf landscape [![LFX Health Score](https://insights.linuxfoundation.org/api/badge/health-score?project=gthulhu)](https://insights.linuxfoundation.org/project/gthulhu) # Gthulhu -Gthulhu helps platform teams understand, automate, and tune Linux scheduling for Kubernetes workloads. +> **DRA chooses what and where; Gthulhu controls how it actually runs.** + +Gthulhu is a cloud-native runtime scheduling platform that connects Kubernetes workload intent to Linux task scheduling with eBPF and `sched_ext`. -It starts with safe, pod-level scheduling observability powered by eBPF. From there, teams can connect those signals to Prometheus, Grafana, and KEDA for scaling decisions that reflect real scheduler pressure. On Linux 6.12+ clusters with `sched_ext`, Gthulhu can also apply workload-aware CPU scheduling policies at the kernel boundary. +Kubernetes can admit a workload, place it on a node, and allocate devices/topology. Gthulhu focuses on the execution gap that follows: identify the Linux tasks that belong to the workload, apply bounded CPU scheduling policy, and verify whether the allocation is actually delivering the workload SLO. [Get Started](k8s.md){: .md-button .md-button--primary } [How It Works](how-it-works.md){: .md-button } -[Share Your Case Study](https://docs.google.com/forms/d/e/1FAIpQLSeT9Ia1iigu45DDbPgfqijWIN7-Ewkm6-AbTc-HsjyHMvBjCA/viewform?usp=publish-editor){: .md-button } +[Claim2Core Roadmap](claim2core.md){: .md-button } -## Why Gthulhu? +## Current Capabilities -Kubernetes schedules pods onto nodes, but the Linux kernel still decides when each process runs on CPU. That last-mile behavior is often invisible, even when it is the reason a workload is waiting, migrating, or missing latency goals. +- **Pod-level scheduling observability** with eBPF. +- **Prometheus / Grafana / KEDA integration** for scheduler-aware operations and scaling. +- **Distributed scheduling intent** through a Manager and per-node Decision Makers. +- **Custom CPU scheduling** on Linux 6.12+ with `sched_ext`. +- **TID-aware node-policy matching** so non-leader worker threads can be targeted directly. +- **Explicit priority semantics** across user-space and kernel scheduler modes. -Gthulhu closes that gap: +## Claim2Core Direction -- **See scheduling pressure** — collect per-process scheduler signals and aggregate them into pod-level metrics. -- **Scale from scheduler reality** — feed Prometheus and KEDA with wait time, runtime, context switches, and CPU migration signals instead of relying only on CPU averages. -- **Configure workloads declaratively** — select pods through the Web UI, REST API, or `PodSchedulingMetrics` CRD. -- **Tune critical workloads** — apply priority and time-slice policies through `sched_ext` when supported by the node kernel. -- **Operate across clusters** — use a Manager plus per-node Decision Makers to turn workload intent into node-local action. +The next architecture step is to connect actual Kubernetes allocation to runtime task scheduling: + +```text +Kueue / Workload API + │ admission / quota + ▼ +kube-scheduler / DRA + │ Node + device + topology allocation + ▼ +Gthulhu Runtime Plane + │ ResourceClaim → Pod/cgroup → TGID/TID + ▼ +sched_ext + eBPF + │ runtime policy + verification + ▼ +Delivered workload SLO +``` -## What You Can Do +The critical correctness rule is: -### Observe Every Workload +- `ResourceSlice` is **inventory**. +- `ResourceClaim.status.allocation` is the workload's **actual allocation**. -Gthulhu attaches eBPF programs to Linux scheduling events and turns raw process activity into Kubernetes-aware metrics. You can track whether a pod is waiting for CPU, moving across CPUs, or spending time in scheduler contention. +Gthulhu should not reimplement kube-scheduler, DRA, or Kueue. It should consume their decisions and control Linux CPU execution **inside** the resource envelope established by Kubernetes/cgroups. -### Automate Smarter Scaling +Read [Claim2Core](claim2core.md) for the implementation phases and safety boundaries. -Scheduling behavior is often a better scaling signal than coarse resource utilization. Gthulhu exports metrics to Prometheus so KEDA can scale workloads based on real pressure observed at the kernel level. +## Why This Matters -### Apply Scheduling Intent +Allocated resources do not automatically become delivered performance. A workload may own a GPU or NIC while its host-side feeder, tokenizer, NCCL/RDMA progress, DPDK, or packet-processing threads are still delayed by CPU contention. -For advanced environments, Gthulhu lets teams define scheduling strategies for selected workloads. The Manager resolves Kubernetes intent, Decision Makers map it to node-local processes, and the scheduler applies policies such as priority and custom time slices. +Gthulhu makes this gap observable and controllable. -### Keep the Kernel Untouched +Immediate validation paths include: -The base monitor runs with eBPF on BTF-enabled Linux kernels and does not require kernel patches. The optional scheduler uses Linux `sched_ext`, so teams can experiment with custom scheduling policies without maintaining a custom kernel. +- **CPU DRA × Gthulhu × free5GC/UPF** for p99/p99.9 latency and jitter; +- **GPU + RDMA + CPU DRA × phase-aware LLM scheduling** for TTFT, ITL, GPU idle, and communication progress. ## Architecture at a Glance -``` +```text User / Web UI / CRD │ ▼ -Manager API ───────▶ MongoDB +Manager API ───────▶ MongoDB / Kubernetes API │ ▼ Decision Maker DaemonSet │ ├── eBPF scheduling metrics collector ──▶ Prometheus / Grafana / KEDA │ - └── sched_ext scheduler integration ───▶ Linux kernel scheduler path + └── task resolution / scheduling intent + │ + ▼ + Gthulhu daemon + │ + ▼ + sched_ext / BPF + │ + ▼ + Linux scheduler ``` -The Manager owns users, RBAC, strategy APIs, and cluster-wide intent. Decision Makers run on each node, discover matching pod processes, expose node-local PID strategies, collect metrics, and forward runtime configuration to the local Gthulhu daemon. The daemon can run in monitor-only mode or enable the advanced scheduler when configured. - -Read the full data flow in [How It Works](how-it-works.md). +## Recent Scheduler Semantics -## Demo +Node-policy matching is thread-aware. The Decision Maker scans `/proc//task/` so a policy can target a named non-leader worker thread rather than only the process leader. - +Priority handling is also explicit: - +- `Priority > 0` means boost; +- `Priority == 0` is non-boosting; +- user-space mode supports TID-first lookup with TGID fallback; +- kernel mode does not insert non-boosting strategies into the priority BPF map, avoiding accidental priority-0 promotion. - +See [How It Works](how-it-works.md) for the details and current limitations. -## Latest News - -!!! success "Gthulhu joins CNCF Landscape" - Gthulhu is part of the [CNCF Landscape](https://landscape.cncf.io/?item=provisioning--automation-configuration--gthulhu), alongside cloud-native infrastructure projects. +## Demo -!!! success "Gthulhu joins eBPF Application Landscape" - Gthulhu is listed in the [eBPF Application Landscape](https://ebpf.io/applications/) as an eBPF-based scheduling and observability project. + ## Next Steps - [Deploy Gthulhu with Kubernetes](k8s.md) -- [Understand the architecture](how-it-works.md) +- [Understand the architecture and current scheduler semantics](how-it-works.md) +- [Read the Claim2Core roadmap](claim2core.md) - [Configure pod scheduling metrics](pod-metrics.md) -- [Explore the API reference](api-reference.md) +- [Contribute](contributing.md) ## Community -- **GitHub**: [Gthulhu](https://github.com/Gthulhu/Gthulhu) | [Qumun](https://github.com/Gthulhu/scx_goland_core) -- **Issues**: Report bugs or request features through GitHub Issues. -- **License**: Apache License 2.0. +- **GitHub**: [Gthulhu/Gthulhu](https://github.com/Gthulhu/Gthulhu) +- **Roadmap**: [Issue #141](https://github.com/Gthulhu/Gthulhu/issues/141) +- **Framework**: [Gthulhu/qumun](https://github.com/Gthulhu/qumun) +- **License**: Apache License 2.0 From 593e90207c8e6c61b9653f768c10c8676b83420e Mon Sep 17 00:00:00 2001 From: gthulhu-work Date: Sat, 5 Sep 2026 18:13:33 +0800 Subject: [PATCH 2/7] docs: align Chinese homepage with Claim2Core direction --- docs/index.zh.md | 112 +++++++++++++++++++++++++++++------------------ 1 file changed, 70 insertions(+), 42 deletions(-) diff --git a/docs/index.zh.md b/docs/index.zh.md index bddeae3..fa07417 100644 --- a/docs/index.zh.md +++ b/docs/index.zh.md @@ -1,94 +1,122 @@ cncf landscape - ebpf landscape [![LFX Health Score](https://insights.linuxfoundation.org/api/badge/health-score?project=gthulhu)](https://insights.linuxfoundation.org/project/gthulhu) # Gthulhu -Gthulhu 協助平台團隊看懂、自動化並調整 Kubernetes 工作負載背後的 Linux 排程行為。 +> **DRA chooses what and where; Gthulhu controls how it actually runs.** + +Gthulhu 是一個雲原生 runtime scheduling 平台，透過 eBPF 與 Linux `sched_ext`，把 Kubernetes 工作負載意圖連接到真正的 Linux task scheduling。 -它先從安全的 Pod 層級排程可觀測性開始：透過 eBPF 收集核心排程訊號，交給 Prometheus、Grafana 與 KEDA，讓擴縮容決策反映真實的排程壓力。若叢集運行 Linux 6.12+ 且支援 `sched_ext`，Gthulhu 也能在核心排程邊界套用工作負載感知的 CPU 排程策略。 +Kubernetes 可以決定 Workload 是否能開始、放在哪個 Node，以及取得哪些裝置或 topology。Gthulhu 專注於 allocation 之後的 execution gap：找出真正屬於該工作負載的 Linux tasks、套用有邊界的 CPU 排程策略，並驗證 allocation 是否真的轉化成 workload SLO。 [開始使用](k8s.md){: .md-button .md-button--primary } [了解運作原理](how-it-works.md){: .md-button } -[分享你的使用案例](https://docs.google.com/forms/d/e/1FAIpQLSeT9Ia1iigu45DDbPgfqijWIN7-Ewkm6-AbTc-HsjyHMvBjCA/viewform?usp=publish-editor){: .md-button } +[Claim2Core Roadmap](claim2core.md){: .md-button } -## 為什麼需要 Gthulhu？ +## 目前已具備的能力 -Kubernetes 負責把 Pod 放到節點上，但每個 Process 何時真正取得 CPU，仍由 Linux 核心排程器決定。這段最後一哩通常很難觀察，卻可能正是工作負載等待、遷移或延遲不穩的原因。 +- **Pod 層級排程可觀測性**：使用 eBPF 收集 scheduler signals。 +- **Prometheus / Grafana / KEDA 整合**：讓操作與 autoscaling 能看見真實 scheduler pressure。 +- **分散式 scheduling intent**：Manager 搭配每節點 Decision Maker。 +- **自訂 CPU scheduling**：Linux 6.12+ 可使用 `sched_ext`。 +- **TID-aware node policy matching**：可以直接命中非 leader 的 worker thread。 +- **明確的 priority semantics**：user-space 與 kernel mode 不再把 non-boosting rule 誤解成最高優先級。 -Gthulhu 補上這個缺口： +## Claim2Core 方向 -- **看見排程壓力** — 收集每個 Process 的排程訊號，並彙整成 Pod 層級指標。 -- **用真實排程狀態擴縮** — 將等待時間、執行時間、上下文切換與 CPU 遷移等訊號送入 Prometheus 與 KEDA，而不只依賴 CPU 平均使用率。 -- **用宣告式方式選擇工作負載** — 透過 Web UI、REST API 或 `PodSchedulingMetrics` CRD 指定要觀測的 Pod。 -- **調整關鍵工作負載** — 在支援 `sched_ext` 的節點上套用優先級與時間片策略。 -- **跨節點落實意圖** — 由 Manager 與各節點 Decision Maker 將工作負載意圖轉成節點本地行動。 +下一個架構階段，是把 Kubernetes 的實際 allocation 接到 runtime task scheduling： + +```text +Kueue / Workload API + │ admission / quota + ▼ +kube-scheduler / DRA + │ Node + device + topology allocation + ▼ +Gthulhu Runtime Plane + │ ResourceClaim → Pod/cgroup → TGID/TID + ▼ +sched_ext + eBPF + │ runtime policy + verification + ▼ +Delivered workload SLO +``` -## 你可以用 Gthulhu 做什麼？ +最重要的 correctness 原則是： -### 觀測每個工作負載 +- `ResourceSlice` 是 **inventory**。 +- `ResourceClaim.status.allocation` 才是 workload 的 **actual allocation**。 -Gthulhu 將 eBPF 程式掛到 Linux 排程事件，並把原始 Process 活動轉成 Kubernetes 可理解的指標。你可以知道 Pod 是否在等待 CPU、是否頻繁跨 CPU 遷移，或是否受到排程競爭影響。 +Gthulhu 不應重新實作 kube-scheduler、DRA 或 Kueue，而是消費它們的決策，並且只在 Kubernetes/cgroup 已允許的 resource envelope 內控制 CPU execution。 -### 讓擴縮容更貼近現況 +完整 implementation phases 與安全邊界請看 [Claim2Core](claim2core.md)。 -排程行為通常比粗略的資源使用率更能反映工作負載壓力。Gthulhu 將指標匯出到 Prometheus，讓 KEDA 能依核心層級觀察到的壓力進行擴縮。 +## 為什麼重要 -### 套用排程意圖 +Allocated resource 不等於 delivered performance。Workload 即使已拿到 GPU 或 NIC，host-side 的 feeder、tokenizer、NCCL/RDMA progress、DPDK 或 packet-processing threads 仍可能因 CPU contention 被延遲。 -在進階環境中，團隊可以為特定工作負載定義排程策略。Manager 解析 Kubernetes 層級的意圖，Decision Maker 對應到節點本地 Process，排程器再套用優先級、自訂時間片等策略。 +Gthulhu 的價值就是讓這個落差可以被觀測、控制與驗證。 -### 不需要維護自訂核心 +近期最值得驗證的兩條路線： -基礎監控功能只需要 BTF-enabled Linux 核心即可透過 eBPF 運作，不需修補核心。選用的進階排程器則基於 Linux `sched_ext`，讓團隊能實驗自訂排程策略，而不必維護客製化核心。 +- **CPU DRA × Gthulhu × free5GC/UPF**：觀察 p99/p99.9 latency 與 jitter； +- **GPU + RDMA + CPU DRA × phase-aware LLM scheduling**：觀察 TTFT、ITL、GPU idle 與 communication progress。 ## 架構一覽 -``` +```text User / Web UI / CRD │ ▼ -Manager API ───────▶ MongoDB +Manager API ───────▶ MongoDB / Kubernetes API │ ▼ Decision Maker DaemonSet │ ├── eBPF 排程指標收集器 ──▶ Prometheus / Grafana / KEDA │ - └── sched_ext 排程整合 ───▶ Linux 核心排程路徑 + └── task resolution / scheduling intent + │ + ▼ + Gthulhu daemon + │ + ▼ + sched_ext / BPF + │ + ▼ + Linux scheduler ``` -Manager 負責使用者、RBAC、策略 API 與叢集層級意圖。Decision Maker 在每個節點上執行，尋找符合條件的 Pod Process、提供節點本地 PID 策略、收集指標，並把 runtime configuration 轉送給本地 Gthulhu daemon。daemon 可以只執行 monitor-only 模式，也可以在配置後啟用進階排程器。 - -完整資料流請參考 [運作原理](how-it-works.md)。 +## 最近已合併的 scheduler semantics -## Demo +Node policy 已經是 thread-aware。Decision Maker 會掃描 `/proc//task/`，因此策略可以直接命中具名的非 leader worker thread，而不是只能命中 process leader。 - +Priority semantics 也已明確化： - +- `Priority > 0` 代表 boost； +- `Priority == 0` 是 non-boosting； +- user-space mode 使用 TID-first lookup，必要時 fallback 到 TGID； +- kernel mode 不會把 non-boosting strategy 塞進 priority BPF map，避免 priority 0 被誤當成最高 preemptive priority。 - +詳細行為與目前限制請看 [運作原理](how-it-works.md)。 -## 最新消息 - -!!! success "Gthulhu 加入 CNCF Landscape" - Gthulhu 已列入 [CNCF Landscape](https://landscape.cncf.io/?item=provisioning--automation-configuration--gthulhu)，成為雲原生基礎設施生態的一員。 +## Demo -!!! success "Gthulhu 加入 eBPF Application Landscape" - Gthulhu 已列入 [eBPF Application Landscape](https://ebpf.io/applications/)，作為基於 eBPF 的排程與可觀測性專案。 + ## 下一步 - [在 Kubernetes 部署 Gthulhu](k8s.md) -- [理解系統架構](how-it-works.md) +- [理解系統架構與目前 scheduler semantics](how-it-works.md) +- [閱讀 Claim2Core roadmap](claim2core.md) - [配置 Pod 排程指標](pod-metrics.md) -- [查看 API 文件](api-reference.md) +- [參與貢獻](contributing.md) ## 社群 -- **GitHub**: [Gthulhu](https://github.com/Gthulhu/Gthulhu) | [Qumun](https://github.com/Gthulhu/scx_goland_core) -- **Issue**: 請透過 GitHub Issues 回報問題或提出功能需求。 -- **授權**: Apache License 2.0。 +- **GitHub**: [Gthulhu/Gthulhu](https://github.com/Gthulhu/Gthulhu) +- **Roadmap**: [Issue #141](https://github.com/Gthulhu/Gthulhu/issues/141) +- **Framework**: [Gthulhu/qumun](https://github.com/Gthulhu/qumun) +- **授權**: Apache License 2.0 From ae8eb90f010d9e0aa3ffa52ce710ff925a918c2a Mon Sep 17 00:00:00 2001 From: gthulhu-work Date: Sat, 5 Sep 2026 18:14:03 +0800 Subject: [PATCH 3/7] docs: document TID-aware scheduling semantics --- docs/how-it-works.md | 254 ++++++++++++++++++++++--------------------- 1 file changed, 128 insertions(+), 126 deletions(-) diff --git a/docs/how-it-works.md b/docs/how-it-works.md index 29cc107..e2df119 100644 --- a/docs/how-it-works.md +++ b/docs/how-it-works.md @@ -1,11 +1,11 @@ # How It Works -Gthulhu connects Kubernetes workload intent with Linux scheduler behavior. It has two layers: +Gthulhu connects Kubernetes workload intent to Linux scheduler behavior. It has two layers: - **Pod scheduling observability**: the base feature, powered by an eBPF monitor that collects scheduler metrics and exports pod-level Prometheus data. -- **Custom CPU scheduling**: an advanced feature for Linux 6.12+ with `sched_ext`, where Gthulhu applies priority and time-slice policies through a user-space scheduler and BPF scheduler program. +- **Custom CPU scheduling**: an advanced feature for Linux 6.12+ with `sched_ext`, where Gthulhu applies bounded scheduling policies through a user-space or kernel scheduler path. -The two layers can run together, or Gthulhu can run in monitor-only mode when no scheduler mode is configured. +The two layers can run together, or Gthulhu can run in monitor-only mode. ## Architecture @@ -19,8 +19,6 @@ graph TB M -->|Scheduling intents| DM1 M -->|Scheduling intents| DM2 - M -->|Runtime config| DM1 - M -->|Runtime config| DM2 subgraph "Node 1" DM1[Decision Maker] --> MON1[eBPF Metrics Collector] @@ -39,43 +37,24 @@ graph TB ### Manager -The Manager is the user-facing control plane service. In the current API server implementation it handles: - -- Authentication and token lifecycle: `/api/v1/auth/login`, `/api/v1/auth/refresh`, `/api/v1/auth/logout` -- RBAC resources: users, roles, and permissions -- Scheduling strategies: `/api/v1/strategies` -- Scheduling intents: `/api/v1/intents/self` -- Pod-to-PID visibility by node: `/api/v1/nodes`, `/api/v1/nodes/:nodeID/pods/pids` -- Pod scheduling metrics configuration and runtime values: `/api/v1/pod-scheduling-metrics`, `/api/v1/pod-scheduling-metrics/runtime`, `/api/v1/classify` -- Scheduler runtime configuration: `/api/v1/scheduler/runtime-config/apply`, `/api/v1/scheduler/runtime-config/status` - -The Manager persists state in MongoDB and uses Kubernetes APIs to resolve workload selectors. +The Manager owns cluster-level intent and persistent control-plane state. It resolves workload selectors through Kubernetes APIs and distributes scheduling intent to Decision Makers. ### Decision Maker -The Decision Maker runs per node. It receives cluster-level intent from the Manager, resolves it into node-local process information, and serves the local scheduler and monitor. - -Important endpoints include: +A Decision Maker runs on each node. It resolves cluster-level intent into node-local Linux tasks, exposes local scheduling strategies, and serves the monitor/scheduler path. -- `POST /api/v1/intents` — receive scheduling intents from the Manager -- `GET /api/v1/scheduling/strategies` — expose PID-level strategies to the local scheduler -- `POST /api/v1/metrics` — receive scheduler BSS metrics from the Gthulhu daemon -- `GET /api/v1/pods/pids` — return pod-to-PID mappings for the node -- `POST /api/v1/runtime-config` and `GET /api/v1/runtime-config` — apply and inspect runtime daemon configuration -- `POST /api/v1/auth/token` — issue the scheduler token used for authenticated local API calls -- `GET /metrics` — expose Decision Maker Prometheus metrics +### Gthulhu daemon -### Gthulhu Daemon +The daemon can run: -The root Gthulhu binary can run the monitor, the scheduler, or both: - -- The **monitor** is enabled by `monitor.enabled` and is the default base feature. -- The **scheduler** is enabled when `scheduler.mode` is set to `gthulhu`, `simple`, or `scx`. -- If `scheduler.mode` is set to `none`, the daemon stays in monitor-only mode. +- monitor-only; +- user-space `sched_ext` scheduling; +- experimental kernel-mode policy application; +- selected upstream `scx` schedulers where configured. ## Pod Scheduling Metrics Flow -The monitor is designed to work without `sched_ext`. It loads `sched_monitor.bpf.o`, attaches eBPF programs to scheduler tracepoints, reads BPF maps, maps PIDs to Kubernetes pods, and exports pod-level metrics. +The monitor does not require `sched_ext`. It attaches eBPF programs to scheduler events, maps process/task activity back to Kubernetes Pods, and exports metrics such as runtime, wait time, context switches, run count, and CPU migrations. ```mermaid sequenceDiagram @@ -86,18 +65,16 @@ sequenceDiagram participant M as Prometheus C->>W: Select pods by namespace and labels - W->>P: Resolve matching pods and processes - W->>B: Update monitored PID/TGID maps - B->>B: Track sched_switch and process_exit events - B->>P: Map PIDs back to pods + W->>P: Resolve matching pods/processes + W->>B: Update monitored identities + B->>B: Track scheduler events + B->>P: Map activity back to pods B->>M: Expose pod metrics on /metrics ``` -The collector tracks signals such as runtime, wait time, voluntary and involuntary context switches, run count, and CPU migrations. These metrics can be consumed by Prometheus, Grafana dashboards, and KEDA-based scaling. - ## Scheduling Strategy Flow -Scheduling strategies start at the Kubernetes workload level and end as PID-level decisions on each node. +Scheduling strategies start at Kubernetes/workload intent and end as node-local task decisions. ```mermaid sequenceDiagram @@ -108,144 +85,169 @@ sequenceDiagram participant G as Gthulhu Scheduler participant B as BPF Scheduler - U->>M: Create strategy with selectors and policy - M->>K: Query matching pods - M->>DM: Distribute scheduling intents - DM->>DM: Resolve pods into local PIDs + U->>M: Create strategy with selectors/policy + M->>K: Resolve workloads + M->>DM: Distribute scheduling intent + DM->>DM: Resolve local process/task identities loop Every api.interval seconds - G->>DM: Fetch PID-level strategies - DM->>G: Return priority / execution_time / pid + G->>DM: Fetch node-local strategies + DM->>G: Return priority / execution_time / task id end - G->>B: Dispatch tasks with selected CPU, vtime, and slice + G->>B: Apply scheduling decision ``` -A PID-level strategy contains: +## TID-aware Node Policy Matching + +Linux schedules **tasks/threads**, not just thread-group leaders. Node policies therefore scan: -```json -{ - "priority": 1, - "execution_time": 20000000, - "pid": 12345 -} +```text +/proc//task/ ``` -- `priority` greater than `0` gives the task priority treatment. -- `execution_time` sets a custom time slice in nanoseconds. -- `pid` identifies the Linux process that receives the policy. +and match each thread's `comm` independently. -## sched_ext Scheduler Internals +For example: -The advanced scheduler is split between BPF and Go: +```text +/proc/3785998/comm = python3.12 +/proc/3785998/task/3786004/comm = EngineCore_DP0 +/proc/3785998/task/3786005/comm = EngineCore_DP1 +``` -- `main.bpf.c` implements the low-level `sched_ext` hooks, dispatch queues, task maps, priority tracking, and ring buffer communication. -- `main.go` loads configuration, initializes the plugin, loads `main.bpf.o`, attaches the scheduler, initializes CPU topology domains, and runs the dispatch loop. -- The plugin layer provides scheduling policy implementations: `gthulhu`, `simple`, and `simple-fifo`. +A node policy matching `^EngineCore(_DP[0-9]+)?$` can target the two worker threads directly even though the process leader is named `python3.12`. -### User-Space Dispatch Loop +The resolved strategy key is the **TID** of the matched worker. This is the entity Linux actually dispatches. -```mermaid -flowchart TD - A[Drain queued tasks from BPF ring buffer] --> B[Select queued task] - B --> C{Task available?} - C -->|No| D[Wait and retry] - C -->|Yes| E[Build dispatched task] - E --> F[Apply priority / vtime] - F --> G[Determine time slice] - G --> H[Select CPU with topology hints] - H --> I{CPU selected?} - I -->|No| D - I -->|Yes| J[Send decision through user ring buffer] - J --> K[Notify completion] - K --> A -``` +This behavior was introduced by the merged thread-aware Decision Maker work in `Gthulhu/Gthulhu#135`. + +## TID-first Lookup with TGID Fallback + +In the user-space scheduling plugin, strategy lookup is now consistent: + +1. prefer an exact **TID** match; +2. if no TID-specific strategy exists, fall back to the **TGID**. + +This allows: + +- a node policy to target one specific worker thread; +- a Pod-level policy keyed by the group leader to still apply across the thread group; +- a TID-specific rule to win when both exist. + +This behavior was introduced by the merged `Gthulhu/plugin#17` change. + +### Current limitation + +The strategy map is still keyed by a bare numeric ID. If a TID-specific target is itself the group leader (`TID == TGID`), sibling threads may resolve the same entry through TGID fallback. A future strategy shape should preserve whether the original match was task-specific or group-wide. + +## Priority and Time-slice Semantics + +The current intended semantics are: + +- `Priority > 0` = **boosting strategy**; +- `Priority == 0` = **non-boosting strategy**; +- `execution_time` = custom time slice in nanoseconds where the scheduler path supports it. + +### User-space mode + +A `Priority == 0` strategy does **not** jump the run queue. It may still carry a custom time slice. -BPF enqueues tasks to user space through a ring buffer. Go drains the queued tasks, asks the active plugin to select work, determines the time slice, picks a CPU, and returns a dispatch decision through the user ring buffer. BPF then performs the final `sched_ext` dispatch. +A `Priority > 0` strategy receives priority treatment through the user-space scheduler's dispatch ordering. -### Priority Handling +### Kernel mode -In user-space scheduler mode, priority is represented by setting a dispatched task's virtual time to the minimum value. BPF tracks priority tasks and can insert them at the head of the dispatch queue or trigger preemption behavior. +Kernel mode uses BPF priority state directly. Because the underlying priority map treats numeric priority `0` as the highest/preemptive level, Gthulhu must **not** insert a non-boosting `Priority == 0` strategy into that map. -In kernel mode, the user-space loop is bypassed for per-task decisions. The Go process watches strategy changes and updates BPF maps through `UpdatePriorityTaskWithPrio` and `RemovePriorityTask`, allowing BPF to dispatch directly in kernel space. +The merged `Gthulhu/Gthulhu#137` fix therefore skips `Priority <= 0` when applying kernel-mode priority state and removes any stale priority entry for that task. + +!!! warning "Kernel-mode slice-only parity" + Kernel mode currently has no separate "custom slice at normal priority" state. Therefore a slice-only (`Priority == 0`) strategy has no scheduling effect in kernel mode instead of being incorrectly promoted. Full parity requires a future qumun/BPF state change. + +## sched_ext Scheduler Internals + +The advanced scheduler is split between Go and BPF: + +- BPF implements `sched_ext` hooks, dispatch queues, task maps, priority state, and ring-buffer communication. +- Go loads configuration, initializes the active scheduler/plugin, attaches the scheduler, and handles control-plane updates. +- User-space mode selects tasks and returns dispatch decisions through ring buffers. +- Kernel mode bypasses per-task user-space selection for the priority path and updates BPF state directly. ## CPU Selection -Gthulhu initializes CPU topology and cache domains before attaching the scheduler. CPU selection prefers locality and idle capacity: +When using the Gthulhu user-space scheduler, CPU selection prefers locality and idle capacity: + +1. reuse the previous CPU when allowed and idle; +2. prefer a fully idle sibling/core when SMT is available; +3. prefer the same L2/L3 cache domain; +4. fall back to another idle CPU; +5. report busy when no suitable CPU is available. + +## Claim2Core: Where the Architecture Is Going + +The next architecture step is to consume actual Kubernetes allocation and compile it into a safe runtime execution plan. + +```text +ResourceClaim allocation + → Pod / cgroup + → TGID / TID / starttime + → proposed execution class + → sched_ext / BPF state + → runtime metrics / workload SLO +``` + +Important boundaries: -1. Reuse the previous CPU when it is allowed and idle. -2. Prefer a fully idle sibling/core when SMT is available. -3. Prefer CPUs in the same L2 or L3 cache domain. -4. Fall back to any idle CPU. -5. Return busy when no suitable CPU is available. +- `ResourceSlice` describes **inventory**; it is not proof that a workload owns a device. +- `ResourceClaim.status.allocation` is the source of truth for actual DRA allocation. +- Kubernetes API objects belong in the control/update path, not the microsecond scheduler hot path. +- cgroup / allocated CPU boundaries are authoritative; Gthulhu must never schedule a task outside them. +- Gthulhu schedules Linux CPU tasks, not CUDA kernels, GPU SMs, MIG, or NIC hardware queues. -The user-space scheduler provides CPU hints, while BPF performs the final dispatch and kick behavior. +See [Claim2Core](claim2core.md) for the roadmap. ## Runtime Configuration -The daemon reads YAML configuration and can also receive runtime configuration from the Decision Maker. +A typical configuration looks like: ```yaml monitor: enabled: true - bpf_object_path: sched_monitor.bpf.o collection_interval_sec: 10 - monitor_all: false - stream_events: false - prometheus_port: 9090 - enable_crd_watcher: true scheduler: slice_ns_default: 20000000 slice_ns_min: 1000000 mode: gthulhu - scheduler_name: "" kernel_mode: false - max_time_watchdog: true api: url: http://127.0.0.1:8080 interval: 5 - public_key_path: ./api/config/jwt_public_key.pem enabled: true auth_enabled: true ``` -Important behavior: +Key points: -- `monitor.enabled` controls the base eBPF metrics collector. -- `scheduler.mode` controls which scheduler process the daemon supervises: `none`, `gthulhu`, `simple`, or `scx`. -- `scheduler.scheduler_name` is required when `scheduler.mode` is `scx`; the daemon validates that `/gthulhu/` is an allowed executable before applying the runtime config. -- `scheduler.kernel_mode` enables experimental BPF-side dispatch. +- `monitor.enabled` controls the eBPF metrics collector; +- `scheduler.mode` selects monitor-only / Gthulhu / upstream scx behavior; +- `scheduler.kernel_mode` enables the experimental BPF-side priority path; - `api.enabled` controls communication with the Decision Maker. -- `api.auth_enabled` enables JWT authentication for scheduler API calls. -- `api.mtls` can enable mutual TLS between the scheduler and API server. - -## Metrics - -Gthulhu reports two families of metrics: - -| Source | Examples | Consumer | -|--------|----------|----------| -| eBPF pod monitor | wait time, runtime, context switches, CPU migrations | Prometheus, Grafana, KEDA | -| sched_ext BSS data | `nr_queued`, `nr_scheduled`, `nr_running`, dispatch counters, congestion counters | Decision Maker API and logs | - -The scheduler periodically reads BSS data from the BPF module. When API communication is enabled, it posts those metrics to the Decision Maker. ## Debugging -Useful commands while developing or operating the scheduler: - ```bash -# Trace BPF debug output -sudo cat /sys/kernel/debug/tracing/trace_pipe - -# Inspect loaded BPF programs and maps sudo bpftool prog show sudo bpftool map show - -# Run the Gthulhu daemon with an explicit config -sudo ./main scheduler -config config/config.yaml +sudo cat /sys/kernel/debug/tracing/trace_pipe ``` -For monitor-only deployments, set `scheduler.mode` to `none` in the runtime configuration. To use an upstream sched_ext scheduler, deploy the scx-flavored image and set `scheduler.mode: scx` with a bundled scheduler name such as `scx_bpfland` or `scx_cake`. +When debugging policy application, always distinguish: + +- TGID vs TID; +- user-space vs kernel mode; +- boosting vs non-boosting strategy; +- intended strategy vs actual BPF state. + +The planned preview/provenance work in `Gthulhu/Gthulhu#134` is intended to make those distinctions directly observable. From 2bc9e8df903bff241c8761fbf1955175efe0ca07 Mon Sep 17 00:00:00 2001 From: gthulhu-work Date: Sat, 5 Sep 2026 18:14:32 +0800 Subject: [PATCH 4/7] docs: document Chinese TID-aware scheduling semantics --- docs/how-it-works.zh.md | 254 ++++++++++++++++++++-------------------- 1 file changed, 128 insertions(+), 126 deletions(-) diff --git a/docs/how-it-works.zh.md b/docs/how-it-works.zh.md index 535fcd8..5ff3e15 100644 --- a/docs/how-it-works.zh.md +++ b/docs/how-it-works.zh.md @@ -2,10 +2,10 @@ Gthulhu 將 Kubernetes 工作負載意圖連接到 Linux 核心排程行為。系統分成兩層： -- **Pod 排程可觀測性**：基礎功能，透過 eBPF monitor 收集排程指標，並匯出 Pod 層級 Prometheus 資料。 -- **自訂 CPU 排程**：進階功能，適用於支援 `sched_ext` 的 Linux 6.12+ 節點，透過使用者空間排程器與 BPF scheduler 套用優先級與時間片策略。 +- **Pod 排程可觀測性**：基礎功能，透過 eBPF monitor 收集排程指標並匯出 Pod 層級 Prometheus 資料。 +- **自訂 CPU 排程**：進階功能，適用於 Linux 6.12+ 與 `sched_ext`，由 user-space 或 kernel scheduler path 套用有邊界的 runtime scheduling policy。 -兩層可以一起執行；若未配置 scheduler mode，Gthulhu 也可以只執行 monitor-only 模式。 +兩層可以一起執行，也可以只執行 monitor-only 模式。 ## 架構 @@ -19,8 +19,6 @@ graph TB M -->|排程意圖| DM1 M -->|排程意圖| DM2 - M -->|Runtime config| DM1 - M -->|Runtime config| DM2 subgraph "節點 1" DM1[Decision Maker] --> MON1[eBPF 指標收集器] @@ -39,43 +37,24 @@ graph TB ### Manager -Manager 是面向使用者的控制平面服務。依目前 API server 實作，它負責： - -- 認證與 token 生命週期：`/api/v1/auth/login`、`/api/v1/auth/refresh`、`/api/v1/auth/logout` -- RBAC 資源：使用者、角色與權限 -- 排程策略：`/api/v1/strategies` -- 排程意圖：`/api/v1/intents/self` -- 依節點查詢 Pod-to-PID：`/api/v1/nodes`、`/api/v1/nodes/:nodeID/pods/pids` -- Pod 排程指標設定與 runtime values：`/api/v1/pod-scheduling-metrics`、`/api/v1/pod-scheduling-metrics/runtime`、`/api/v1/classify` -- Scheduler runtime configuration：`/api/v1/scheduler/runtime-config/apply`、`/api/v1/scheduler/runtime-config/status` - -Manager 會將狀態持久化到 MongoDB，並透過 Kubernetes API 解析工作負載 selector。 +Manager 負責 cluster-level intent 與 control-plane state，透過 Kubernetes API 解析 workload selector，並把 scheduling intent 分發到各節點 Decision Maker。 ### Decision Maker -Decision Maker 以每節點一個服務的形式運作。它接收 Manager 發出的叢集層級意圖，解析成節點本地 Process 資訊，並服務本地 scheduler 與 monitor。 - -重要端點包含： +Decision Maker 每個 Node 一個。它把 cluster-level intent 解析成 node-local Linux task，提供 local scheduling strategies，並服務 monitor / scheduler path。 -- `POST /api/v1/intents` — 接收 Manager 發出的排程意圖 -- `GET /api/v1/scheduling/strategies` — 提供 PID 層級策略給本地 scheduler -- `POST /api/v1/metrics` — 接收 Gthulhu daemon 上報的 scheduler BSS metrics -- `GET /api/v1/pods/pids` — 回傳該節點的 Pod-to-PID mapping -- `POST /api/v1/runtime-config` 與 `GET /api/v1/runtime-config` — 套用與查看 runtime daemon configuration -- `POST /api/v1/auth/token` — 簽發 scheduler 對本地 API 呼叫使用的 token -- `GET /metrics` — 暴露 Decision Maker 的 Prometheus metrics +### Gthulhu daemon -### Gthulhu Daemon +Daemon 可以執行： -Gthulhu 根目錄的 binary 可以執行 monitor、scheduler，或兩者同時執行： - -- **monitor** 由 `monitor.enabled` 啟用，是預設的基礎功能。 -- **scheduler** 在 `scheduler.mode` 設為 `gthulhu`、`simple` 或 `scx` 時啟用。 -- 若 `scheduler.mode` 設為 `none`，daemon 會停留在 monitor-only 模式。 +- monitor-only； +- user-space `sched_ext` scheduling； +- experimental kernel-mode policy application； +- 配置後的 upstream `scx` scheduler。 ## Pod 排程指標流程 -monitor 的設計不依賴 `sched_ext`。它會載入 `sched_monitor.bpf.o`，將 eBPF 程式掛到 scheduler tracepoints，讀取 BPF maps，將 PID 對應回 Kubernetes Pod，最後匯出 Pod 層級 metrics。 +Monitor 不依賴 `sched_ext`。它將 eBPF 程式掛到 scheduler events，將 process/task activity 對應回 Kubernetes Pod，並匯出 runtime、wait time、context switch、run count、CPU migration 等 metrics。 ```mermaid sequenceDiagram @@ -86,18 +65,16 @@ sequenceDiagram participant M as Prometheus C->>W: 依 namespace 與 labels 選擇 Pods - W->>P: 解析符合條件的 Pods 與 Processes - W->>B: 更新 monitored PID/TGID maps - B->>B: 追蹤 sched_switch 與 process_exit events - B->>P: 將 PIDs 對應回 Pods + W->>P: 解析符合條件的 Pods / Processes + W->>B: 更新 monitored identities + B->>B: 追蹤 scheduler events + B->>P: 對應回 Pods B->>M: 在 /metrics 暴露 Pod metrics ``` -collector 追蹤 runtime、wait time、自願/非自願上下文切換、run count、CPU migrations 等訊號。這些 metrics 可供 Prometheus、Grafana dashboard 與 KEDA-based scaling 使用。 - ## 排程策略流程 -排程策略從 Kubernetes 工作負載層級開始，最後變成每個節點上的 PID 層級決策。 +Scheduling strategy 從 Kubernetes/workload intent 開始，最後落成 node-local task decision。 ```mermaid sequenceDiagram @@ -108,144 +85,169 @@ sequenceDiagram participant G as Gthulhu Scheduler participant B as BPF Scheduler - U->>M: 建立包含 selectors 與 policy 的策略 - M->>K: 查詢符合條件的 Pods - M->>DM: 分發排程意圖 - DM->>DM: 將 Pods 解析成本地 PIDs + U->>M: 建立 selector / policy + M->>K: 解析 workloads + M->>DM: 分發 scheduling intent + DM->>DM: 解析本地 process/task identity loop 每 api.interval 秒 - G->>DM: 取得 PID 層級策略 - DM->>G: 回傳 priority / execution_time / pid + G->>DM: 取得 node-local strategies + DM->>G: 回傳 priority / execution_time / task id end - G->>B: 帶著 CPU、vtime 與 slice 決策分派 tasks + G->>B: 套用 scheduling decision ``` -PID 層級策略格式如下： +## TID-aware Node Policy Matching + +Linux 排的是 **task/thread**，不是只排 thread-group leader。因此 node policy 會掃描： -```json -{ - "priority": 1, - "execution_time": 20000000, - "pid": 12345 -} +```text +/proc//task/ ``` -- `priority` 大於 `0` 時，該 task 會獲得優先處理。 -- `execution_time` 代表自訂時間片，單位為奈秒。 -- `pid` 是套用策略的 Linux Process。 +並獨立比對每個 thread 的 `comm`。 -## sched_ext Scheduler 內部設計 +例如： -進階 scheduler 分成 BPF 與 Go 兩部分： +```text +/proc/3785998/comm = python3.12 +/proc/3785998/task/3786004/comm = EngineCore_DP0 +/proc/3785998/task/3786005/comm = EngineCore_DP1 +``` -- `main.bpf.c` 實作低階 `sched_ext` hooks、dispatch queues、task maps、priority tracking 與 ring buffer communication。 -- `main.go` 載入設定、初始化 plugin、載入 `main.bpf.o`、attach scheduler、初始化 CPU topology domains，並執行 dispatch loop。 -- plugin layer 提供排程策略實作：`gthulhu`、`simple`、`simple-fifo`。 +即使 process leader 名稱是 `python3.12`，`^EngineCore(_DP[0-9]+)?$` 仍可以直接命中兩個 worker threads。 -### 使用者空間 Dispatch Loop +解析後的 strategy key 是被命中 worker 的 **TID**，也就是 Linux 真正 dispatch 的 entity。 -```mermaid -flowchart TD - A[從 BPF ring buffer 排出 queued tasks] --> B[選擇 queued task] - B --> C{有 task?} - C -->|否| D[等待後重試] - C -->|是| E[建立 dispatched task] - E --> F[套用 priority / vtime] - F --> G[決定 time slice] - G --> H[依 topology hints 選擇 CPU] - H --> I{選到 CPU?} - I -->|否| D - I -->|是| J[透過 user ring buffer 送回決策] - J --> K[通知完成] - K --> A -``` +這個行為來自已合併的 `Gthulhu/Gthulhu#135`。 + +## TID-first Lookup 與 TGID Fallback + +User-space scheduling plugin 現在使用一致的 lookup semantics： + +1. 先找 exact **TID** match； +2. 沒有 TID-specific strategy 才 fallback 到 **TGID**。 + +因此： + +- node policy 可以只命中單一 worker thread； +- Pod-level policy 仍可透過 TGID 對整個 thread group 生效； +- 當 TID 與 TGID rule 同時存在時，TID-specific rule 優先。 + +這個行為來自已合併的 `Gthulhu/plugin#17`。 + +### 目前限制 + +Strategy map 仍以 bare numeric ID 為 key。如果 target 本身就是 group leader（`TID == TGID`），sibling thread 可能透過 TGID fallback 讀到同一筆 entry。後續應讓 strategy shape 保留它原本是 task-specific 還是 group-wide。 + +## Priority 與 Time-slice Semantics + +目前語意是： + +- `Priority > 0` = **boosting strategy**； +- `Priority == 0` = **non-boosting strategy**； +- `execution_time` = scheduler path 支援時使用的 custom time slice，單位 ns。 + +### User-space mode + +`Priority == 0` 不會跳到 run queue 前面，但仍可攜帶 custom time slice。 -BPF 透過 ring buffer 將 tasks 排入使用者空間。Go 端排出 tasks，交給啟用中的 plugin 選擇工作、決定時間片、選擇 CPU，再透過 user ring buffer 回傳 dispatch decision。BPF 最後執行實際的 `sched_ext` dispatch。 +`Priority > 0` 才會在 user-space scheduler dispatch ordering 中獲得 priority treatment。 -### 優先級處理 +### Kernel mode -在 user-space scheduler mode 中，priority 透過將 dispatched task 的 virtual time 設為最小值來表達。BPF 會追蹤 priority tasks，並可將其插入 dispatch queue 前端或觸發 preemption 行為。 +Kernel mode 直接使用 BPF priority state。底層 priority map 將數值 `0` 解讀成最高 / preemptive priority，因此 Gthulhu **不能**把 non-boosting `Priority == 0` strategy 插進這個 map。 -在 kernel mode 中，每次 task 決策不再經過 user-space loop。Go process 會監看策略變更，並透過 `UpdatePriorityTaskWithPrio` 與 `RemovePriorityTask` 更新 BPF maps，讓 BPF 直接在核心空間 dispatch。 +已合併的 `Gthulhu/Gthulhu#137` 因此會在 kernel-mode priority apply 時跳過 `Priority <= 0`，並移除該 task 可能殘留的 stale priority entry。 + +!!! warning "Kernel mode 的 slice-only parity" + Kernel mode 目前沒有獨立的「normal priority + custom slice」state。因此 slice-only (`Priority == 0`) strategy 在 kernel mode 目前是無效果，而不是被錯誤提升成最高 priority。要做到完整 parity，之後需要修改 qumun/BPF state model。 + +## sched_ext Scheduler 內部設計 + +進階 scheduler 由 Go 與 BPF 共同構成： + +- BPF 實作 `sched_ext` hooks、dispatch queues、task maps、priority state 與 ring-buffer communication。 +- Go 載入設定、初始化 scheduler/plugin、attach scheduler，並處理 control-plane updates。 +- User-space mode 由 Go/plugin 選 task，透過 ring buffer 回傳 dispatch decision。 +- Kernel mode 對 priority path 省略逐 task 的 user-space selection，直接更新 BPF state。 ## CPU 選擇 -Gthulhu 會在 attach scheduler 前初始化 CPU topology 與 cache domains。CPU 選擇偏好 locality 與 idle capacity： +Gthulhu user-space scheduler 的 CPU selection 偏好 locality 與 idle capacity： + +1. 前一次 CPU 仍可用且 idle 時優先重用； +2. SMT 系統優先 fully idle sibling/core； +3. 優先同一 L2/L3 cache domain； +4. fallback 到其他 idle CPU； +5. 沒有適合 CPU 時回報 busy。 + +## Claim2Core：下一階段架構 + +下一步是消費 Kubernetes 的 actual allocation，將它編譯成安全的 runtime execution plan。 + +```text +ResourceClaim allocation + → Pod / cgroup + → TGID / TID / starttime + → proposed execution class + → sched_ext / BPF state + → runtime metrics / workload SLO +``` + +重要邊界： -1. 如果前一次 CPU 允許且 idle，優先重用。 -2. 在 SMT 系統上，優先選擇 fully idle sibling/core。 -3. 優先選擇同一個 L2 或 L3 cache domain 的 CPU。 -4. 最後才選擇任意 idle CPU。 -5. 若沒有合適 CPU，回傳 busy。 +- `ResourceSlice` 描述 **inventory**，不能證明 workload 已取得 device。 +- `ResourceClaim.status.allocation` 才是 DRA actual allocation 的 source of truth。 +- Kubernetes API object 應存在 control/update path，不能放進微秒級 scheduler hot path。 +- cgroup / allocated CPU boundary 是硬邊界，Gthulhu 不得把 task 移出允許集合。 +- Gthulhu 排的是 Linux CPU task，不是 CUDA kernel、GPU SM、MIG 或 NIC hardware queue。 -使用者空間 scheduler 提供 CPU hints，BPF 負責最後 dispatch 與 kick 行為。 +完整 roadmap 請看 [Claim2Core](claim2core.md)。 ## Runtime Configuration -daemon 會讀取 YAML 設定，也可以透過 Decision Maker 接收 runtime configuration。 +典型設定： ```yaml monitor: enabled: true - bpf_object_path: sched_monitor.bpf.o collection_interval_sec: 10 - monitor_all: false - stream_events: false - prometheus_port: 9090 - enable_crd_watcher: true scheduler: slice_ns_default: 20000000 slice_ns_min: 1000000 mode: gthulhu - scheduler_name: "" kernel_mode: false - max_time_watchdog: true api: url: http://127.0.0.1:8080 interval: 5 - public_key_path: ./api/config/jwt_public_key.pem enabled: true auth_enabled: true ``` -重要行為： +重點： -- `monitor.enabled` 控制基礎 eBPF metrics collector。 -- `scheduler.mode` 控制 daemon 要監督哪一種 scheduler process：`none`、`gthulhu`、`simple` 或 `scx`。 -- `scheduler.scheduler_name` 在 `scheduler.mode` 為 `scx` 時必填；daemon 會在套用 runtime config 前確認 `/gthulhu/` 是允許且可執行的 binary。 -- `scheduler.kernel_mode` 啟用實驗性的 BPF-side dispatch。 +- `monitor.enabled` 控制 eBPF metrics collector； +- `scheduler.mode` 控制 monitor-only / Gthulhu / upstream scx 行為； +- `scheduler.kernel_mode` 啟用 experimental BPF-side priority path； - `api.enabled` 控制是否與 Decision Maker 溝通。 -- `api.auth_enabled` 啟用 scheduler API calls 的 JWT authentication。 -- `api.mtls` 可啟用 scheduler 與 API server 之間的 mutual TLS。 - -## Metrics - -Gthulhu 回報兩類 metrics： - -| 來源 | 範例 | 消費者 | -|------|------|--------| -| eBPF pod monitor | wait time、runtime、context switches、CPU migrations | Prometheus、Grafana、KEDA | -| sched_ext BSS data | `nr_queued`、`nr_scheduled`、`nr_running`、dispatch counters、congestion counters | Decision Maker API 與 logs | - -scheduler 會定期從 BPF module 讀取 BSS data。若 API communication 已啟用，會將這些 metrics POST 給 Decision Maker。 ## 除錯 -開發或操作 scheduler 時常用的指令： - ```bash -# 追蹤 BPF debug output -sudo cat /sys/kernel/debug/tracing/trace_pipe - -# 檢查已載入的 BPF programs 與 maps sudo bpftool prog show sudo bpftool map show - -# 使用指定設定啟動 Gthulhu daemon -sudo ./main scheduler -config config/config.yaml +sudo cat /sys/kernel/debug/tracing/trace_pipe ``` -若要部署 monitor-only 模式，請在 runtime configuration 中將 `scheduler.mode` 設為 `none`。若要使用 upstream sched_ext scheduler，請部署 scx-flavored image，並設定 `scheduler.mode: scx` 與 image 內建的 scheduler name，例如 `scx_bpfland` 或 `scx_cake`。 +排查 policy application 時，務必區分： + +- TGID vs TID； +- user-space vs kernel mode； +- boosting vs non-boosting strategy； +- intended strategy vs actual BPF state。 + +`Gthulhu/Gthulhu#134` 規劃中的 preview/provenance 功能，就是要讓這些差異可以被直接觀測與驗證。 From d07c73ab46492e5cc4cf558bcdfab7f9d5edb784 Mon Sep 17 00:00:00 2001 From: gthulhu-work Date: Sat, 5 Sep 2026 18:14:58 +0800 Subject: [PATCH 5/7] docs: add Claim2Core roadmap page --- docs/claim2core.md | 213 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 213 insertions(+) create mode 100644 docs/claim2core.md diff --git a/docs/claim2core.md b/docs/claim2core.md new file mode 100644 index 0000000..aa4b8f1 --- /dev/null +++ b/docs/claim2core.md @@ -0,0 +1,213 @@ +# Claim2Core + +> **DRA chooses what and where; Gthulhu controls how it actually runs.** + +Claim2Core is the Gthulhu roadmap for connecting Kubernetes resource allocation to Linux task scheduling. + +The core problem is simple: **allocated resources do not automatically become delivered performance**. + +A workload may receive a GPU, NIC, CPU set, NUMA domain, or other device and still miss its latency/throughput target because its host-side Linux tasks are delayed, placed poorly, or starved by CPU contention. + +## Responsibility Boundary + +```text +Kueue / Workload API + admission, quota, fair sharing + │ + ▼ +kube-scheduler / DRA + Node + device + topology allocation + │ + ▼ +Gthulhu Runtime Plane + Claim → Pod/cgroup → TGID/TID + │ + ▼ +sched_ext + eBPF + runtime policy + verification + │ + ▼ +Delivered workload SLO +``` + +| Layer | Typical timescale | Core question | +|---|---:|---| +| Kueue / workload admission | seconds → minutes | Can this workload start now? | +| kube-scheduler / DRA | milliseconds → seconds | Which Node/device/topology does it get? | +| Gthulhu / `sched_ext` | microseconds → milliseconds | Which workload threads run when and on which CPUs? | + +## Source of Truth + +The important correctness distinction is: + +- `ResourceSlice` = **inventory**; +- `ResourceClaim.status.allocation` = **actual allocation**. + +The target lineage is: + +```text +Workload / PodGroup UID + → Pod UID + → ResourceClaim UID + generation + → allocated driver / pool / device + → NUMA / PCIe / network topology + → Pod cgroup + → TGID / TID / starttime + → sched_ext DSQ / BPF-map entry + → runtime metrics + → workload SLO +``` + +## Implementation Order + +### 1. Correct DRA semantics + +- use modern `resource.k8s.io/v1` semantics; +- handle all supported ResourceSlice node-selection forms; +- separate DeviceClass from driver identity; +- keep inventory and allocation code paths clearly separate. + +Tracking: [Gthulhu/Gthulhu#133](https://github.com/Gthulhu/Gthulhu/issues/133) + +### 2. Read-only ResourceClaim observer + +Observe allocated claims and build workload-to-device binding without changing scheduler behavior. + +The observer should maintain identities such as: + +```go +type ClaimDeviceBinding struct { + ClaimUID types.UID + ClaimGeneration int64 + PodUID types.UID + NodeName string + Driver string + Pool string + Device string + NUMANodes []int + PCIeRoot string + PCIBusID string +} +``` + +Kubernetes API state belongs in the control/update path. Microsecond-level scheduling must use node-local cached state. + +### 3. Claim-to-Task preview and provenance + +Before writing scheduler state, Gthulhu should be able to explain: + +```text +Claim → Pod → cgroup → TGID/TID/starttime → proposed runtime policy +``` + +The preview path must be read-only and should expose matched tasks, conflicts, warnings, policy generation, and intended-vs-actual runtime state. + +Tracking: [Gthulhu/Gthulhu#134](https://github.com/Gthulhu/Gthulhu/issues/134) + +The TID-aware groundwork is already merged in [Gthulhu/Gthulhu#135](https://github.com/Gthulhu/Gthulhu/pull/135). + +### 4. Static DRAExecutionPolicy + +The user should express portable intent instead of raw scheduler internals. + +Example direction: + +```yaml +apiVersion: scheduling.gthulhu.io/v1alpha1 +kind: DRAExecutionPolicy +metadata: + name: llm-decode +spec: + workloadSelector: + matchLabels: + llm-d.ai/role: decode + resourceClaims: + - gpu + - rdma + - cpu + topology: + respectAllocatedCPUSet: true + preferSameNUMA: true + preferSamePCIeRoot: true + taskRoles: + - name: decode + selector: + registeredRole: decode + executionClass: latency-critical + safety: + requirePreview: true + maxBoostDuration: 30s + failClosedOnStaleClaim: true + fallback: default-scheduler +``` + +The controller compiles this intent into concrete DSQ / slice / weight / locality decisions based on the allocated cpuset and topology. + +### 5. One workload adapter + +Prove the model on one workload before attempting generic automatic classification. + +Two high-value paths: + +- **CPU DRA × Gthulhu × free5GC/UPF** — fastest credible end-to-end validation; +- **GPU + RDMA + CPU DRA × LLM phase-aware scheduling** — highest research upside. + +Task-role discovery should mature in this order: + +1. explicit role hint; +2. workload-specific adapter; +3. eBPF/uprobes/activity-based classification with confidence. + +### 6. Closed-loop runtime controller + +Only after static policy and provenance are trustworthy should Gthulhu adapt policy from runtime signals. + +Start with bounded, explainable rules, not unconstrained ML/RL. + +## Correctness Invariants + +Claim2Core needs correctness properties, not just benchmark wins. + +- Claim deallocation must not leave stale policy that can affect a future task. +- TID reuse must not cause an old `(TID, starttime)` policy to affect a new task. +- Generation rollback must not resurrect invalid execution state. +- Policy generation must be monotonic. +- Actual BPF state must not contain unexplained extra entries. +- DRA/cgroup cpuset is a hard boundary. +- Decision Maker restart must reconstruct only currently valid state. + +## Experimental Methodology + +Do not compare only `default scheduler vs Gthulhu`. + +Use a 2×2 design: + +| Group | DRA topology-aware allocation | Gthulhu runtime scheduling | +|---|---|---| +| A | off | off | +| B | on | off | +| C | off | on | +| D | on | on | + +This separates allocation benefit, runtime scheduling benefit, and interaction between them. + +### Example KPIs + +**LLM**: TTFT, ITL, tokens/s, GPU idle gap. + +**5G/UPF**: p50/p95/p99/p99.9 RTT, jitter, packet loss. + +**Runtime**: runnable-to-running latency, context switches, CPU/NUMA migrations, device-local CPU time, scheduler apply latency. + +**Safety**: background slowdown, starvation duration, stale BPF entries, task reuse correctness, restart recovery, rollback latency. + +## Hard Boundaries + +- Gthulhu is **not a GPU scheduler**. It schedules Linux CPU tasks, not CUDA kernels, GPU SMs, MIG, or NIC hardware queues. +- ResourceClaim/ResourceSlice API calls do **not** belong in the microsecond scheduler hot path. +- CPU DRA/cgroups/kubelet define the allowed CPU envelope; Gthulhu optimizes only inside it. +- Multi-tenant policy must be bounded, scoped, auditable, and deterministic. + +## Roadmap Discussion + +The living roadmap is [Gthulhu/Gthulhu#141](https://github.com/Gthulhu/Gthulhu/issues/141). From 7631cbfebf282ec981ae4df9c110d02f6813782d Mon Sep 17 00:00:00 2001 From: gthulhu-work Date: Sat, 5 Sep 2026 18:15:20 +0800 Subject: [PATCH 6/7] docs: add Chinese Claim2Core roadmap page --- docs/claim2core.zh.md | 209 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 209 insertions(+) create mode 100644 docs/claim2core.zh.md diff --git a/docs/claim2core.zh.md b/docs/claim2core.zh.md new file mode 100644 index 0000000..626e04e --- /dev/null +++ b/docs/claim2core.zh.md @@ -0,0 +1,209 @@ +# Claim2Core + +> **DRA chooses what and where; Gthulhu controls how it actually runs.** + +Claim2Core 是 Gthulhu 用來連接 Kubernetes resource allocation 與 Linux task scheduling 的核心 roadmap。 + +最重要的問題是：**allocated resource 不等於 delivered performance**。 + +Workload 即使已經取得 GPU、NIC、CPU set、NUMA domain 或其他 device，仍可能因為 host-side Linux tasks 被 CPU contention 延遲、locality 不佳或 starvation，而無法達成 latency / throughput SLO。 + +## 責任邊界 + +```text +Kueue / Workload API + admission, quota, fair sharing + │ + ▼ +kube-scheduler / DRA + Node + device + topology allocation + │ + ▼ +Gthulhu Runtime Plane + Claim → Pod/cgroup → TGID/TID + │ + ▼ +sched_ext + eBPF + runtime policy + verification + │ + ▼ +Delivered workload SLO +``` + +| 層級 | 典型時間尺度 | 核心問題 | +|---|---:|---| +| Kueue / workload admission | 秒 → 分鐘 | 這個 Workload 現在能不能開始？ | +| kube-scheduler / DRA | 毫秒 → 秒 | 要用哪個 Node/device/topology？ | +| Gthulhu / `sched_ext` | 微秒 → 毫秒 | Workload 裡哪些 threads 何時、在哪些 CPU 上執行？ | + +## Source of Truth + +最重要的 correctness 區分： + +- `ResourceSlice` = **inventory**； +- `ResourceClaim.status.allocation` = **actual allocation**。 + +目標 lineage： + +```text +Workload / PodGroup UID + → Pod UID + → ResourceClaim UID + generation + → allocated driver / pool / device + → NUMA / PCIe / network topology + → Pod cgroup + → TGID / TID / starttime + → sched_ext DSQ / BPF-map entry + → runtime metrics + → workload SLO +``` + +## 實作順序 + +### 1. 修正 DRA 基礎語意 + +- 使用現代 `resource.k8s.io/v1` semantics； +- 支援 ResourceSlice 的各種 node-selection forms； +- 分清 DeviceClass 與 driver identity； +- 明確分離 inventory 與 allocation code paths。 + +Tracking: [Gthulhu/Gthulhu#133](https://github.com/Gthulhu/Gthulhu/issues/133) + +### 2. Read-only ResourceClaim observer + +先建立 workload-to-device binding，不改 scheduler 行為。 + +```go +type ClaimDeviceBinding struct { + ClaimUID types.UID + ClaimGeneration int64 + PodUID types.UID + NodeName string + Driver string + Pool string + Device string + NUMANodes []int + PCIeRoot string + PCIBusID string +} +``` + +Kubernetes API state 應存在 control/update path；微秒級 scheduling 必須使用 node-local cached state。 + +### 3. Claim-to-Task preview / provenance + +在寫入 scheduler state 前，Gthulhu 應先能解釋： + +```text +Claim → Pod → cgroup → TGID/TID/starttime → proposed runtime policy +``` + +Preview 必須 read-only，並回傳 matched tasks、conflicts、warnings、policy generation，以及 intended-vs-actual runtime state。 + +Tracking: [Gthulhu/Gthulhu#134](https://github.com/Gthulhu/Gthulhu/issues/134) + +TID-aware groundwork 已由 [Gthulhu/Gthulhu#135](https://github.com/Gthulhu/Gthulhu/pull/135) 合併完成。 + +### 4. Static DRAExecutionPolicy + +使用者應表達 portable intent，而不是直接填 scheduler internals。 + +```yaml +apiVersion: scheduling.gthulhu.io/v1alpha1 +kind: DRAExecutionPolicy +metadata: + name: llm-decode +spec: + workloadSelector: + matchLabels: + llm-d.ai/role: decode + resourceClaims: + - gpu + - rdma + - cpu + topology: + respectAllocatedCPUSet: true + preferSameNUMA: true + preferSamePCIeRoot: true + taskRoles: + - name: decode + selector: + registeredRole: decode + executionClass: latency-critical + safety: + requirePreview: true + maxBoostDuration: 30s + failClosedOnStaleClaim: true + fallback: default-scheduler +``` + +Controller 再依 allocated cpuset、NUMA/LLC/topology 等條件，編譯成 DSQ / slice / weight / locality decisions。 + +### 5. 先做一個 workload adapter + +先證明整個模型可行，再做 generic automatic classification。 + +兩條高價值路線： + +- **CPU DRA × Gthulhu × free5GC/UPF**：最快建立可信 end-to-end demo； +- **GPU + RDMA + CPU DRA × LLM phase-aware scheduling**：研究影響力最高。 + +Task-role discovery 建議依序演進： + +1. explicit role hint； +2. workload-specific adapter； +3. eBPF/uprobes/activity-based classification + confidence。 + +### 6. Closed-loop runtime controller + +只有 static policy + provenance 已可信後，才讓 Gthulhu 根據 runtime signal 自動調整 policy。 + +第一版應使用有上下界、可解釋的 rule，不要直接上 unconstrained ML/RL。 + +## Correctness Invariants + +Claim2Core 不只需要 benchmark，也需要可驗證的 correctness： + +- Claim deallocation 後不能留下會影響新 task 的 stale policy； +- TID reuse 不得讓舊 `(TID, starttime)` policy 套到新 task； +- generation rollback 不得 resurrect invalid execution state； +- policy generation 必須 monotonic； +- actual BPF state 不得存在無法解釋的額外 entry； +- DRA/cgroup cpuset 是 hard boundary； +- Decision Maker restart 只能重建目前仍有效的 state。 + +## 實驗方法 + +不要只做 `default scheduler vs Gthulhu`。 + +應採 2×2： + +| 組別 | DRA topology-aware allocation | Gthulhu runtime scheduling | +|---|---|---| +| A | off | off | +| B | on | off | +| C | off | on | +| D | on | on | + +這樣才能分離 allocation benefit、runtime scheduling benefit，以及兩者的 interaction。 + +### KPI 範例 + +**LLM**：TTFT、ITL、tokens/s、GPU idle gap。 + +**5G/UPF**：p50/p95/p99/p99.9 RTT、jitter、packet loss。 + +**Runtime**：runnable-to-running latency、context switches、CPU/NUMA migrations、device-local CPU time、scheduler apply latency。 + +**Safety**：background slowdown、starvation duration、stale BPF entries、task reuse correctness、restart recovery、rollback latency。 + +## Hard Boundaries + +- Gthulhu **不是 GPU scheduler**；它排 Linux CPU task，不直接排 CUDA kernel、GPU SM、MIG 或 NIC hardware queue。 +- ResourceClaim/ResourceSlice API call 不能放進微秒級 scheduler hot path。 +- CPU DRA/cgroup/kubelet 決定可用 CPU envelope，Gthulhu 只能在這個範圍內最佳化。 +- Multi-tenant policy 必須 bounded、scoped、auditable、deterministic。 + +## Roadmap 討論 + +Living roadmap 在 [Gthulhu/Gthulhu#141](https://github.com/Gthulhu/Gthulhu/issues/141)。 From 80489175fabadc2d26f74c95dea35786c27cff29 Mon Sep 17 00:00:00 2001 From: gthulhu-work Date: Sat, 5 Sep 2026 18:15:32 +0800 Subject: [PATCH 7/7] docs: add Claim2Core to site navigation --- mkdocs.yml | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/mkdocs.yml b/mkdocs.yml index efff65b..62c3dbd 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,5 +1,5 @@ site_name: Gthulhu -site_description: Cloud-native workload optimization scheduler based on Linux Scheduler Extension +site_description: Cloud-native runtime scheduling from Kubernetes allocation to Linux task execution site_url: https://gthulhu.org repo_url: https://github.com/Gthulhu/Gthulhu edit_uri: https://github.com/Gthulhu/docs/blob/main/docs/ @@ -16,8 +16,10 @@ nav: - Debugging with gthulhu-cli: cli.md - API Reference: https://pkg.go.dev/github.com/Gthulhu/qumun/goland_core - Troubleshooting: faq.md - - Blog: + - Architecture: - How It Works: how-it-works.md + - Claim2Core: claim2core.md + - Blog: - Improving Network Performance with Custom eBPF-based Schedulers: https://free5gc.org/blog/20250726/index.en/ - Implementing GTP-driven Automatic Scheduling Optimization with eBPF-based Scheduler: https://free5gc.org/blog/20251126/20251126/ - Hardening Gthulhu Network Security with Istio Ambient Mode: istio.md @@ -32,7 +34,6 @@ theme: custom_dir: docs/overrides language: en logo: https://raw.githubusercontent.com/Gthulhu/Gthulhu/main/assets/logo.png - # favicon: assets/favicon.png palette: - media: "(prefers-color-scheme: light)" scheme: default @@ -59,7 +60,7 @@ theme: plugins: - search: - lang: + lang: - en - zh - i18n: @@ -80,13 +81,18 @@ plugins: Get Started: 開始使用 Installation: 安裝 K8s Deployment: K8s 部署 - Pod-Level Scheduling Metrics: Pod 級排程指標 + Configuring the scheduling policies: 設定排程策略 Loading sched_ext schedulers: 載入 sched_ext 排程器 + Pod-Level Scheduling Metrics: Pod 級排程指標 + Working with multi-node cluster: 多節點叢集 + Debugging with gthulhu-cli: 使用 gthulhu-cli 除錯 + Architecture: 架構 How It Works: 運作原理 - Project Goals: 專案目標 - Development History: 開發歷史 + Claim2Core: Claim2Core API Reference: API 參考 - FAQ: 常見問題 + Troubleshooting: 疑難排解 + Blog: 文章 + Tutorials: 教學 Demo: Demo 系統 Mentioned: 相關報導 Contributing: 貢獻指南