Summary
The cgrp_local_storage/yes_rcu_lock and cgrp_local_storage/cgroup_iter_sleepable subtests fail intermittently in BPF CI when running with parallel execution (-j). The root cause is transient rqspinlock timeout in bpf_local_storage_update(), causing bpf_cgrp_storage_get() to return NULL.
Failure Details
- Test / Component:
cgrp_local_storage/yes_rcu_lock, cgrp_local_storage/cgrp1_iter_sleepable
- Frequency: Occasional — observed in ~2-3 out of 20 CI runs across independent PRs
- Failure mode: Flaky —
bpf_cgrp_storage_get returns NULL due to lock contention
- Affected architectures: x86_64 (observed in
test_progs_no_alu32 with both gcc-15 and llvm-21)
- CI runs observed:
Root Cause Analysis
Commit 8dabe34b9d5b ("bpf: Change local_storage->lock and b->lock to rqspinlock") by Amery Hung converted bpf_local_storage locks from raw_spin_lock to rqspinlock (resilient spinlock). Unlike traditional spinlocks, rqspinlocks have a 250ms timeout (RES_DEF_TIMEOUT in include/asm-generic/rqspinlock.h:62) and can fail with -EDEADLK or -ETIMEDOUT.
In bpf_local_storage_update() (kernel/bpf/bpf_local_storage.c:611):
err = raw_res_spin_lock_irqsave(&local_storage->lock, flags);
if (err)
goto free_selem;When this lock times out (due to contention from other tests running in parallel and accessing the same cgroup's local storage), the function returns an error pointer. The caller bpf_cgrp_storage_get() then returns NULL to the BPF program.
The CI log from run 29586716132 confirms this with: update_err: actual -11 != expected 0 (where -11 = -EAGAIN from the cmpxchg race path at bpf_local_storage.c:528 or the empty-list check at line 622).
A diagnostic tracing patch was merged as commit 7c528b364bd8 ("selftests/bpf: Trace bpf_local_storage_update to debug flaky local storage tests") in April 2026 by the same developer, confirming the issue was known but unresolved. No fix has followed in the 3 months since.
Proposed Fix
The attached patch (0001-selftests-bpf-Fix-flaky-cgrp_local_storage-subtests.patch) adds retry logic to both affected subtests:
-
test_yes_rcu_lock: Retries syscall(SYS_getpgid) up to 5 times with 1ms delays until cgroup_id is set by the BPF program.
-
test_cgroup_iter_sleepable: Retries the iterator creation and read cycle up to 5 times with 1ms delays, resetting update_err and cgroup_id between attempts.
This is a test-level fix. A complementary kernel-side fix could add retry logic in bpf_local_storage_update() for the specific -EAGAIN case from bpf_local_storage_alloc() (cmpxchg race, line 577-581), since that failure is always transient — the storage was just created by another thread.
Impact
Without this fix, the cgrp_local_storage test will continue to flake in CI parallel mode, blocking unrelated PRs. The flakiness rate is approximately 10-15% of runs (2-3 failures out of 20 examined runs). The retry approach is conservative (5 attempts × 1ms = 5ms worst case) and won't mask genuine regressions since a real bug would fail all 5 retries.
References
8dabe34b9d5b bpf: Change local_storage->lock and b->lock to rqspinlock7c528b364bd8 selftests/bpf: Trace bpf_local_storage_update to debug flaky local storage tests- Lore thread: https://lore.kernel.org/bpf/20260417233631.1443199-1-ameryhung@gmail.com/ (diagnostic patch)
include/asm-generic/rqspinlock.h:62 — RES_DEF_TIMEOUT = 250mskernel/bpf/bpf_local_storage.c:611 — lock timeout propagation path
Summary
The
cgrp_local_storage/yes_rcu_lockandcgrp_local_storage/cgroup_iter_sleepablesubtests fail intermittently in BPF CI when running with parallel execution (-j). The root cause is transient rqspinlock timeout inbpf_local_storage_update(), causingbpf_cgrp_storage_get()to return NULL.Failure Details
cgrp_local_storage/yes_rcu_lock,cgrp_local_storage/cgrp1_iter_sleepablebpf_cgrp_storage_getreturns NULL due to lock contentiontest_progs_no_alu32with both gcc-15 and llvm-21)yes_rcu_lockFAIL, BTF unique field PR)cgrp1_iter_sleepableFAIL, arm64 cast_user PR)task_local_dataFAIL, same root cause, linux-next merge)Root Cause Analysis
Commit
8dabe34b9d5b("bpf: Change local_storage->lock and b->lock to rqspinlock") by Amery Hung convertedbpf_local_storagelocks fromraw_spin_locktorqspinlock(resilient spinlock). Unlike traditional spinlocks, rqspinlocks have a 250ms timeout (RES_DEF_TIMEOUTininclude/asm-generic/rqspinlock.h:62) and can fail with-EDEADLKor-ETIMEDOUT.In
bpf_local_storage_update()(kernel/bpf/bpf_local_storage.c:611):When this lock times out (due to contention from other tests running in parallel and accessing the same cgroup's local storage), the function returns an error pointer. The caller
bpf_cgrp_storage_get()then returns NULL to the BPF program.The CI log from run 29586716132 confirms this with:
update_err: actual -11 != expected 0(where -11 = -EAGAIN from the cmpxchg race path atbpf_local_storage.c:528or the empty-list check at line 622).A diagnostic tracing patch was merged as commit
7c528b364bd8("selftests/bpf: Trace bpf_local_storage_update to debug flaky local storage tests") in April 2026 by the same developer, confirming the issue was known but unresolved. No fix has followed in the 3 months since.Proposed Fix
The attached patch (
0001-selftests-bpf-Fix-flaky-cgrp_local_storage-subtests.patch) adds retry logic to both affected subtests:test_yes_rcu_lock: Retriessyscall(SYS_getpgid)up to 5 times with 1ms delays untilcgroup_idis set by the BPF program.test_cgroup_iter_sleepable: Retries the iterator creation and read cycle up to 5 times with 1ms delays, resettingupdate_errandcgroup_idbetween attempts.This is a test-level fix. A complementary kernel-side fix could add retry logic in
bpf_local_storage_update()for the specific-EAGAINcase frombpf_local_storage_alloc()(cmpxchg race, line 577-581), since that failure is always transient — the storage was just created by another thread.Impact
Without this fix, the
cgrp_local_storagetest will continue to flake in CI parallel mode, blocking unrelated PRs. The flakiness rate is approximately 10-15% of runs (2-3 failures out of 20 examined runs). The retry approach is conservative (5 attempts × 1ms = 5ms worst case) and won't mask genuine regressions since a real bug would fail all 5 retries.References
8dabe34b9d5bbpf: Change local_storage->lock and b->lock to rqspinlock7c528b364bd8selftests/bpf: Trace bpf_local_storage_update to debug flaky local storage testsinclude/asm-generic/rqspinlock.h:62— RES_DEF_TIMEOUT = 250mskernel/bpf/bpf_local_storage.c:611— lock timeout propagation path