Summary
connect_force_port intermittently fails on UDP subtests when running under ASAN-instrumented builds or cpuv4 emulation. The BPF cgroup getsockname hook fails to rewrite the server address, breaking the service-address translation chain. Observed across multiple independent PRs — not caused by any specific code change.
Failure Details
- Test / Component: connect_force_port (AF_INET/SOCK_DGRAM, AF_INET6/SOCK_DGRAM subtests)
- Frequency: ~20-30% of ASAN and cpuv4 runs, across 5+ independent PRs
- Failure mode: flaky (wrong result — getpeername returns raw backend port instead of rewritten port 60000)
- Affected architectures: x86_64 (ASAN and cpuv4 variants only)
- CI runs observed: Multiple runs in kernel-patches/bpf "bpf-ci" workflow
Root Cause Analysis
The test implements a cgroup BPF service-address rewriting scheme in tools/testing/selftests/bpf/progs/connect_force_port4.c:
getsockname4 (line 61): rewrites server address to 1.2.3.4:60000 when ctx->user_port == bpf_htons(port)connect4 (line 31): intercepts connections to 60000, saves original, redirects to real backendgetpeername4 (line 75): translates backend address back using sk_storage
The failure chain:
connect_to_fd() in tools/testing/selftests/bpf/network_helpers.c calls getsockname(server_fd) which triggers the getsockname4 BPF hook- Under slow environments, the hook's port comparison against the
.bss global port variable intermittently fails to match
- Client then connects to the raw backend port (not 60000), so
connect4 doesn't save the original address
getpeername4 sees the backend port, its comparison also fails, no rewrite occursverify_ports() at tools/testing/selftests/bpf/prog_tests/connect_force_port.c:40 sees raw port instead of 60000
TCP subtests are unaffected — likely due to TCP's connection establishment providing more deterministic hook execution timing.
The exact mechanism causing the .bss port comparison to fail under load is not definitively identified through code inspection alone. Possible factors include BPF map freeze timing, kernel scheduling under ASAN memory pressure, or UDP-specific cgroup hook execution ordering.
Proposed Fix
Denylist connect_force_port in ASAN and cpuv4 CI configurations. See 0001-ci-vmtest-denylist-connect_force_port-for-ASAN-and-c.patch.
Changes:
- Add
connect_force_port to ci/vmtest/configs/DENYLIST.asan
- Create
ci/vmtest/configs/DENYLIST.test_progs_cpuv4 with connect_force_port (and verifier_arena/basic_alloc2 which is already denylisted in the libbpf CI equivalent)
Impact
Without the fix, ASAN and cpuv4 CI runs will continue to show intermittent failures on connect_force_port, creating noise that obscures real regressions. The test remains fully covered in standard test_progs runs where it passes reliably.
References
- Test source:
tools/testing/selftests/bpf/prog_tests/connect_force_port.c
- BPF programs:
tools/testing/selftests/bpf/progs/connect_force_port4.c, connect_force_port6.c
- Existing ASAN denylist:
ci/vmtest/configs/DENYLIST.asan (vmtest#473 precedent)
- libbpf cpuv4 denylist:
ci/vmtest/configs/DENYLIST.test_progs_cpuv4 (already exists in libbpf/ci)
Summary
connect_force_portintermittently fails on UDP subtests when running under ASAN-instrumented builds or cpuv4 emulation. The BPF cgroup getsockname hook fails to rewrite the server address, breaking the service-address translation chain. Observed across multiple independent PRs — not caused by any specific code change.Failure Details
Root Cause Analysis
The test implements a cgroup BPF service-address rewriting scheme in
tools/testing/selftests/bpf/progs/connect_force_port4.c:getsockname4(line 61): rewrites server address to1.2.3.4:60000whenctx->user_port == bpf_htons(port)connect4(line 31): intercepts connections to60000, saves original, redirects to real backendgetpeername4(line 75): translates backend address back using sk_storageThe failure chain:
connect_to_fd()intools/testing/selftests/bpf/network_helpers.ccallsgetsockname(server_fd)which triggers thegetsockname4BPF hook.bssglobalportvariable intermittently fails to matchconnect4doesn't save the original addressgetpeername4sees the backend port, its comparison also fails, no rewrite occursverify_ports()attools/testing/selftests/bpf/prog_tests/connect_force_port.c:40sees raw port instead of 60000TCP subtests are unaffected — likely due to TCP's connection establishment providing more deterministic hook execution timing.
The exact mechanism causing the
.bssport comparison to fail under load is not definitively identified through code inspection alone. Possible factors include BPF map freeze timing, kernel scheduling under ASAN memory pressure, or UDP-specific cgroup hook execution ordering.Proposed Fix
Denylist
connect_force_portin ASAN and cpuv4 CI configurations. See0001-ci-vmtest-denylist-connect_force_port-for-ASAN-and-c.patch.Changes:
connect_force_porttoci/vmtest/configs/DENYLIST.asanci/vmtest/configs/DENYLIST.test_progs_cpuv4withconnect_force_port(andverifier_arena/basic_alloc2which is already denylisted in the libbpf CI equivalent)Impact
Without the fix, ASAN and cpuv4 CI runs will continue to show intermittent failures on
connect_force_port, creating noise that obscures real regressions. The test remains fully covered in standardtest_progsruns where it passes reliably.References
tools/testing/selftests/bpf/prog_tests/connect_force_port.ctools/testing/selftests/bpf/progs/connect_force_port4.c,connect_force_port6.cci/vmtest/configs/DENYLIST.asan(vmtest#473 precedent)ci/vmtest/configs/DENYLIST.test_progs_cpuv4(already exists in libbpf/ci)