Summary
The struct_ops_assoc/st_ops_assoc_in_timer_no_uref selftest flakily fails on s390x with timer_test_1_ret returning 1234 instead of the expected -1. The root cause is a timing assumption: the test expects the struct_ops map to be freed within 500ms, but the multi-stage deferred BPF program free chain (RCU grace period + work queue + map free) can exceed 500ms under QEMU s390x emulation. A similar issue on PowerPC (timer never fires) has been reported on lore but is not yet resolved.
Failure Details
- Test / Component:
struct_ops_assoc/st_ops_assoc_in_timer_no_uref in tools/testing/selftests/bpf/prog_tests/test_struct_ops_assoc.c
- Frequency: Occasional — observed in 2 of ~50 recent failed CI runs (across independent PRs, ~1 month apart)
- Failure mode: Wrong result (flaky) —
timer_test_1_ret is 1234 (MAP_MAGIC) instead of expected -1; deterministic within affected VM instances (fails on retry too)
- Affected architectures: s390x (QEMU emulation). PowerPC reports a related hang on lore.
- CI runs observed:
- Run 34860837587 (2026-09-14, "mm: make VMA flag semantics explicit") — s390x gcc-15 test_progs_no_alu32
- Run 32173885559 (2026-08-18, "netdev CI testing") — s390x gcc-15 test_progs_cpuv4
Root Cause Analysis
The test verifies that after detaching a struct_ops and closing all FDs, a BPF timer callback cannot access the freed struct_ops. The test flow:
- Associates
syscall_prog with st_ops_map — takes a map refcount via bpf_map_inc() (kernel/bpf/bpf_struct_ops.c:1514)
- Attaches struct_ops via link — takes another map refcount via
bpf_map_get() (kernel/bpf/bpf_struct_ops.c:1445)
- Runs
syscall_prog, which invokes test_1() starting a 500ms BPF timer
- Destroys link →
bpf_struct_ops_map_link_detach() drops link's map ref (kernel/bpf/bpf_struct_ops.c:1409)
- Closes
syscall_prog FD → program free is deferred through two asynchronous stages
- Closes
st_ops_map FD → drops FD's map ref
The map cannot be freed until all refcounts drop to zero. The syscall_prog association refcount (from step 1) is only released inside bpf_prog_free_deferred() (kernel/bpf/core.c:3073), which runs after:
call_rcu() in __bpf_prog_put_noref() (kernel/bpf/syscall.c:2467) — waits for an RCU grace periodschedule_work() in bpf_prog_free() (kernel/bpf/core.c:3106) — waits for work queue scheduling
Only after both stages complete does bpf_prog_disassoc_struct_ops() call bpf_map_put() (kernel/bpf/bpf_struct_ops.c:1534), which drops the last map ref, triggering bpf_struct_ops_map_free() → bpf_struct_ops_map_dissoc_progs() (kernel/bpf/bpf_struct_ops.c:1038) → sets test_1->aux->st_ops_assoc = NULL.
On s390x under QEMU emulation, the combined delay of the RCU GP + work queue exceeds 500ms, so the timer fires while the map is still alive and test_1->aux->st_ops_assoc is still valid. The kfunc bpf_kfunc_multi_st_ops_test_1_assoc() successfully calls test_1() and returns 1234 instead of -1.
Proposed Fix
Increase timer_ns from 500ms to 2000ms to give the deferred free chain sufficient time on slow emulated platforms. See attached patch 0001-selftests-bpf-increase-struct_ops_assoc-timer-delay.patch.
A more robust alternative (not in patch): save the map ID before closing FDs, then poll with bpf_map_get_fd_by_id() to confirm the map is actually freed before relying on the timer result. This avoids hardcoding any timeout but requires more test code changes.
Impact
False CI failures on s390x across unrelated PRs. When triggered, the failure is deterministic within the VM instance (fails on retry), causing the entire s390x test suite to be marked as failed. This creates misleading signal for patch authors whose changes are unrelated to struct_ops.
References
- Commit
0e841d19263a ("selftests/bpf: Test getting associated struct_ops in timer callback") — introduced the test
- Commit
bd06b977e02d ("selftests/bpf: Migrate struct_ops_assoc test to KF_IMPLICIT_ARGS") — most recent change
- Lore: struct_ops_assoc test hangs on PowerPC — related issue on PowerPC (timer never fires), reported 2026-05-29 by Venkat Rao Bagalkote, unresolved
- Author of original test: Amery Hung ameryhung@gmail.com
Summary
The
struct_ops_assoc/st_ops_assoc_in_timer_no_urefselftest flakily fails on s390x withtimer_test_1_retreturning 1234 instead of the expected -1. The root cause is a timing assumption: the test expects the struct_ops map to be freed within 500ms, but the multi-stage deferred BPF program free chain (RCU grace period + work queue + map free) can exceed 500ms under QEMU s390x emulation. A similar issue on PowerPC (timer never fires) has been reported on lore but is not yet resolved.Failure Details
struct_ops_assoc/st_ops_assoc_in_timer_no_urefintools/testing/selftests/bpf/prog_tests/test_struct_ops_assoc.ctimer_test_1_retis 1234 (MAP_MAGIC) instead of expected -1; deterministic within affected VM instances (fails on retry too)Root Cause Analysis
The test verifies that after detaching a struct_ops and closing all FDs, a BPF timer callback cannot access the freed struct_ops. The test flow:
syscall_progwithst_ops_map— takes a map refcount viabpf_map_inc()(kernel/bpf/bpf_struct_ops.c:1514)bpf_map_get()(kernel/bpf/bpf_struct_ops.c:1445)syscall_prog, which invokestest_1()starting a 500ms BPF timerbpf_struct_ops_map_link_detach()drops link's map ref (kernel/bpf/bpf_struct_ops.c:1409)syscall_progFD → program free is deferred through two asynchronous stagesst_ops_mapFD → drops FD's map refThe map cannot be freed until all refcounts drop to zero. The
syscall_progassociation refcount (from step 1) is only released insidebpf_prog_free_deferred()(kernel/bpf/core.c:3073), which runs after:call_rcu()in__bpf_prog_put_noref()(kernel/bpf/syscall.c:2467) — waits for an RCU grace periodschedule_work()inbpf_prog_free()(kernel/bpf/core.c:3106) — waits for work queue schedulingOnly after both stages complete does
bpf_prog_disassoc_struct_ops()callbpf_map_put()(kernel/bpf/bpf_struct_ops.c:1534), which drops the last map ref, triggeringbpf_struct_ops_map_free()→bpf_struct_ops_map_dissoc_progs()(kernel/bpf/bpf_struct_ops.c:1038) → setstest_1->aux->st_ops_assoc = NULL.On s390x under QEMU emulation, the combined delay of the RCU GP + work queue exceeds 500ms, so the timer fires while the map is still alive and
test_1->aux->st_ops_associs still valid. The kfuncbpf_kfunc_multi_st_ops_test_1_assoc()successfully callstest_1()and returns 1234 instead of -1.Proposed Fix
Increase
timer_nsfrom 500ms to 2000ms to give the deferred free chain sufficient time on slow emulated platforms. See attached patch0001-selftests-bpf-increase-struct_ops_assoc-timer-delay.patch.A more robust alternative (not in patch): save the map ID before closing FDs, then poll with
bpf_map_get_fd_by_id()to confirm the map is actually freed before relying on the timer result. This avoids hardcoding any timeout but requires more test code changes.Impact
False CI failures on s390x across unrelated PRs. When triggered, the failure is deterministic within the VM instance (fails on retry), causing the entire s390x test suite to be marked as failed. This creates misleading signal for patch authors whose changes are unrelated to struct_ops.
References
0e841d19263a("selftests/bpf: Test getting associated struct_ops in timer callback") — introduced the testbd06b977e02d("selftests/bpf: Migrate struct_ops_assoc test to KF_IMPLICIT_ARGS") — most recent change