High URCU callback backlog may delay main-thread synchronization and cause Fast LACP flapping

[danieldin95]

Hi OVS community,

We added some statistics in ovsrcu_call_postponed() to help analyze URCU callback processing capacity. With this instrumentation, we observed logs like the following:

2026-07-03T16:07:42.318Z|1982161|ovs_rcu(urcu2)|INFO|callback 55494 times with 1682 ms
2026-07-03T16:07:24.733Z|1982156|ovs_rcu(urcu2)|INFO|callback 88984 times with 1655 ms
2026-07-03T16:02:41.086Z|1982077|ovs_rcu(urcu2)|INFO|callback 55056 times with 1639 ms

In our environment, when the URCU callback backlog reaches around 50k callbacks, processing them can take about 1.6 seconds.

This seems unfriendly to the main thread. If the main thread triggers URCU synchronization during this period, it may delay the sending of LACP packets. In Fast LACP mode, this delay can cause LACP state flapping.

We would like to ask for suggestions from the community:

• Is this callback processing time expected when the backlog reaches this size?
• Are there recommended ways to reduce URCU callback backlog or improve callback processing efficiency?
• Would it make sense to process callbacks in smaller batches to avoid long blocking periods?
• Is there any existing tuning or design recommendation for avoiding impact on time-sensitive protocols such as Fast LACP?

Any advice or direction would be appreciated.

The following table summarizes the hourly URCU callback processing statistics from the OVS logs on 2026-07-03, including the total callback count, total elapsed time, normalized processing time per 40k callbacks, and the maximum single callback processing time.

Hour	Log Entries	Total Callbacks	Total Elapsed Time	Avg Callbacks per Entry	Avg Time per 40k Callbacks	Max Single Elapsed Time
2026-07-03 19:00	62	2,604,624	58,556 ms	42,010	899.3 ms	1,781 ms
2026-07-03 20:00	212	9,062,478	205,491 ms	42,748	907.0 ms	1,997 ms
2026-07-03 21:00	36	1,132,193	26,823 ms	31,450	947.6 ms	1,340 ms
2026-07-03 22:00	47	1,519,919	35,448 ms	32,339	932.9 ms	1,339 ms
2026-07-03 23:00	36	1,175,051	27,492 ms	32,640	935.9 ms	1,345 ms

[chaudron]

On 8 Jul 2026, at 13:50, Daniel Ding wrote: danieldin95 created an issue (openvswitch/ovs-issues#379) Hi OVS community, We added some statistics in `ovsrcu_call_postponed()` to help analyze URCU callback processing capacity. With this instrumentation, we observed logs like the following: ```text 2026-07-03T16:07:42.318Z|1982161|ovs_rcu(urcu2)|INFO|callback 55494 times with 1682 ms 2026-07-03T16:07:24.733Z|1982156|ovs_rcu(urcu2)|INFO|callback 88984 times with 1655 ms 2026-07-03T16:02:41.086Z|1982077|ovs_rcu(urcu2)|INFO|callback 55056 times with 1639 ms ``` In our environment, when the URCU callback backlog reaches around 50k callbacks, processing them can take about 1.6 seconds. This seems unfriendly to the main thread. If the main thread triggers URCU synchronization during this period, it may delay the sending of LACP packets. In Fast LACP mode, this delay can cause LACP state flapping. We would like to ask for suggestions from the community: - Is this callback processing time expected when the backlog reaches this size? - Are there recommended ways to reduce URCU callback backlog or improve callback processing efficiency? - Would it make sense to process callbacks in smaller batches to avoid long blocking periods? - Is there any existing tuning or design recommendation for avoiding impact on time-sensitive protocols such as Fast LACP? Any advice or direction would be appreciated. The following table summarizes the hourly URCU callback processing statistics from the OVS logs on 2026-07-03, including the total callback count, total elapsed time, normalized processing time per 40k callbacks, and the maximum single callback processing time.

My initial thought was to run the callbacks in a quiescent state so the urcu thread does not block other threads calling ovsrcu_synchronize(). However, that is not safe: some callbacks read RCU-protected pointers directly, and others call ovsrcu_postpone() themselves. The system is therefore behaving as designed: ovsrcu_synchronize() correctly waits up to 1.6 seconds for the rcu thread to finish accessing RCU-protected data before continuing. The more relevant question is why so many resources are accumulating in the callback backlog in the first place. That volume of deferred frees points to a high rate of flow or rule churn in your environment. I would suggest making that the focus of the next investigation, i.e. find out what is generating tens of thousands of RCU callbacks per batch and whether that churn can be reduced at the source. Eelco

…

| Hour | Log Entries | Total Callbacks | Total Elapsed Time | Avg Callbacks per Entry | Avg Time per 40k Callbacks | Max Single Elapsed Time | |---|---:|---:|---:|---:|---:|---:| | 2026-07-03 19:00 | 62 | 2,604,624 | 58,556 ms | 42,010 | 899.3 ms | 1,781 ms | | 2026-07-03 20:00 | 212 | 9,062,478 | 205,491 ms | 42,748 | 907.0 ms | 1,997 ms | | 2026-07-03 21:00 | 36 | 1,132,193 | 26,823 ms | 31,450 | 947.6 ms | 1,340 ms | | 2026-07-03 22:00 | 47 | 1,519,919 | 35,448 ms | 32,339 | 932.9 ms | 1,339 ms | | 2026-07-03 23:00 | 36 | 1,175,051 | 27,492 ms | 32,640 | 935.9 ms | 1,345 ms | -- Reply to this email directly or view it on GitHub: #379 You are receiving this because you are subscribed to this thread. Message ID: ***@***.***>

[danieldin95]

Yes @chaudron, as you said, in my environment there are a large number of burst disconnections, around 100,000 entries.

[chaudron]

Can you elaborate more on what you mean by burst disconnections? Which ovsrcu_postpone() calls are involved? I would like to
understand why you see such a high callback volume before we start thinking about a solution.

In theory we could quiesce the urcu thread every N callbacks, but we would first need to check all current callers of ovsrcu_synchronize() to confirm that none of them require ovsrcu_barrier() semantics, since quiescing mid-loop would change the observable behaviour for such callers.

[danieldin95]

A five-tuple session that sees only one packet every 10 seconds, or a session identified by a source and destination address pair that sees only one packet every 10 seconds, can be treated as a short-lived or low-rate flow. Such sparse, short, or low-rate flows are frequently reclaimed by the revalidator from the datapath flow table, which increases URCU reclamation pressure.

[chaudron]

It would still be good to figure out which callback is taking so long and why. You might want to profile this, as sometimes you process 50K+ entries and other times 80K+ entries in almost the same amount of time. What about the system load during these tests?

Also, processing 50K entries every time seems like a lot. If the revalidator runs every 500 ms (worst case), do you have a lot of bursts in your environment that are creating many DP entries?

Which version of OVS are you running, and do you have any additional code additions?

[danieldin95]

We do have a large number of bursty flows in our environment, and they can generate a huge number of datapath entries.
At the moment, the only local code addition is the code we added to collect URCU callback statistics. Other than that, there are no extra
patches in our OVS build.
We are running OVS 2.16.2.

[chaudron]

2.16 is pretty old and no longer supported. Can you test with the latest OVS and see if it has the same problem?