Skip to content

Add hw_ppe.qos: shape in the switch where the switch offloads the flow - #193

Open
JuliusBairaktaris wants to merge 6 commits into
tohojo:mainfrom
JuliusBairaktaris:hw-ppe-offload
Open

Add hw_ppe.qos: shape in the switch where the switch offloads the flow#193
JuliusBairaktaris wants to merge 6 commits into
tohojo:mainfrom
JuliusBairaktaris:hw-ppe-offload

Conversation

@JuliusBairaktaris

@JuliusBairaktaris JuliusBairaktaris commented Aug 20, 2026

Copy link
Copy Markdown

A connection the kernel has offloaded to a switch's flow engine is never handed to a qdisc again, so a shaper built on cake and an ifb does not shape it — the CPU never sees those packets. Where the switch can be told a rate instead, it can still be governed. hw_ppe.qos does that: a root tbf on the port, which a driver implementing TC_SETUP_QDISC_TBF programs into the port's hardware token bucket, and a matchall policer on the port's ingress in place of the redirect to an ifb — a switch cannot queue an arriving packet in order to shape it, but it can meter one. No ifb is created and nothing is redirected. The ingress filter is skip_sw, so a kernel or switch that cannot take it fails loudly instead of quietly policing on the CPU.

Two defaults do the heavy lifting, both measured:

  • The tbf carries an explicit limit of 1 ms of the shaped rate (never less than eight full frames) instead of the customary latency 300ms. The limit is how deep the queue behind the shaper may get, and a driver that drains a real queue sizes that queue from it: at 165 Mbit/s the wide limit held 5.4 ms of standing queue under upload saturation — the bufferbloat the shaper was installed against — where the 1 ms limit holds the added latency under measurement noise at the same throughput.
  • The policer's burst defaults to 10 ms of the rate instead of the generic 1 ms. A meter has no queue to absorb a flow's bursts — it can only drop, and TCP pays every drop with a retransmit: on a 336 Mbit/s line policed at 335 Mbit/s, 1 ms of burst costs 43% of the throughput, 10 ms delivers 96%, and neither adds measurable latency.

The ingress side is still a trade (thanks @moeller0 for keeping this honest): a policer drops where a shaper would have delayed, so it adds no queueing delay but not no delay, and ECN cannot be honoured. What it buys is that offloaded connections are governed at all, at zero CPU. Raise ishaper_burst_dur_us to tolerate more, or set download to 0 to keep only hardware egress shaping.

Configure it on the physical port (wan), not on a pppoe or vlan device above it — the hardware only knows the port. That placement is also what covers traffic the switch cannot offload, Wi-Fi included: it shares the port.

What a user can configure

All ordinary /etc/config/sqm options; run-openwrt.sh now exports the burst durations, so those work from uci (and LuCI's dropdown picks the script up from /usr/lib/sqm/).

option reaches effect
interface both the physical switch port — not pppoe-wan or a VLAN above it
upload / download hardware token bucket / policer rate in kbit/s; 0 disables that direction
eshaper_burst_dur_us hardware egress bucket depth as a duration (default 1 ms of the rate)
ishaper_burst_dur_us hardware policer burst as a duration (default 10 ms of the rate, see above)
qdisc, elimit, etarget, eecn, eqdisc_opts CPU path the qdisc under the tbf, carrying what is not offloaded (Wi-Fi, local traffic); cake falls back to fq_codel with a warning
linklayer, overhead, tcMTU, tcMPU, tcTSIZE neither warned about and ignored — the hardware meters the wire frame including preamble, gap and CRC
ilimit, itarget, iecn, iqdisc_opts neither ingress is a meter, not a queue; nothing for them to configure

Testing

Qualcomm IPQ8074 (Xiaomi AX3600), kernel 6.18, a driver offloading both directions, on a 336 Mbit/s PPPoE line shaped to 335000/165000. Latency probed at 20 Hz from a wired LAN host, load from public speed endpoints:

latency avg / worst throughput
idle 10.0 / 10.5 ms
upload saturated (3 streams) 9.7 / 10.6 ms 159 Mbit/s
download saturated (6 streams) 10.0 / 11.4 ms 325 Mbit/s
both directions at once 10.0 / 11.9 ms 318 + 153 Mbit/s

Zero added latency at 96–97% of the configured rates, both directions loaded. The rates and the queue limit read back exactly in the hardware registers; stop clears the token bucket and the meter (verified by register readback); the uci burst knobs were verified end to end (option → filter burst → register).

The functions.sh change

get_ifb_associated_with_if() treats any filter on parent ffff: that does not name an ifb as a parse failure and prints the cut-here block asking the user to open an issue. It now looks for the redirect before deciding the name should have been there; a real parse failure still reports.

Copilot AI lite review requested due to automatic review settings August 20, 2026 16:29

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@moeller0

Copy link
Copy Markdown
Collaborator

So for what it is worth, I am all for more approaches here and more diverse qos scripts and if this works well for you, you can ignore the rest.
That said, policers are indeed less heavy weight than shapers, but that comes at a cost, a policer will drop packets more readily than a traffic shaper, especially one using ECN, and a dropped packet often is retransmitted, so instead of adding zero delay, the policer adds at least one RTT in such cases, e.g. for TCP traffic. A policer might still be what you want in your script, but I would not try to sell this as a unproblematic latency optimisation.

@JuliusBairaktaris
JuliusBairaktaris force-pushed the hw-ppe-offload branch 4 times, most recently from 89c9738 to a58036b Compare August 22, 2026 18:57
A connection the kernel has offloaded to a switch's flow engine is never seen
by a qdisc again, so a shaper built on cake and an ifb does not shape it. Where
the switch can be told the rate instead, it can.

This script puts a root tbf on the port, which a driver that implements
TC_SETUP_QDISC_TBF turns into that port's hardware token bucket, and a matchall
policer on the port's ingress in place of the redirect to an ifb: a switch
cannot queue an arriving packet in order to shape it, but it can meter one. No
ifb is created and nothing is redirected, so sqm_start and sqm_stop are
overridden to leave that machinery out.

The tbf is given an explicit limit of one millisecond of the shaped rate, never
less than eight full frames, rather than the customary latency 300ms. The limit
is how deep the queue behind the shaper may get, a driver that drains a real
queue sizes that queue from it, and 300 ms of queue is the bufferbloat the
shaper was installed against: measured on an IPQ8074 at 165 Mbit/s, the wide
limit lands on the hardware's ceiling and holds 5.4 ms of standing queue under
upload saturation, where the one-millisecond limit holds the added latency
under measurement noise at the same throughput.

The policer's burst defaults to 10 ms of the rate instead of the generic 1 ms.
A meter has no queue to absorb a flow's bursts - it can only drop, and TCP pays
for every drop with at least a retransmit: measured on a 336 Mbit/s line
policed at 335 Mbit/s, 1 ms of burst costs 43% of the throughput where 10 ms
delivers 96% of it, with no measurable added latency either way. defaults.sh
now marks a burst duration it filled in so a script can tell that fallback from
a value the user chose, and run-openwrt.sh exports the three burst durations
from uci, which also makes them reachable on the platform where a hardware
shaper will mostly run.

The ingress side stays a trade: a policer drops where a shaper would have
delayed, so it adds no queueing delay but not no delay. The filter is installed
skip_sw so that a kernel or a switch which cannot take it fails loudly rather
than quietly policing on the CPU, which is the one outcome this script exists
to avoid.

It has to be configured on the physical port, not on a pppoe or vlan device
above it, because the hardware only knows the port. That is also what makes it
cover traffic the switch cannot offload at all, Wi-Fi included: such traffic
still leaves through the same port and arrives on it. What reaches the CPU is
carried by the qdisc under the tbf, which stays configurable as usual.

Tested on a Qualcomm IPQ8074 (Xiaomi AX3600) with a driver that offloads both,
against a 336 Mbit/s PPPoE line shaped to 335000/165000, probed at 20 Hz from a
wired LAN host while saturating the link with 6 download and 3 upload streams:
idle 10.0 ms, upload saturated 9.7 ms at 159 Mbit/s, download saturated 10.0 ms
at 325 Mbit/s, both directions at once 10.0 ms average and 11.9 ms worst at
318/153 Mbit/s. The rates read back exactly in the hardware registers, stop
clears both, and restart reprograms them.

Signed-off-by: Julius Bairaktaris <julius@bairaktaris.de>
get_ifb_associated_with_if() treats any filter on parent ffff: that does not
name an ifb as a parse failure and prints a cut-here block asking the user to
file an issue. A script that puts something else on the ingress qdisc - a
policer, say - therefore prints that block on every stop.

Look for the redirect before deciding the name should have been there. A real
parse failure still reports.

Signed-off-by: Julius Bairaktaris <julius@bairaktaris.de>
The tbf limit was fixed at one millisecond of the shaped rate, chosen when
every packet shared that queue and its depth was the latency the shaper
added. One millisecond cannot hold a single TCP flow at the shaped rate:
measured on a 165 Mbit/s uplink, it caps one flow at 50 Mbit/s where 8 ms
delivers 162.

With the switch classifying latency-sensitive traffic into higher-priority
queues (the ppe-qos package: small-packet priority and DSCP maps), only the
bulk traffic inside the queue pays its depth, so the default becomes 8 ms
and the duration is a tunable of its own, eshaper_queue_dur_us, following
the shaper-burst pattern.

Assisted-by: Claude <noreply@anthropic.com>
Signed-off-by: Julius Bairaktaris <julius@bairaktaris.de>
The download direction was a matchall policer on the arriving port, because
the switch cannot queue a packet on ingress. It can queue one on egress
though, and every packet the WAN delivers to a wired client leaves by a user
port of the same switch. Putting the tbf there gives the download a real
queue of a bounded depth instead of a meter that can only drop.

Measured on an IPQ8074 with a 330 Mbit/s downstream, one wired client, the
browser bufferbloat test the users of this script reach for: the policer
delivered 233 Mbit/s at +20 ms of loaded latency, the port shapers 295-307
Mbit/s at +3 to +8 ms. The throughput is the meter's bursty tail-loss, and
the latency is what a meter does to a sparse flow - a latency probe is a few
small packets on its own connection, and a meter drops it exactly as readily
as the bulk it was aiming at, where a queue simply delays it behind less.

A port shaper only reaches what leaves by a user port, so the meter stays for
the rest: a frame on its way to a Wi-Fi client leaves by the CPU port, and the
arriving port is the only place in the switch that sees it at all. Wired
traffic is held below the meter by its own port's queue, so the meter is left
governing what no queue reached. Verified by metering at 60 Mbit/s and
watching a Wi-Fi client's download follow it down.

The qdisc under the tbf matters as much as the tbf here: a DSA user port
defaults to noqueue, so a bare tbf would put a FIFO of the whole limit in
front of every packet the switch did not forward itself - and the flows a
latency test probes with are not offloaded, so that is exactly the path they
take. egress() has always put $QDISC under its tbf; ingress() now does too.

The port set comes from phys_switch_id rather than from a name or a count,
since it differs across boards, and the same walk serves the teardown.

Two costs, both in the header: a user port's token bucket meters everything
leaving that port, so traffic between two local hosts is held to the download
rate as well; and a stale ingress qdisc from an older version of this script
is removed on stop, since nothing else here touches it any more.

Assisted-by: Claude <noreply@anthropic.com>
Signed-off-by: Julius Bairaktaris <julius@bairaktaris.de>
A port shaper reaches what leaves by a user port, which left one gap: a frame
on its way to a Wi-Fi client leaves the switch by the port the host is behind,
and that port has no netdev, so no tbf can name it. The download direction kept
a meter on the arriving port to cover it - the one place in the switch that saw
that traffic at all.

The driver can shape that port now, taking its rate directly, so the meter has
nothing left to govern: every port a packet can leave the switch by carries a
queue. Measured on a Wi-Fi client held to 60 Mbit/s, the meter delivered
18.3 Mbit/s of it and the queue delivers 39.6, and running both delivered 37.4 -
the meter costing on top of the queue that replaced it. At the rate this line
runs the client reads 138.9 Mbit/s where the metered configuration read 132,
and the wired direction is unchanged at 296 Mbit/s.

Where the driver cannot shape that port, the meter stays: this script runs on
whatever switch offloads a tbf, and without the meter that traffic would pass
no shaper at all there. The meter is the fallback now rather than the rule.

The burst duration the meter needs stays with it, unused by the shaper path that existed only for it: a shaper
absorbs a burst in its queue and does not need ten milliseconds of tolerance to
hold its rate.

Assisted-by: Claude <noreply@anthropic.com>
Signed-off-by: Julius Bairaktaris <julius@bairaktaris.de>
The egress bulk queue takes its depth from uci and the ingress one was a
constant in the script, so the two could not be compared the same way. They
want the same treatment: a shaper's queue depth is the one number that trades
latency against a single flow's throughput, and which way that trade falls is a
property of the line, not of the script.

Ten milliseconds stays the default, and now on evidence rather than on the
absence of a way to test it. Interleaved on an IPQ8074 against three
milliseconds, run for run inside one window, the shallower queue read +5/+8/+2
ms of download bufferbloat where 10 ms read +1/+1/+1, and gave up throughput as
well. A shaper wants a deep bulk queue; what must not be sitting in it is the
interactive traffic, which is what the switch's own classifier is for.

Assisted-by: Claude <noreply@anthropic.com>
Signed-off-by: Julius Bairaktaris <julius@bairaktaris.de>
@JuliusBairaktaris

JuliusBairaktaris commented Aug 26, 2026

Copy link
Copy Markdown
Author

@moeller0 saw the comment on the typo, do you want me to fix that?

Also please wait until: openwrt/openwrt#24806 is merged. Thanks!

@sppmasterspp

Copy link
Copy Markdown

saw the comment on the typo, do you want me to fix that?

What typo?

Comment thread src/functions.sh
if [ ! -z "${TMP}" ] && echo "${TMP}" | grep -q mirred; then
# oops, a redirect is there but we failed to parse it? Ask for a user report
sqm_error "#---- CUT HERE ----#"
sqm_error "get_ifb_associated_with_if failed to extrect the ifb name from:"

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants