Add hw_ppe.qos: shape in the switch where the switch offloads the flow - #193
Add hw_ppe.qos: shape in the switch where the switch offloads the flow#193JuliusBairaktaris wants to merge 6 commits into
Conversation
fc46927 to
e39091e
Compare
|
So for what it is worth, I am all for more approaches here and more diverse qos scripts and if this works well for you, you can ignore the rest. |
89c9738 to
a58036b
Compare
A connection the kernel has offloaded to a switch's flow engine is never seen by a qdisc again, so a shaper built on cake and an ifb does not shape it. Where the switch can be told the rate instead, it can. This script puts a root tbf on the port, which a driver that implements TC_SETUP_QDISC_TBF turns into that port's hardware token bucket, and a matchall policer on the port's ingress in place of the redirect to an ifb: a switch cannot queue an arriving packet in order to shape it, but it can meter one. No ifb is created and nothing is redirected, so sqm_start and sqm_stop are overridden to leave that machinery out. The tbf is given an explicit limit of one millisecond of the shaped rate, never less than eight full frames, rather than the customary latency 300ms. The limit is how deep the queue behind the shaper may get, a driver that drains a real queue sizes that queue from it, and 300 ms of queue is the bufferbloat the shaper was installed against: measured on an IPQ8074 at 165 Mbit/s, the wide limit lands on the hardware's ceiling and holds 5.4 ms of standing queue under upload saturation, where the one-millisecond limit holds the added latency under measurement noise at the same throughput. The policer's burst defaults to 10 ms of the rate instead of the generic 1 ms. A meter has no queue to absorb a flow's bursts - it can only drop, and TCP pays for every drop with at least a retransmit: measured on a 336 Mbit/s line policed at 335 Mbit/s, 1 ms of burst costs 43% of the throughput where 10 ms delivers 96% of it, with no measurable added latency either way. defaults.sh now marks a burst duration it filled in so a script can tell that fallback from a value the user chose, and run-openwrt.sh exports the three burst durations from uci, which also makes them reachable on the platform where a hardware shaper will mostly run. The ingress side stays a trade: a policer drops where a shaper would have delayed, so it adds no queueing delay but not no delay. The filter is installed skip_sw so that a kernel or a switch which cannot take it fails loudly rather than quietly policing on the CPU, which is the one outcome this script exists to avoid. It has to be configured on the physical port, not on a pppoe or vlan device above it, because the hardware only knows the port. That is also what makes it cover traffic the switch cannot offload at all, Wi-Fi included: such traffic still leaves through the same port and arrives on it. What reaches the CPU is carried by the qdisc under the tbf, which stays configurable as usual. Tested on a Qualcomm IPQ8074 (Xiaomi AX3600) with a driver that offloads both, against a 336 Mbit/s PPPoE line shaped to 335000/165000, probed at 20 Hz from a wired LAN host while saturating the link with 6 download and 3 upload streams: idle 10.0 ms, upload saturated 9.7 ms at 159 Mbit/s, download saturated 10.0 ms at 325 Mbit/s, both directions at once 10.0 ms average and 11.9 ms worst at 318/153 Mbit/s. The rates read back exactly in the hardware registers, stop clears both, and restart reprograms them. Signed-off-by: Julius Bairaktaris <julius@bairaktaris.de>
get_ifb_associated_with_if() treats any filter on parent ffff: that does not name an ifb as a parse failure and prints a cut-here block asking the user to file an issue. A script that puts something else on the ingress qdisc - a policer, say - therefore prints that block on every stop. Look for the redirect before deciding the name should have been there. A real parse failure still reports. Signed-off-by: Julius Bairaktaris <julius@bairaktaris.de>
The tbf limit was fixed at one millisecond of the shaped rate, chosen when every packet shared that queue and its depth was the latency the shaper added. One millisecond cannot hold a single TCP flow at the shaped rate: measured on a 165 Mbit/s uplink, it caps one flow at 50 Mbit/s where 8 ms delivers 162. With the switch classifying latency-sensitive traffic into higher-priority queues (the ppe-qos package: small-packet priority and DSCP maps), only the bulk traffic inside the queue pays its depth, so the default becomes 8 ms and the duration is a tunable of its own, eshaper_queue_dur_us, following the shaper-burst pattern. Assisted-by: Claude <noreply@anthropic.com> Signed-off-by: Julius Bairaktaris <julius@bairaktaris.de>
The download direction was a matchall policer on the arriving port, because the switch cannot queue a packet on ingress. It can queue one on egress though, and every packet the WAN delivers to a wired client leaves by a user port of the same switch. Putting the tbf there gives the download a real queue of a bounded depth instead of a meter that can only drop. Measured on an IPQ8074 with a 330 Mbit/s downstream, one wired client, the browser bufferbloat test the users of this script reach for: the policer delivered 233 Mbit/s at +20 ms of loaded latency, the port shapers 295-307 Mbit/s at +3 to +8 ms. The throughput is the meter's bursty tail-loss, and the latency is what a meter does to a sparse flow - a latency probe is a few small packets on its own connection, and a meter drops it exactly as readily as the bulk it was aiming at, where a queue simply delays it behind less. A port shaper only reaches what leaves by a user port, so the meter stays for the rest: a frame on its way to a Wi-Fi client leaves by the CPU port, and the arriving port is the only place in the switch that sees it at all. Wired traffic is held below the meter by its own port's queue, so the meter is left governing what no queue reached. Verified by metering at 60 Mbit/s and watching a Wi-Fi client's download follow it down. The qdisc under the tbf matters as much as the tbf here: a DSA user port defaults to noqueue, so a bare tbf would put a FIFO of the whole limit in front of every packet the switch did not forward itself - and the flows a latency test probes with are not offloaded, so that is exactly the path they take. egress() has always put $QDISC under its tbf; ingress() now does too. The port set comes from phys_switch_id rather than from a name or a count, since it differs across boards, and the same walk serves the teardown. Two costs, both in the header: a user port's token bucket meters everything leaving that port, so traffic between two local hosts is held to the download rate as well; and a stale ingress qdisc from an older version of this script is removed on stop, since nothing else here touches it any more. Assisted-by: Claude <noreply@anthropic.com> Signed-off-by: Julius Bairaktaris <julius@bairaktaris.de>
A port shaper reaches what leaves by a user port, which left one gap: a frame on its way to a Wi-Fi client leaves the switch by the port the host is behind, and that port has no netdev, so no tbf can name it. The download direction kept a meter on the arriving port to cover it - the one place in the switch that saw that traffic at all. The driver can shape that port now, taking its rate directly, so the meter has nothing left to govern: every port a packet can leave the switch by carries a queue. Measured on a Wi-Fi client held to 60 Mbit/s, the meter delivered 18.3 Mbit/s of it and the queue delivers 39.6, and running both delivered 37.4 - the meter costing on top of the queue that replaced it. At the rate this line runs the client reads 138.9 Mbit/s where the metered configuration read 132, and the wired direction is unchanged at 296 Mbit/s. Where the driver cannot shape that port, the meter stays: this script runs on whatever switch offloads a tbf, and without the meter that traffic would pass no shaper at all there. The meter is the fallback now rather than the rule. The burst duration the meter needs stays with it, unused by the shaper path that existed only for it: a shaper absorbs a burst in its queue and does not need ten milliseconds of tolerance to hold its rate. Assisted-by: Claude <noreply@anthropic.com> Signed-off-by: Julius Bairaktaris <julius@bairaktaris.de>
The egress bulk queue takes its depth from uci and the ingress one was a constant in the script, so the two could not be compared the same way. They want the same treatment: a shaper's queue depth is the one number that trades latency against a single flow's throughput, and which way that trade falls is a property of the line, not of the script. Ten milliseconds stays the default, and now on evidence rather than on the absence of a way to test it. Interleaved on an IPQ8074 against three milliseconds, run for run inside one window, the shallower queue read +5/+8/+2 ms of download bufferbloat where 10 ms read +1/+1/+1, and gave up throughput as well. A shaper wants a deep bulk queue; what must not be sitting in it is the interactive traffic, which is what the switch's own classifier is for. Assisted-by: Claude <noreply@anthropic.com> Signed-off-by: Julius Bairaktaris <julius@bairaktaris.de>
27ce43c to
6b32d92
Compare
|
@moeller0 saw the comment on the typo, do you want me to fix that? Also please wait until: openwrt/openwrt#24806 is merged. Thanks! |
What typo? |
| if [ ! -z "${TMP}" ] && echo "${TMP}" | grep -q mirred; then | ||
| # oops, a redirect is there but we failed to parse it? Ask for a user report | ||
| sqm_error "#---- CUT HERE ----#" | ||
| sqm_error "get_ifb_associated_with_if failed to extrect the ifb name from:" |
A connection the kernel has offloaded to a switch's flow engine is never handed to a qdisc again, so a shaper built on cake and an ifb does not shape it — the CPU never sees those packets. Where the switch can be told a rate instead, it can still be governed.
hw_ppe.qosdoes that: a roottbfon the port, which a driver implementingTC_SETUP_QDISC_TBFprograms into the port's hardware token bucket, and amatchallpolicer on the port's ingress in place of the redirect to an ifb — a switch cannot queue an arriving packet in order to shape it, but it can meter one. No ifb is created and nothing is redirected. The ingress filter isskip_sw, so a kernel or switch that cannot take it fails loudly instead of quietly policing on the CPU.Two defaults do the heavy lifting, both measured:
limitof 1 ms of the shaped rate (never less than eight full frames) instead of the customarylatency 300ms. The limit is how deep the queue behind the shaper may get, and a driver that drains a real queue sizes that queue from it: at 165 Mbit/s the wide limit held 5.4 ms of standing queue under upload saturation — the bufferbloat the shaper was installed against — where the 1 ms limit holds the added latency under measurement noise at the same throughput.The ingress side is still a trade (thanks @moeller0 for keeping this honest): a policer drops where a shaper would have delayed, so it adds no queueing delay but not no delay, and ECN cannot be honoured. What it buys is that offloaded connections are governed at all, at zero CPU. Raise
ishaper_burst_dur_usto tolerate more, or setdownloadto 0 to keep only hardware egress shaping.Configure it on the physical port (
wan), not on a pppoe or vlan device above it — the hardware only knows the port. That placement is also what covers traffic the switch cannot offload, Wi-Fi included: it shares the port.What a user can configure
All ordinary
/etc/config/sqmoptions;run-openwrt.shnow exports the burst durations, so those work from uci (and LuCI's dropdown picks the script up from/usr/lib/sqm/).interfacepppoe-wanor a VLAN above itupload/downloadeshaper_burst_dur_usishaper_burst_dur_usqdisc,elimit,etarget,eecn,eqdisc_optscakefalls back tofq_codelwith a warninglinklayer,overhead,tcMTU,tcMPU,tcTSIZEilimit,itarget,iecn,iqdisc_optsTesting
Qualcomm IPQ8074 (Xiaomi AX3600), kernel 6.18, a driver offloading both directions, on a 336 Mbit/s PPPoE line shaped to 335000/165000. Latency probed at 20 Hz from a wired LAN host, load from public speed endpoints:
Zero added latency at 96–97% of the configured rates, both directions loaded. The rates and the queue limit read back exactly in the hardware registers;
stopclears the token bucket and the meter (verified by register readback); the uci burst knobs were verified end to end (option → filter burst → register).The functions.sh change
get_ifb_associated_with_if()treats any filter onparent ffff:that does not name an ifb as a parse failure and prints the cut-here block asking the user to open an issue. It now looks for the redirect before deciding the name should have been there; a real parse failure still reports.