Client TUN offload (3/3): client GSO/GRO offload implementation - #504
Draft
xv-thomas-leong wants to merge 11 commits into
Draft
Client TUN offload (3/3): client GSO/GRO offload implementation#504xv-thomas-leong wants to merge 11 commits into
xv-thomas-leong wants to merge 11 commits into
Conversation
This was referenced Aug 7, 2026
|
Code coverage summary for 8a01710: ✅ Region coverage 72% passes |
xv-thomas-leong
force-pushed
the
main-tun-offload-features
branch
from
August 7, 2026 03:05
bd6137a to
cd53e1b
Compare
xv-thomas-leong
force-pushed
the
main-tun-offload-features
branch
from
August 7, 2026 03:16
cd53e1b to
96d1252
Compare
xv-thomas-leong
force-pushed
the
main-tun-offload-features
branch
from
August 7, 2026 03:39
96d1252 to
7632068
Compare
xv-thomas-leong
changed the title
xv-thomas-leong
force-pushed
the
main-tun-offload-features
branch
from
August 7, 2026 04:24
7632068 to
286849c
Compare
xv-thomas-leong
force-pushed
the
main-tun-offload-features
branch
from
August 13, 2026 05:18
286849c to
6c2d4e3
Compare
xv-thomas-leong
force-pushed
the
main-tun-offload-features
branch
from
August 17, 2026 02:51
6c2d4e3 to
8f30da5
Compare
xv-thomas-leong
force-pushed
the
main-tun-offload-features
branch
from
August 17, 2026 04:13
8f30da5 to
5867772
Compare
xv-thomas-leong
force-pushed
the
main-tun-offload-features
branch
from
August 17, 2026 07:58
5867772 to
e66d9a7
Compare
xv-thomas-leong
force-pushed
the
main-tun-offload-features
branch
from
August 19, 2026 04:03
e66d9a7 to
537d1a3
Compare
xv-thomas-leong
force-pushed
the
main-tun-offload-features
branch
from
August 19, 2026 05:43
537d1a3 to
6c4dde1
Compare
xv-thomas-leong
force-pushed
the
main-tun-offload-features
branch
2 times, most recently
from
August 21, 2026 03:56
765a97f to
b58ce7e
Compare
xv-thomas-leong
force-pushed
the
main-tun-offload-features
branch
from
August 21, 2026 04:00
b58ce7e to
15451ef
Compare
xv-thomas-leong
force-pushed
the
main-tun-offload-features
branch
3 times, most recently
from
August 24, 2026 07:59
e8ae254 to
27b3574
Compare
Trust `tun_device.tcp_gso()` over `config.offload`: `build_async` succeeds even when `TUNSETOFFLOAD` is rejected (tun-rs only warns), so config alone could leave recv_gso/try_send_gso handling a header the device never negotiated. Warn when offload was requested but not granted. Note the flag tracks TSO/USO capability, not the IFF_VNET_HDR framing that TUNSETIFF already granted; traffic never flows in the mismatched state because the as_gso() startup check aborts first.
build_segment recomputes each segment's checksum from scratch, so folding the NEEDS_CSUM aggregate first is a wasted one's-complement sum over up to 64KB per superpacket. Gate the fold on non-GSO packets. Uses is_gso_none() so the ECN bit is masked.
xv-thomas-leong
force-pushed
the
main-tun-offload-features
branch
2 times, most recently
from
August 25, 2026 04:56
3ccc964 to
f78bf17
Compare
A UDP_SEGMENT sendmsg is bounded by two kernel limits: total bytes (MAX_GSO_SEND_BYTES = 65535-8-40, EMSGSIZE beyond) and segment count (gso_size * UDP_MAX_SEGMENTS, EINVAL beyond). A TUN aggregate can reach 65535 bytes before the per-segment wire::Header, so split on segment boundaries to keep UDP_SEGMENT's uniform stride, clamping segs_per_send to MAX_GSO_SEGS. A mid-batch failure drops the remainder; datagram semantics let the peer recover. Gates MAX_GSO_SEND_BYTES to Linux, its only consumer.
send_gso_chunked and udp_send_gso returned IOCallbackResult<usize>, but the byte count was misleading: the unit was inconsistent between legs (inside bytes when plugins dropped everything, wire bytes on success), and a send_gso implementation may report a batch as fully sent that the kernel actually shed, so no caller may treat a successful return as a delivery count. Return IOCallbackResult<()> so failures are surfaced for logging only and nothing reads a bogus count.
Receive-side mirror of gso: merge in-order same-flow IPv4 TCP segments into one TSO superpacket behind a virtio_net_hdr so the kernel traverses its receive path once per batch. TcpGroTable coalesces several flows per window while preserving within-flow write order. Rules mirror the kernel's tcp_gro_receive; PSH/FIN-first segments are written directly, uncopied. The header compare in `append` is a byte-range whitelist-of-exclusions, not field accessors: every byte is compared except the fields that legitimately differ, so any unanticipated field forces a safe flush. Field accessors would invert this into a blacklist that splices two TCP states on a forgotten field. Rationale is recorded at the code. Other fields use pnet_packet, matching gso.rs. NEEDS_CSUM seeding is the inverse of gso_none_checksum. Adds VirtioNetHdr::to_bytes.
With --enable-tun-offload on Linux the TUN opens with IFF_VNET_HDR and the inside loop reads TSO superpackets, forwarding each via inside_data_received_gso (segmented and encrypted per segment, shipped as chunked sendmsg(UDP_SEGMENT) batches). The capability is an upgrade trait (InsideIORecvGso via as_gso), checked once at startup; kernels refusing IFF_VNET_HDR fail with a clear error before traffic moves. The io-uring inside backend cannot supply a GSO device, so that combination is rejected in validation. Offload is also rejected with the inside packet codec: the codec takes the packet before segmentation and would ship a ~64KB superpacket whose total_length lies to the peer. Skips the checksum fold for aggregates (mirrors the server loop); both gso_type tests use is_gso_none() to mask ECN. Adds client-side and both-sides offload e2e targets.
socket_enable_udp_gro flips the UDP_GRO sockopt (Linux 5.0+), and the cmsg iterator learns the UDP_GRO control message reporting coalesced segment size. The sockopt is what makes the kernel coalesce (valid peer checksums are necessary but not sufficient), and it changes every subsequent receive on the socket: a plain recv(2) with no control space silently concatenates datagrams. The doc comment says so. The new arm's two unsafe ops are split into separate SAFETY blocks (multiple_unsafe_ops_per_block is denied).
Mirror of the GSO upload path: in-order TCP segments are merged per flow by a TcpGroTable and written to the TUN once as a TSO superpacket via Tun::try_send_gso. Coalescing runs in a window the receive loop opens per batch (gro_open/gro_flush on InsideIORecv, no-ops without offload); the window flag is an AtomicBool with relaxed load, since all opens, sends and flushes run on one task. The send path takes a TunSend seam with the GRO fields in a Gro struct generic over it (monomorphised, so the closed-window path stays a relaxed load). Seven tests over a FakeTun cover the load-bearing ordering guarantee (flushed superpackets reach the TUN before a non-consumed packet's direct write), the no-op/bypass/drain cases, and drop accounting; the ordering test is mutation-verified. Write failures are logged and dropped, not propagated -- the absorbed sends already reported success. The log carries the segment count.
Route UDP receives through one recvmmsg per readiness event when offload is on: up to MAX_IO_BATCH_SIZE datagrams per syscall, each with its own control buffer, so kernel-coalesced aggregates arrive with their UDP_GRO segment size and split zero-copy on that boundary. Without coalescing (old kernel, or zero-checksum peer) each slot holds one datagram and recvmmsg still cuts syscalls. The GRO window is flushed inside the conn lock, not after it: send() only runs under that mutex, so closing the window while held leaves no gap for another task to coalesce past the drain and strand a tail packet. write_super only issues non-blocking try_sends, so holding the lock longer is free. gso_size == 0 falls back to whole-buffer at the split. The capability is an upgrade trait (OutsideIORecvGro via as_gro), selected at startup; TCP mode keeps the per-packet loop. The sockopt is flipped only when that loop consumes it.
On an egress device without TX checksum offload the kernel refuses a UDP_SEGMENT send in udp_send_skb() with EIO (UDP GSO needs CHECKSUM_PARTIAL, which such a device cannot provide), dropping every coalesced batch. Observed on a Realtek USB 5GbE adapter (cdc_ncm); checksum-offload NICs are unaffected. Latch a per-socket flag on EIO/EINVAL/EOPNOTSUPP, exposed via the new OutsideIOSendCallback::gso_enabled(); the connection then drops to the existing per-segment fallback (send_to_outside). Transient errors and the sizing error EMSGSIZE do not latch. One batch is lost at latching; the socket never attempts GSO again.
parse_coalescable accepted any structurally-valid segment without checking its TCP checksum, and take() reseeds VIRTIO_NET_HDR_F_NEEDS_CSUM so the kernel *completes* the checksum rather than verifying it. A download segment corrupted upstream of the tunnel (origin->server leg, outside the DTLS AEAD) would then be merged and handed to the app with a freshly-valid checksum over corrupt bytes — laundering a packet the non-offload path drops. Kernel GRO validates for exactly this reason. Verify the checksum as the last (most expensive) check; a failing segment falls out to Incompatible and is written directly, where a plain (non-NEEDS_CSUM) write lets the kernel validate and drop it and TCP retransmits. Single-segment packets were already safe (take() returns them untouched with a GSO_NONE header).
xv-thomas-leong
force-pushed
the
main-tun-offload-features
branch
from
August 25, 2026 05:20
f78bf17 to
503d266
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 3 of 3 — client GSO/GRO offload implementation
Final part of a three-PR stack that adds client-side TUN GSO/GRO offload.
Base this on part 2 (refactorings). This part adds the actual feature,
one logical change per commit: deriving the vnet header from the
negotiated
IFF_VNET_HDR, chunking GSO flushes to the kernel's per-sendlimits, TCP GRO coalescing, the client GSO send path,
UDP_GROreceive,TSO superpacket coalescing on download, and batched
recvmmsgGROreceive.
How Has This Been Tested?
CI and Manually tested
Types of changes
Checklist:
main