Carry jumbo frames as UMEM chains instead of oversized frames - #2
Merged
Conversation
Any packet over 4100 bytes failed to start with a bare "XDP_UMEM_REG: invalid argument". frameSizeFor grew the UMEM frame to fit the largest packet, picking 8192 or 16384, but an aligned-chunk UMEM must satisfy 2048 <= chunk_size <= PAGE_SIZE and go-afxdp never sets XDP_UMEM_UNALIGNED_CHUNK_FLAG, so the kernel refused it. This is not architecture-specific, and it needed no unusual configuration: EC2 ships ENA at MTU 9001, and a receive-only run sizes frames as mtu + 18, so plain "--mode receive" on a stock instance was enough to hit it. Frames now stop at a page and packets larger than one span several, which is what go-afxdp's WithMultiBuffer has provided since v0.7.0. It engages on its own when the largest frame exceeds the frame size, so normal traffic keeps the single-frame path and its zero-copy generation untouched. Receive moves to ReceivePackets so a chain counts as one packet at its full length, with the protocol read from the fragment that carries the headers. Transmit keeps SendFunc, which writes straight into the UMEM frame but cannot chain; the jumbo path stages into a preallocated arena and lets SendBatch split it. Preflight reports chaining rather than refusing it, and now bounds the real limit, how many frames one packet may span. fleetKey gains the flag too: it is fixed at bind time, and the frame size does not imply it, since at 4096-byte frames a 4100-byte run chains nothing while a 9000-byte run chains. Verified on c7gn.xlarge (aarch64, AL2023, ENA 2.17.2g, MTU 9001), where the old binary reproduced the reported failure exactly: 9000-byte frames now run at 40 Gbit/s and 399,970 of 399,970 packets arrive. 64-byte and 1518-byte rates are unchanged. Closes #1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix Jumbo frames, actually use multi-buffer