Conversation
The two reducing kernels issued a vector reduction for every group of a row rather than carrying the result in a vector and reducing at the end, which is what the generic implementations do. A reduction costs several times what an add or a max costs, so a row became a chain of them and the vector unit spent most of it waiting. The group width goes from LMUL 1 to LMUL 4 at the same time. The two are worth separating in principle but not in this file: the polynomial in the exponential is a dependency chain either way, and what covers it is having a group wide enough to keep independent work in flight. LMUL 8 is not taken, since it leaves the register file four names and spills on the longer rows. The polynomial stays a multiply and an add. Horner's form wants the addend to be the constant, and the fused instruction takes its addend from a vector, so each step would first have to broadcast the constant into one: an instruction and a register group per step, for no gain. Carrying the sum in a vector changes the order the row is added in, so the result is not the one the previous kernel produced, in the same way that the generic implementations already differ from each other by lane count. The exponentials themselves are untouched.
The convolution and pooling kernels here hold a channel block in one
register group, and each of them accumulated a single output position at
a time. Unlike a back end whose registers are narrower than a block,
that leaves no work to overlap: every multiply accumulate for a position
depends on the one before it, so the kernels ran at the latency of the
multiply accumulate rather than its throughput, and the filter vector was
loaded again for every position that read it.
Accumulate four adjacent output positions at once instead. The chains are
then independent of one another and one filter load serves all four. A
register group cannot be held in an array, so the positions are spelled
out, and a group holding fewer than four repeats the first position in the
slots it does not use, which is what lets one body serve both a full group
and the remainder. The filter block becomes the outer loop so that it stays
resident while the positions that read it are swept.
Convolution shapes from resnet50 and mobilenetv2, single core, in GFLOPS
against the XuanTie C920V2 cores of an A210, at 128 bits, and a SpacemiT K3,
whose X100 cores are at 256:
C920V2 X100
56x56 64->64 3x3 4.8 -> 14.1 8.7 -> 15.5
56x56 64->256 1x1 5.8 -> 10.5 9.2 -> 16.0
28x28 128->128 3x3 7.3 -> 18.2 8.6 -> 15.7
14x14 256->256 3x3 6.8 -> 14.0 8.0 -> 14.2
7x7 512->512 3x3 3.9 -> 11.2 8.2 -> 14.7
224x224 3->64 7x7 5.7 -> 13.6 9.0 -> 15.2
112x112 32->16 1x1 1.1 -> 7.5 8.8 -> 14.1
Every one of those shapes was slower through this path than through the
convolution the dispatch selects when the NCHWc path is unavailable, and
none of them is now.
A channel block is sixteen floats, and these kernels asked for it from a
group of four registers whatever the part was. That is the right group at
128 bits and too wide above it: the cost of a vector instruction follows
the multiplier it was issued with rather than the number of elements it was
asked for, so at 256 bits every instruction in these kernels was paid for
at four registers and half of what it was paid for went unused.
Write the kernels once against a set of accessors and instantiate them for
each multiplier a block can live in, then enter the instantiation the part
wants: four registers at 128 bits, two at 256, one at 512. The block size
and therefore the memory layout are unchanged, so a filter reordered on one
part is still read correctly on another, and the results are unchanged as
well.
The same convolution shapes, single core, in GFLOPS. The C920V2 is at 128
bits and keeps four registers, so it is unaffected; the X100 is at 256 and
drops to two:
C920V2 X100
56x56 64->64 3x3 14.1 15.5 -> 32.4
56x56 64->256 1x1 10.5 16.0 -> 26.7
28x28 128->128 3x3 18.2 15.7 -> 31.9
14x14 256->256 3x3 14.6 14.2 -> 27.8
7x7 512->512 3x3 11.1 14.7 -> 24.4
224x224 3->64 7x7 13.5 15.2 -> 27.6
112x112 32->16 1x1 7.4 14.1 -> 24.9
The direct depthwise path admits the three by three shape at stride one, and every other shape goes through im2col and a GEMM per channel. A depthwise convolution gives that GEMM a single row, so nearly all of its time goes into building the column buffer and packing a panel for one row of arithmetic. There is now a kernel for the general shape at dilation one, and MlasConvPrepare admits those shapes. Admitting them is only safe because the kernel reports success for every shape it is offered: the fallback the dispatch would otherwise reach handles three by three alone and would silently compute the wrong result. Two bodies are emitted. The three by three one follows the shape of the kernel it sits beside, whose row pointers already make a row's taps contiguous. The general one cannot borrow that shape, because a wider kernel loses too much to the edges: on a seven by seven image a five by five kernel pads by two, so leaving the padded columns scalar would leave four of seven columns scalar. It vectorizes the whole row and asks per tap which output columns that tap reaches, which is an interval whose two ends are not alike -- a tap starting before the image loses leading lanes, which no vl or pointer offset reaches, while one reaching past the end loses trailing lanes, which shortening vl drops exactly. LMUL is chosen at run time rather than fixed. A row is only as wide as the image and a vector operation costs what its LMUL says regardless of how much of it the current vl uses, so a tile wider than the row wastes most of the machine, and how wide a fixed LMUL is depends on the vector length. The body is written once and instantiated at LMUL 1, 2 and 4. The results are not bit-identical to the kernel this extends, and cannot be: that kernel leaves its expression for the compiler to contract and the summation topology differs. This is the same class of difference the direct path already introduces against the im2col path it replaces. The widened routing is conditional on that kernel being installed, since a build for the vector extension still has to run on a part that does not implement it, and the algorithm it would otherwise select handles 3x3 alone.
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Zestion
changed the title
Contributor
Author
|
Hariharan Seshadri (@hariharans29) Xavier Dupré (@xadupre) Could you please review this PR when you get a chance? Thanks! |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
platform.cpp references ActivationRoutine outside the MLAS_USE_RVV guard, which breaks compilation for riscv64 builds that don’t enable RVV.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR improves FP32 inference performance on riscv64 by expanding and optimizing MLAS RVV kernels (softmax/log-softmax, multiple convolution variants, and fused activations) while keeping public interfaces unchanged.
Changes:
- Optimize RVV softmax/log-softmax primitives by accumulating across a row and reducing once, using LMUL=4.
- Rework RVV NCHW/NCHWc/depthwise/pointwise conv kernels to compute multiple adjacent outputs per iteration and dispatch LMUL based on vector length.
- Add an RVV
MlasActivationfast path for elementwise fused activations (e.g., ReLU, Clip, HardSigmoid) and wire it into platform initialization and CMake.
File summaries
| File | Description |
|---|---|
| onnxruntime/core/mlas/lib/riscv64/softmax_kernel_rvv.cpp | Softmax/log-softmax RVV row-accumulation and LMUL=4 refactor. |
| onnxruntime/core/mlas/lib/riscv64/sconv_nchwc_kernel_rvv.cpp | NCHW/NCHWc/depthwise/pointwise/pooling RVV kernels restructured with multi-output accumulation and LMUL dispatch. |
| onnxruntime/core/mlas/lib/riscv64/sconv_depthwise_kernel_rvv.cpp | Extend RVV depthwise CHW kernel beyond 3x3 and add runtime LMUL selection for general kernels. |
| onnxruntime/core/mlas/lib/riscv64/conv_activation_kernel_rvv.cpp | New RVV fused-activation implementation for elementwise activation kinds (with optional bias). |
| onnxruntime/core/mlas/lib/platform.cpp | Wire RVV activation routine into riscv64 platform dispatch (plus other RVV hooks). |
| onnxruntime/core/mlas/lib/mlasi.h | Add MLAS_ACTIVATION_ROUTINE, RVV activation symbol, and riscv64 RVV depthwise-kernel-width constant. |
| onnxruntime/core/mlas/lib/convolve.cpp | Widen depthwise routing on riscv64 when RVV kernels are installed (avoids im2col+GEMM-per-channel). |
| onnxruntime/core/mlas/lib/activate.cpp | Add riscv64 RVV activation fast path before generic activation switch. |
| cmake/onnxruntime_mlas.cmake | Add the new RVV activation source to the MLAS build when RVV is enabled. |
Review details
- Files reviewed: 9/9 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Contributor
Author
|
Hi velonica0, this PR builds on your RVV NCHWc work in #28411. Since you previously benchmarked those kernels on K3, would you be willing to cross-check these changes on your board when convenient? Testing a few models, such as ResNet50, MobileNetV2, or ShuffleNet, for correctness and performance would be very helpful. Thanks! |
MlasActivation is written against MLAS_FLOAT32X4 and splits every row into a four wide body and a scalar remainder, so a convolution's fused activation moved four elements at a time however wide the vector unit is. The element-wise kinds are covered here by a platform routine, which returns false for the two that MlasActivation applies the bias for and then routes through MlasComputeTanh and MlasComputeLogistic; those keep going through the switch. Each kind reproduces the arithmetic of its MLAS_ACTIVATION_FUNCTION specialization, in the same order, so the results are the same.
Zestion
force-pushed
the
riscv64-rvv-kernel-optimizations
branch
from
September 10, 2026 10:44
7a9b090 to
b44bb44
Compare
Member
|
CC: mirounga for the review |
Contributor
|
Yes, these modifications significantly improve performance! Results (1 thread, ms)
|
Contributor
Author
|
Hi mirounga, could you take a look when you have a chance? I've addressed the Copilot comments, and the updated activation tests pass on A210 and K3 in RVV and scalar modes. Happy to provide any additional details. Thanks! |
mirounga
self-requested a review
September 16, 2026 16:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Improve FP32 inference in the MLAS riscv64 RVV backend with five commits:
sconv_depthwise_kernel_rvv.cppto additional shapes and strides,with unit dilation and kernel width up to sixteen. Enable the wider route only when RVV kernels are installed.
MlasActivationroutine, including the clamp used by ReLU6.Changes are limited to MLAS and its CMake source list. Public interfaces remain unchanged.
Performance
A210/C920V2: VLEN=128; SpacemiT K3/X100: VLEN=256.
Baseline is unmodified upstream
mainat9f913ae52, which already includes #32406.Optimized is that baseline with this PR's five RVV kernel changes.
Both use the CPU Execution Provider,
ORT_ENABLE_ALL, and RVV kernels enabled, with the same toolchainand build configuration. Runs use one pinned core, one thread, warmup, fixed frequency, and best of three.
Speedup is baseline latency divided by optimized latency. These measurements precede the final depthwise
file consolidation and latest rebase.
RetinaFace uses the official serengil TensorFlow/ResNet50 implementation and weights, exported with tf2onnx
at opset 13.
Validation
Before the latest rebase: XuanTie cross-compilation passed under
-Werror. After depthwise consolidation,the K3 convolution/activation subset passed 1109 tests with RVV and 565 with forced scalar,
with zero failures and three HalfConv skips in each mode. Softmax/log-softmax differential checks against
a double-precision reference passed on both boards.
Motivation and Context
Follow-up to #28261 and #28411, building on the runtime gate restored in #32406.
The existing NCHWc kernels reached only 0.20x–0.84x of the alternative convolution path's throughput
across seven measured shapes. Independent accumulators and appropriately sized register groups address
these bottlenecks while preserving the blocked layout.
Broader depthwise routing avoids im2col plus a single-row GEMM per channel. It benefits convolutions
that cannot use NCHWc, including some ShuffleNet layers at the default optimization level, as well as
ORT_ENABLE_EXTENDED. Fewer softmax reductions and vectorized activation address the remaining overhead.