GPU memory optimization and kernel fusion for the Alpaka backend

[ioantreas]

Summary

This PR extends the SOFIE Alpaka GPU backend with two main optimizations: GPU intermediate-memory management and GPU kernel fusion. It also expands the benchmark infrastructure used to evaluate these changes against other GPU inference backends.

Main changes

GPU intermediate-memory allocation

• Add an intermediate-memory pool for GPU tensors instead of allocating each intermediate tensor independently.
• Reuse memory based on tensor lifetimes.
• Track free regions and merge released blocks to reduce unnecessary allocations.
• Support intermediate and dynamic tensor placement in the GPU memory pool.
• Integrate the allocator into generated Alpaka session code.

GPU kernel fusion

• Add a generic fusion framework for compatible GPU operators.
• Build fusion candidates from operator input/output dependencies and mapping characteristics.
• Select valid non-conflicting fusion groups before code generation.
• Generate fused Alpaka kernels while keeping intermediate values internal where possible.
• Add specialised fusion support for patterns such as GEMM followed by activation.
• Move fusion logic into RModel_Fusion_ALPAKA.cxx to keep the main Alpaka code generator separated from fusion planning.

GPU operator/model support

• Extend GPU operator support needed by the benchmark models.
• Add support for persistent GPU state used by recurrent/sequence operators.
• Add and optimise MambaV2 selective-scan execution.
• Include fixes required for correct GPU execution of Mamba, Griffin RGLRU, RWKV WKV6, and related operators.

Benchmarking

Expand the SOFIE GPU benchmark to support comparisons with:

• ONNX Runtime GPU
• TensorRT
• PyTorch AOTInductor

The benchmark now includes:

• inference latency and throughput;
• host/device transfer timing;
• peak GPU memory usage;
• SOFIE memory-pool statistics;
• GPU utilisation sampling through NVML;
• per-operator SOFIE profiling;
• Nsight Compute cross-backend profiling;
• benchmark result aggregation and plotting;
• model-size/scaling variants.

The build configuration also supports optional custom paths for ONNX Runtime, TensorRT, benchmark models, and the Mamba selective_scan_cuda library.

PyTorch AOTInductor and Mamba

• Add AOTInductor benchmark support using precompiled .pt2 packages.
• Add exporters for the benchmark model families.

Documentation

Update the benchmark README with:

• dependency and backend setup;
• build configuration options;
• ONNX Runtime, TensorRT, and AOTInductor setup;
• Mamba-specific setup;
• benchmark execution options;
• internal GPU profiling;
• Nsight Compute profiling;
• result summarisation and plotting.

[sanjibansg]

/runtest h100-47gb

[github-actions]

/runtest (h100-47gb): triggered - view run

[github-actions]

/runtest (h100-47gb): GPU Unit Tests ✅ passed - view run