Add CPU fp16 insert-cast policy controls

[patryk-kaiser-ARM]

Carry the current CPU fp16 opt-in and fallback heuristic policy plus transformer coverage as its own stacked layer.

This PR builds on https://github.com/JonathanC-ARM/onnxruntime/tree/fp16-split/03-cpu-fp16-kernels

This is pr 4 of 5 as per #28786 splitting suggestion

This PR introduces an explicit CPU execution policy for FP16 MatMul and Gemm following the kernel support added.

This PR adds the tri-state session.cpu_fp16_mode option:

• off - Use the existing FP32 cast fallback. Default.
• heuristic - Keep native FP16 only for supported, expected-profitable shapes.
• forced - Allow the CPU EP to claim registered FP16 kernels without profitability filtering.

The policy is implemented in CPU EP capability selection. InsertCastTransformer remains responsible for generic FP16-to-FP32 fallback and isolated FP16 islands.

Motivation and Context
CPU FP16 support is not uniformly faster across every shape or backend. Automatically assigning every supported MatMul and Gemm to FP16 could introduce performance regressions.

This change provides:

• A safe default preserving existing behaviour.
• A conservative, backend-aware heuristic mode.
• A forced mode for testing and explicit opt-in.
• Kernel type-resolution checks before creating FP32 fallback paths.
• Consistent eligibility checks between graph partitioning and runtime dispatch.

Scope
This PR covers only CPU FP16 policy and cast handling for MatMul and Gemm.
Apple ARM64 enablement, dense MOPA kernels, Gemm prepacking, and FP16 FusedMatMul support are handled separately.

Testing
Coverage includes:

• Default, off, heuristic, and forced behaviour.
• Invalid option values.
• Supported and unsupported MatMul/Gemm configurations.
• KleidiAI-disabled fallback.
• Mixed execution-provider graphs.
• FP16 fallback output-type preservation.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

🟡 Changes recommended

Kernel capability checks can disagree with runtime lookup, duplicate assignment records, and regress initialization scalability.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds configurable CPU FP16 execution and FP32 fallback policies, backed by MLAS capability detection and transformer coverage.

Changes:

• Adds CPU FP16 opt-in and fallback-heuristic session options.
• Extends InsertCastTransformer with kernel and profitability checks.
• Adds transposed FP16 RHS support and comprehensive tests.

File summaries

File	Description
onnxruntime/core/session/inference_session.cc	Wires session options into cast transformation.
include/onnxruntime/core/session/onnxruntime_session_options_config_keys.h	Defines the new configuration keys.
onnxruntime/core/optimizer/insert_cast_transformer.h	Extends transformer configuration and state.
onnxruntime/core/optimizer/insert_cast_transformer.cc	Implements FP16 eligibility and fallback policy.
onnxruntime/core/providers/cpu/math/gemm.cc	Enables accelerated transposed-RHS FP16 Gemm.
onnxruntime/core/mlas/inc/mlas.h	Adds transposed-RHS metadata and documentation.
onnxruntime/core/mlas/lib/halfgemm.cpp	Guards unsupported generic transposed execution.
onnxruntime/core/mlas/lib/kleidiai/halfgemm_kleidiai.cpp	Implements KleidiAI transposed-RHS packing.
onnxruntime/core/mlas/lib/kleidiai/mlasi_kleidiai.h	Updates native packing contract.
onnxruntime/test/framework/insert_cast_transformer_test.cc	Covers policy, heuristics, configuration, and runtime behavior.
onnxruntime/test/mlas/unittest/test_halfgemm.cpp	Tests transposed packing and dispatch.
onnxruntime/test/providers/cpu/math/matmul_test.cc	Verifies native prepacked weights are not shared.

Review details

Suppressed comments (1)

onnxruntime/core/optimizer/insert_cast_transformer.cc:1034

• Clearing this CPU assignment without adding the node to nodes_with_recorded_cpu_assignment makes the later cast path call RecordPartitionAssignment again. For the normal case where the partitioner already assigned an fp16 MatMul/Gemm, enabling EP assignment recording therefore emits a duplicate OrtEpAssignedSubgraph entry. Track the assignment before clearing it, as the heuristic branch below already does.

    if (!enable_cpu_fp16_ &&
        node->GetExecutionProviderType() == kCpuExecutionProvider &&
        IsCpuFp16OptInPolicyOp(*node) &&
        HasFp16Input(*node) &&
        !node->ContainsSubgraph()) {
      node->SetExecutionProviderType("");
    }

• Files reviewed: 12/12 changed files
• Comments generated: 2
• Review effort level: Balanced

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.