Summary
Send reasoning traces, tool call metrics, token usage, and session metadata to MLFlow in RHOAI for structured model comparison experiments.
Plugin Details
- Red Hat product: OpenShift AI (RHOAI) 3.5 — MLFlow component
- Plugin hooks:
tool.execute.after, session.end, session.model.change, event
- Complexity: Medium
- Priority: 12 (Medium value, already built externally)
What it tracks
- Reasoning traces (thinking/chain-of-thought from the model)
- Tool call count, types, success/failure rates per session
- Token usage (input/output) per turn
- Model switches within a session
- Session duration and compaction count
- Final session outcome (completed, abandoned, errored)
MLFlow integration
- Creates an MLFlow experiment per project
- Each tinycode session = one MLFlow run
- Reasoning traces logged as MLFlow artifacts
- Metrics logged with step tracking
- Model ID logged as MLFlow parameter for A/B comparison
Use case
Evaluating Qwen 3 32B vs Granite Code 34B on your codebase. Run the same task with each model. Compare tool call success rates, token efficiency, and task completion in MLFlow's comparison view.
Notes
An external implementation already exists — include in the plugin ecosystem and document as reference.
From: Red Hat Product Integration Plugin Ideas
Summary
Send reasoning traces, tool call metrics, token usage, and session metadata to MLFlow in RHOAI for structured model comparison experiments.
Plugin Details
tool.execute.after,session.end,session.model.change,eventWhat it tracks
MLFlow integration
Use case
Evaluating Qwen 3 32B vs Granite Code 34B on your codebase. Run the same task with each model. Compare tool call success rates, token efficiency, and task completion in MLFlow's comparison view.
Notes
An external implementation already exists — include in the plugin ecosystem and document as reference.
From: Red Hat Product Integration Plugin Ideas