Skip to content

docs(guides): add MLflow experiment tracking guide - #93

Merged
julpayne merged 4 commits into
mainfrom
docs/mlflow-guide
Jul 22, 2026
Merged

docs(guides): add MLflow experiment tracking guide#93
julpayne merged 4 commits into
mainfrom
docs/mlflow-guide

Conversation

@julpayne

@julpayne julpayne commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add a dedicated MLflow guide covering server config, experiment schema, per-benchmark runs, SDK backends (odh/upstream), CLEAR trace fetch vs result upload, and troubleshooting — verified against eval-hub, eval-hub-sdk, and eval-hub-contrib
  • Cross-link related docs (home, quickstart, local mode, CLI, server API, system overview, MCP tools, CLEAR adapter) and fill gaps such as full mlflow.* config keys and CLEAR mlflow_traces_* parameters

Test plan

  • Docs site builds successfully (npm run build)
  • New sidebar entry Guides → MLflow opens /guides/mlflow/
  • Links from home, quickstart, local mode, and CLEAR pages resolve
  • Spot-check experiment tag examples match REST ({key,value}) vs MCP (map) behavior

Made with Cursor

Summary by CodeRabbit

  • New Features

    • Added a comprehensive MLflow guide covering setup, experiment tracking, configuration, APIs, troubleshooting, and adapter integrations.
    • Added MLflow to the Guides navigation and homepage feature links.
    • Expanded IBM CLEAR documentation with MLflow trace input and result logging workflows.
    • Documented additional MLflow server settings, tags, run identifiers, and backend options.
  • Documentation

    • Updated quick-start, CLI, local-mode, MCP, and architecture documentation with MLflow examples and guidance.

Document current MLflow behavior from eval-hub, eval-hub-sdk, and
eval-hub-contrib, and cross-link related pages that previously only
mentioned tracking in passing.

Assisted-by: Cursor
Signed-off-by: Julian Payne <julpayne@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@coderabbitai

coderabbitai Bot commented Jul 22, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@julpayne, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 5 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 652b07d6-9a41-45f1-bd55-ac278d455449

📥 Commits

Reviewing files that changed from the base of the PR and between 4ddb5dc and 6d5626f.

📒 Files selected for processing (4)
  • src/content/docs/adapters/clear/index.md
  • src/content/docs/architecture/system-overview.md
  • src/content/docs/getting-started/quickstart.mdx
  • src/content/docs/guides/mlflow.mdx
📝 Walkthrough

Walkthrough

Adds a new MLflow guide and updates EvalHub documentation for tracking configuration, experiment submission, adapter callbacks, CLEAR trace ingestion, troubleshooting, navigation, and related references.

Changes

MLflow documentation

Layer / File(s) Summary
MLflow tracking guide
src/content/docs/guides/mlflow.mdx
Adds configuration guidance, experiment schemas, CLI/YAML/SDK/REST examples, logged metadata, adapter behavior, CLEAR usage, troubleshooting, and related links.
Adapter and CLEAR MLflow integration
src/content/docs/adapters/clear/index.md, src/content/docs/guides/mlflow.mdx
Documents MLflow callback persistence, CLEAR trace retrieval, result uploads, filtering, and separate input/output experiment settings.
Configuration and submission references
src/content/docs/architecture/system-overview.md, src/content/docs/reference/server-api.md, src/content/docs/getting-started/quickstart.mdx, src/content/docs/mcp/tools.md
Updates MLflow result fields, backend settings, server environment variables, experiment tags, and MCP/REST payload formats.
MLflow discovery and cross-links
astro.config.mjs, src/content/docs/home.mdx, src/content/docs/guides/cli.mdx, src/content/docs/guides/local-mode.mdx, src/content/docs/guides/local-mode-tutorial.mdx
Adds navigation and homepage links and connects CLI and local-mode documentation to the MLflow guide.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

Suggested labels: kind/docs, documentation

Suggested reviewers: ruivieira, gnaulak-redhat

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description covers the summary and test plan but omits required template sections like What and why, Closes #, Type, and Breaking changes. Reformat it to match the template: add What and why, Closes #, Type, Testing, and Breaking changes sections, even if some are briefly filled in.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely summarizes the main change: adding an MLflow experiment tracking guide.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch docs/mlflow-guide

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/content/docs/adapters/clear/index.md`:
- Line 65: Update the CLEAR adapter documentation to state that experiment.name
is the default experiment for logging results, while
parameters.mlflow_experiment_name overrides it when provided. Also revise the
workflow wording near the earlier experiment description to use this same
default-and-override behavior, while preserving the separate input-trace
setting.

In `@src/content/docs/getting-started/quickstart.mdx`:
- Around line 313-324: Update the Python quickstart example’s imports to include
JobSubmissionRequest, ModelConfig, and BenchmarkConfig alongside the existing
ExperimentConfig and ExperimentTag imports, so every model constructed in the
submission example is defined.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: e40905f7-a6a4-4c41-be0b-a25ec6462c6f

📥 Commits

Reviewing files that changed from the base of the PR and between 8de40b7 and 4ddb5dc.

📒 Files selected for processing (11)
  • astro.config.mjs
  • src/content/docs/adapters/clear/index.md
  • src/content/docs/architecture/system-overview.md
  • src/content/docs/getting-started/quickstart.mdx
  • src/content/docs/guides/cli.mdx
  • src/content/docs/guides/local-mode-tutorial.mdx
  • src/content/docs/guides/local-mode.mdx
  • src/content/docs/guides/mlflow.mdx
  • src/content/docs/home.mdx
  • src/content/docs/mcp/tools.md
  • src/content/docs/reference/server-api.md

Comment thread src/content/docs/adapters/clear/index.md Outdated
Comment thread src/content/docs/getting-started/quickstart.mdx Outdated
julpayne added 3 commits July 22, 2026 11:19
Signed-off-by: Julian Payne <julpayne@redhat.com>
…ocumentation

Signed-off-by: Julian Payne <julpayne@redhat.com>
…verview documentation

Signed-off-by: Julian Payne <julpayne@redhat.com>
@julpayne
julpayne merged commit e144d6a into main Jul 22, 2026
3 checks passed
@julpayne
julpayne deleted the docs/mlflow-guide branch July 22, 2026 09:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants