Skip to content

docs(deepeval): add upstream adapter documentation - #95

Merged
ruivieira merged 3 commits into
eval-hub:mainfrom
ruivieira:feat/deepeval-docs
Jul 27, 2026
Merged

docs(deepeval): add upstream adapter documentation#95
ruivieira merged 3 commits into
eval-hub:mainfrom
ruivieira:feat/deepeval-docs

Conversation

@ruivieira

@ruivieira ruivieira commented Jul 27, 2026

Copy link
Copy Markdown
Member

What and why

Creates upstream documentation for the DeepEval adapter on eval-hub.github.io. Covers: usage guide, benchmark reference, dataset format schemas, MLflow integration, and configuration reference.

Type

  • feat
  • docs
  • fix
  • refactor / chore
  • test / ci

Testing

  • Tested manually
  • Tests added or updated

Pages added

  • adapters/deepeval/ — Overview with EvalHub-first quick start (SDK CLI, Python SDK, REST API tabs)
  • adapters/deepeval/configuration — JobSpec structure, all parameters grouped by concern, MLflow integration, environment variables
  • adapters/deepeval/benchmarks — All 8 benchmarks (5 single-turn, 3 multi-turn) with required columns, output metrics, and dataset format reference (CSV / JSONL / JSON)
  • adapters/deepeval/examples — Worked examples: faithfulness, hallucination detection, multi-turn role adherence, conversation completeness, local testing, MLflow-tracked evaluation

Summary by CodeRabbit

  • New Features
    • Added a DeepEval adapter documentation section with Overview, Configuration, Benchmarks, and Examples pages.
    • Documented supported single-turn and multi-turn evaluation metrics, dataset formats, required columns, and configuration options.
    • Added quick-start guidance for SDK, REST API, local execution, container usage, and MLflow tracking.
    • Included practical examples covering faithfulness, hallucination detection, role adherence, and conversation completeness.

Signed-off-by: Rui Vieira <ruidevieira@googlemail.com>
@coderabbitai

coderabbitai Bot commented Jul 27, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 518cd755-c690-4482-a8f5-64220e69562b

📥 Commits

Reviewing files that changed from the base of the PR and between 9e0c499 and ef83a83.

📒 Files selected for processing (2)
  • src/content/docs/adapters/deepeval/configuration.md
  • src/content/docs/adapters/deepeval/examples.mdx
🚧 Files skipped from review as they are similar to previous changes (2)
  • src/content/docs/adapters/deepeval/configuration.md
  • src/content/docs/adapters/deepeval/examples.mdx

📝 Walkthrough

Walkthrough

Adds DeepEval adapter documentation covering overview, configuration, benchmarks, examples, local execution, MLflow integration, and sidebar navigation.

Changes

DeepEval adapter documentation

Layer / File(s) Summary
Overview and navigation
astro.config.mjs, src/content/docs/adapters/deepeval/index.mdx
Adds the DeepEval sidebar subsection, adapter workflow overview, quick-start commands, provider details, container usage, and direct execution instructions.
Configuration and benchmark references
src/content/docs/adapters/deepeval/configuration.md, src/content/docs/adapters/deepeval/benchmarks.md
Documents JobSpec fields, parameters, environment variables, MLflow settings, supported benchmarks, dataset formats, and column requirements.
Worked evaluation examples
src/content/docs/adapters/deepeval/examples.mdx
Adds faithfulness, hallucination, multi-turn, local testing, and MLflow-tracked evaluation examples.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

Suggested labels: kind/docs

Suggested reviewers: ppadashe-psp, julpayne, gnaulak-redhat

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise and accurately summarizes the main change: adding upstream DeepEval adapter documentation.
Description check ✅ Passed The description covers what, type, testing, and added pages, with only a missing issue reference and no breaking changes noted.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/content/docs/adapters/deepeval/configuration.md`:
- Around line 26-28: Update the JobSpec example’s "parameters" value in the
configuration documentation to valid JSON by replacing the inline comment with
an empty object or a concrete parameter entry; preserve the surrounding example
structure.

In `@src/content/docs/adapters/deepeval/examples.mdx`:
- Around line 277-291: Update the documented EvalHub job submission example to
configure MLflow using the top-level JobSpec experiment_name field rather than
passing experiment_name through DeepEval-specific --param values. Remove the
CLI-side MLFLOW_TRACKING_URI export and note or demonstrate that cluster-mode
tracking URI configuration is supplied by the EvalHub server/sidecar.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: c6eab020-427c-47b6-b84f-185fa517242a

📥 Commits

Reviewing files that changed from the base of the PR and between fe7071e and 9e0c499.

📒 Files selected for processing (5)
  • astro.config.mjs
  • src/content/docs/adapters/deepeval/benchmarks.md
  • src/content/docs/adapters/deepeval/configuration.md
  • src/content/docs/adapters/deepeval/examples.mdx
  • src/content/docs/adapters/deepeval/index.mdx

Comment thread src/content/docs/adapters/deepeval/configuration.md
Comment thread src/content/docs/adapters/deepeval/examples.mdx Outdated

@ppadashe-psp ppadashe-psp left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved

@ruivieira
ruivieira merged commit 0fb49b1 into eval-hub:main Jul 27, 2026
2 of 3 checks passed
@github-project-automation github-project-automation Bot moved this from Todo to Done in EvalHub Planning Jul 27, 2026
@ruivieira
ruivieira deleted the feat/deepeval-docs branch July 27, 2026 08:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants