Feature/auto eval

[simonweigold]

This pull request updates the AI agent and reasoning kit documentation to add support for automated evaluation of workflow steps using a judge LLM, introduces the kit.json configuration file for kit-level defaults, and clarifies evaluation modes and CLI flags. It also improves formatting and code examples for clarity. Below are the most important changes:

---

Automated Evaluation and Judge LLM Support

• Added documentation for auto-evaluating reasoning kit steps using a judge LLM, enabled via the --auto-evaluate flag and/or the auto_evaluate=True argument in Python. Describes how steps are scored automatically using evaluator_N.txt or evaluator_N.json files, and how the judge model can be specified per run or via kit.json. [1] [2] [3]
• Updated Python usage and CLI examples to demonstrate auto-evaluation, judge model selection, and aggregation strategies for multi-dimension evaluators. [1] [2]

Kit Configuration Enhancements

• Introduced the kit.json file for kit-level configuration defaults, including model, judge_model, mode, and evaluator_aggregation. Added documentation for the file format, precedence rules, and integration with the ReasoningKit and new KitConfig data model. [1] [2]

Evaluation File and Aggregation Support

• Added support and documentation for single- and multi-dimension evaluator files (evaluator_N.txt, evaluator_N.json), and the evaluator_aggregation strategy (average, min, max, first). [1] [2] [3]

Formatting, Tech Stack, and Command Documentation

• Improved formatting of tables and code blocks for consistency and readability. Updated the LLM provider name to gpt-5.4-nano throughout. [1] [2] [3] [4] [5]
• Clarified CLI commands, environment setup, and common code patterns in .github/instructions/AGENTS.md.instructions.md. [1] [2] [3]

Minor Improvements

• Added or improved code style guidelines, docstring templates, and best practices for Python code in the agent instructions. [1] [2] [3] [4]

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
clerk	Ready	Preview	Aug 3, 2026 7:28pm