Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Empty file removed .commandcode/taste/taste.md
Empty file.
5 changes: 5 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,11 @@ jobs:
- name: Run specs
run: bundle exec rake spec

- name: Replay recorded workflow cassettes
env:
VCR_RECORD_MODE: none
run: bundle exec rspec --tag live

- name: Run RuboCop
run: bundle exec rubocop

Expand Down
32 changes: 32 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,35 @@
.bundle
vendor/bundle
Gemfile.lock

# Local agent-tooling state
/.claude/
/.claude-flow/
/.junie/

# Local secrets
.env
.env.local
.env.*.local

# Codex local configuration
.codex/

# Local agent tooling (kept on disk, never packaged)
/.agents/
/.swarm/
/.mcp.json
/AGENTS.md
/CLAUDE.md
examples/blog/pipeline_trace.md
.commandcode/

# Generated by the examples on every run; regenerate rather than track
examples/*/trace.md
examples/*/pipeline_trace.md
examples/*/choice_trace.md
examples/blog/output.md
examples/blog/eval.json
examples/code_review/review.md
examples/decision_panel/decision.md
examples/topic_analyst/plan.md
40 changes: 40 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Changelog

## 0.1.0 — experimental

First public release. The API is deliberately small but not yet stable: artifact ordering,
error types, and trace serialization all changed shortly before this release. Pin an exact
version, and read [docs/DECISIONS.md](docs/DECISIONS.md) for what the gem refuses to do and
why — those refusals are the stable part.

### Added

- Named immutable artifact versions with `as:`/`from:` handoffs and the thin `Team#run` API.
- Machine-readable traces: `Session#to_h`/`#to_json` with per-call and run-total best-known
token usage; prompts and results export only with `include_content: true`.
- Typed `BudgetExceededError < CollaborationError` for budget exhaustion.
- `examples/code_review/` — parallel fan-out/fan-in with a VCR-replayed spec and a
line comparison against the upstream plain-Ruby pattern.

### Fixed

- Artifact versions are reserved in submission order, so `artifact(name)` is deterministic
when parallel work completes out of order.
- Non-`StandardError` crashes finalize their call as `:failed` and re-raise instead of
leaving it `:running` with a burned budget slot.
- Fiber siblings settle before a crash propagates; thread joins no longer mask the first crash.
- A coworker instance delegating back into its own call fails with a clear error instead of
`deadlock; recursive locking`.
- Duplicate coworkers in one `parallel` batch are rejected before reserving budget instead of
silently dropping results.
- `share_context: false` sessions no longer record handoff inputs the coworker never received.

### Changed

- `Run#step` omitted `from:` now hands over every completed artifact, matching `Session#ask`.
- The published gem contains only `lib/`, README, CHANGELOG, and LICENSE.

### Foundation

- Coworker registry, `delegate_work`/`ask_question` tools, session call budgets,
thread/fiber `parallel`, selected handoffs, and the Markdown trace.
6 changes: 6 additions & 0 deletions Gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,13 @@ source 'https://rubygems.org'
gemspec

group :development, :test do
gem 'async', require: false
gem 'rake'
gem 'rspec', '~> 3.0'
gem 'rubocop', require: false
gem 'ruby_llm-mcp', '~> 1.0', require: false
gem 'vcr'
gem 'webmock', '~> 3.18'

gem 'ruby_llm-tribunal', '~> 0.1', require: false if Gem::Version.new(RUBY_VERSION) >= Gem::Version.new('3.2')
end
207 changes: 73 additions & 134 deletions PITCH.md
Original file line number Diff line number Diff line change
@@ -1,159 +1,98 @@
# PITCH: `ruby_llm-team` — the delegation primitive RubyLLM won't ship, as a gem
# PITCH: `ruby_llm-team`

One sentence: **a `RubyLLM::Team` is the one multi-agent primitive that plain Ruby cannot
express on its own — letting the *model* choose which named coworker to route to at
runtime — and it belongs in a small extension gem, not in RubyLLM core and not inside a
CrewAI-style orchestration framework.**
`ruby_llm-team` packages the repeated plumbing around named RubyLLM coworkers:
delegation tools, exact result handoffs, bounded calls, concurrent review, and an
inspectable collaboration record. It is a small extension to RubyLLM, not a process
engine.

This document grounds that claim in the rejection of PR #891, the maintainer's stated
philosophy, the CrewAI model and its community pain, and RubyLLM's own issue history.
## Why a separate gem

---
RubyLLM's Agentic Workflows guidance intentionally uses ordinary Ruby for sequencing,
routing, parallel work, fan-in, and evaluator/reviser loops. The maintainer closed the
original Team contribution to RubyLLM core for that reason: the demonstrated workflows
did not justify another core abstraction.

## 1. The seed: why this gem exists
That decision sets the boundary for this gem. Team must remove repeated integration code
without taking workflow policy away from the application.

PR [#891](https://github.com/crmne/ruby_llm/pull/891) ("Add RubyLLM::Team for multi-agent
collaboration") was closed by the maintainer. The rejection is the single most important
fact about this gem, because it defines the *correct* scope:
## What the CrewAI review actually shows

> "This does not belong in the library. The existing Agentic Workflows documentation
> already shows multi-agent sequencing, routing, handoffs, parallel work, and fan-in
> using plain Ruby and the existing Agent and Tool APIs. Those examples are clearer, use
> less code, and do not require adding another abstraction or public API to RubyLLM."
> — [crmne, PR #891 comment](https://github.com/crmne/ruby_llm/pull/891#issuecomment-5435551502)
CrewAI offers two related layers:

Read carefully. The maintainer did **not** say the idea is bad. He said:
- **Crews** organize autonomous agents into sequential or hierarchical processes.
- **Flows** give the application explicit, event-driven control over state, branches, and
execution paths.

1. The **workflow patterns** (sequencing, routing, handoffs, parallel, fan-in) are already
well served by plain Ruby + Agent + Tool. Adding a process engine would be worse.
2. A new abstraction does not belong in **core**.
3. If a concrete capability can't be expressed with existing APIs, open an issue first.
CrewAI tasks can name an agent, expected output, prior task context, guardrails, and
asynchronous execution. A later task that depends on asynchronous tasks forms a clear
fan-out/fan-in boundary. These are useful collaboration mechanics, independent of
CrewAI's larger framework.

`ruby_llm-team` answers all three. It adds **no process, no scheduler, no graph**. It is a
thin tool-layer. And it is a **gem**, not a core addition — precisely the escape hatch the
maintainer's "does not belong in the library" leaves open.
The review does **not** support describing CrewAI as simply rigid or claiming that its
framework decides every execution path. CrewAI itself recommends Crews for autonomous
work, Flows for deterministic work, and a hybrid for applications needing both.

---
## What Team adopts

## 2. The two ways, compared
- Named specialists with explicit responsibilities.
- Exact outputs from completed work as inputs to dependent work.
- Concurrent execution for independent tasks, followed by a synchronization barrier.
- Artifact-preserving reviewer handoffs with application-owned revision limits.
- Visible call limits, errors, inputs, and results.
- A choice between application-directed calls and model-directed delegation tools.

### The RubyLLM way (maintainer's stance, current docs)
These mechanics map naturally to Ruby agents, tools, threads, and fibers. They do not
require a second workflow language.

Orchestration is **ordinary Ruby**. `Agent` is a configured chat; `Tool` is a capability;
the docs show sequential, routing, handoff, parallel, fan-in, and evaluator-optimizer as
small plain-Ruby classes. Applications own task order, dependencies, persistence, and
resume. `RubyLLM.workflow` only adds instrumentation correlation — it does not take over
execution.
## What Team deliberately leaves out

Strengths: total flexibility, nothing hidden, debuggable, idiomatic, durable (the loop is
interruptible and resumable). Weakness: every team re-writes the same delegation boundary
by hand.
- `Crew` / `Task` / `Process` or graph DSLs.
- YAML workflow definitions and generated project structure.
- A built-in hierarchical manager or automatic planner.
- Framework-owned state persistence, scheduling, deployment, or remote transport.
- Built-in memory, knowledge stores, RAG, or MCP clients.
- Hidden retry, model-selection, or concurrency policy.
- Runtime quality claims based only on an LLM judge.

### The CrewAI way
Those capabilities can be valuable, but RubyLLM, ordinary Ruby, and focused ecosystem
gems already provide composition points for them. Adding them to Team would turn a small
collaboration primitive into a competing agent platform.

`Crew` / `Agent` / `Task` / `Process` abstractions. Agents declare role/goal/backstory;
tasks declare expected output; the crew runs a `sequential` or `hierarchical` process
(hierarchical needs a manager LLM). Simple to explain, fast to prototype.
## The product boundary

Strengths: approachable, opinionated, quick demos; CrewAI reports enterprise adoption and
"14x less code" vs graph frameworks. Weakness: a fixed process model that is **not very
flexible** — the exact tradeoff the user named.
Team owns:

### The decisive evidence
- a named coworker registry;
- `delegate_work` and `ask_question` tools;
- per-run collaboration state and exact handoffs;
- immutable named artifacts, revision lineage, and a thin Run API;
- atomic call budgets;
- thread or fiber fan-out/fan-in;
- normalized errors, results, and collaboration traces.

CrewAI's own engineering blog, after "2 billion agentic workflows", lands on the **RubyLLM
position**, not the Crew abstraction:
The application owns task dependencies, conditional policy, quality gates, revision and
escalation limits, persistence, authorization, cancellation, and approvals. A model may
choose coworkers through `session.tools`; explicit workflows may call `session.ask` and
`session.parallel` directly.

> "Architecture choices compound fast... separating the predictable from the
> unpredictable. Having deterministic workflows handling the structure, and agents
> deployed strategically where judgment actually matters."
>
> "Many engineers regret graph-based architectures... too many abstraction layers stacked
> on top of each other... when something breaks, the engineers dig through multiple
> indirections just to try finding which prompt or tool caused it."
> — [Lessons From 2 Billion Agentic Workflows](https://blog.crewai.com/lessons-from-2-billion-agentic-workflows/)
## Evaluation boundary

The community reports the same friction:
Runtime validators protect production invariants such as Ruby syntax and real APIs.
Tribunal is an optional test-time grader for relevance, faithfulness, hallucination, and
regression evaluation. Its report is evidence about a saved artifact; it is not a Team
coworker, runtime gate, or telemetry system.

- r/crewai: "Overwhelmed with limitations... outdated dependencies, slow performance."
- r/AI_Agents: "it gets fragile and you lose fine-grained control" in production.
- r/LangChain: "LangGraph and CrewAI are overcomplicating agents... So I abandoned these
libraries, as a bonus dropped the necessity to use Python in production."
- r/AI_Agents "Who's using CrewAI really?": few teams report production use.
## Evidence standard

**Conclusion: do not ship a CrewAI-style `Crew`/`Task`/`Process` abstraction.** It would
contradict the maintainer's philosophy, the docs, and CrewAI's own hard-won lessons. The
gem's job is the *opposite*: give Ruby developers the one missing low-level primitive and
let them keep orchestration in plain Ruby.
Product decisions should rely on current primary documentation, source code, and observed
behavior. Anonymous community complaints may suggest questions to investigate, but they
are not sufficient evidence for permanent scope decisions.

---
## Sources

## 3. Shared requests and pains (grounding)

These are the signals that a delegation primitive is genuinely wanted, from RubyLLM's own
issue tracker and the broader community:

| Signal | Source | What it says |
|---|---|---|
| Multi-agent is wanted, but not as transport | [#670 A2A protocol](https://github.com/crmne/ruby_llm/issues/670) (declined, `not_planned`) | People want multi-agent; maintainer drew the line at external transport. **Local delegation is the acceptable scope.** |
| Team idea itself | [#891](https://github.com/crmne/ruby_llm/pull/891) | Rejected on process/scope, not on value. The code was correct and fully tested (17 specs, 97.74% coverage). |
| Long-running, resumable work | [#635 "Interrupting the agentic loop"](https://github.com/crmne/ruby_llm/issues/635) (completed) | Real pain: multi-step loops that must pause/resume across deploys. A team that composes with durable agents fits this. |
| The "one more abstraction" fatigue | Maintainer's rejection; CrewAI blog; community threads | Nobody wants another rigid framework. The gem must stay a primitive, not a platform. |
| Ruby landscape gap | langchain.rb (huge/complex), FlowNodes (minimalist) | No Ruby-idiomatic, provider-agnostic multi-agent delegation primitive exists. RubyLLM is the natural host ecosystem. |

The through-line: **Ruby developers want multi-agent capability without sacrificing
control.** CrewAI-style frameworks sell the former and tax the latter. A small gem that
sells the primitive and leaves control alone is the gap.

---

## 4. What the gem is, and is not

### Is

- `RubyLLM::Team` — a named coworker registry.
- Two ordinary `RubyLLM::Tool`s the model calls at runtime: `delegate_work` and
`ask_question`. The model, not the developer, picks the coworker.
- Recoverable error contract (`{ error: ... }`), shared `context:`, attachment
round-tripping, class-vs-instance lifecycle, snapshot concurrency safety.
- Composes with the existing RubyLLM way: plain-Ruby workflow classes, durable agents,
`RubyLLM.workflow` instrumentation.

### Is not

- Not a `Crew`/`Task`/`Process` engine. No sequential/hierarchical process, no scheduler,
no executor, no graph.
- Not an A2A transport. No host-boundary communication.
- Not a replacement for the Agentic Workflows patterns — those stay in plain Ruby, per the
maintainer.

The differentiator in one line: **Team is where the model decides; the workflow is where
the developer decides.** CrewAI lets the framework decide both; plain Ruby leaves both to
you; this gem takes only the part the model must own.

---

## 5. The pitch

> RubyLLM gives you one beautiful API for every provider, and says "orchestrate with
> plain Ruby." That's right — until you want the *model* to route work to a named
> specialist at runtime. Today every team hand-rolls that delegation boundary: a registry,
> two tool classes, an error contract, attachment handling. `ruby_llm-team` is that
> boundary, extracted, tested, and composable — so your orchestration stays plain Ruby and
> the model's delegation stays first-class.
>
> - **For RubyLLM users:** one line of setup, two tools, no framework.
> - **For the maintainer's philosophy:** no new abstraction in core, no process engine,
> no transport — just a Tool boundary that plain Ruby couldn't express.
> - **For the market:** the Ruby gap between "single agent" and "crew" — without the crew's
> rigidity.

---

## 6. Sources

- Maintainer rejection: https://github.com/crmne/ruby_llm/pull/891#issuecomment-5435551502
- RubyLLM Agentic Workflows docs: `docs/_advanced/agentic-workflows.md` (working tree)
- CrewAI lessons: https://blog.crewai.com/lessons-from-2-billion-agentic-workflows/
- RubyLLM issues: #670 (A2A, declined), #635 (interrupt loop), #891 (Team), #889 (JSON format)
- Community: r/crewai, r/AI_Agents, r/LangChain threads (linked in section 2)
- [RubyLLM Agentic Workflows](https://rubyllm.com/agentic-workflows/)
- [RubyLLM Team PR discussion](https://github.com/crmne/ruby_llm/pull/891)
- [CrewAI introduction: Crews and Flows](https://docs.crewai.com/core-concepts/Agents)
- [CrewAI tasks and asynchronous context](https://docs.crewai.com/en/concepts/tasks)
- [CrewAI source](https://github.com/crewAIInc/crewAI)
- [RubyLLM ecosystem](https://rubyllm.com/ecosystem/)
Loading