Evaluation frameworks for AI-assisted facilitation — measuring facilitation quality through structured comparison rather than subjective assessment.
Instead of asking "Is this good facilitation?" (subjective), OFL evals ask comparative questions:
- "How similar is this conversation to Socratic dialogue?"
- "How close is this facilitation to restorative justice principles?"
This relative comparison produces measurable feedback signals for training and improving AI facilitators.
Every facilitation conversation can be characterized along three dimensions:
| Outcome Type | Description | Example Methods |
|---|---|---|
| Agreement Building | Help group reach consensus | Delphi, Consensus Workshop |
| Preference Elicitation | Surface individual preferences | Polling, Harmonica |
| Error Surfacing | Identify gaps in thinking | Devil's Advocate, Red Team |
| Perspective Taking | Expose to other viewpoints | Cross-pollination, Fishbowl |
| Synthesis | Aggregate into actionable output | Affinity Mapping |
| Ideation | Generate new ideas | Brainstorming, Six Hats |
| Conflict Resolution | Resolve disagreements | NVC, Mediation |
Intervention styles: Non-directive (logistics only) → Semi-directive (questions, summaries) → Directive (guide thinking, challenge)
Question types: Open, Closed, Probing, Clarifying, Challenging
Timing: Scheduled (predetermined) / Responsive (react to input) / Threshold-based (conditions met)
Interaction modes: One-to-one, Small group (3-12), Large group (12+), Plenary
Power dynamics: Hierarchy sensitivity, anonymity support, minority voice protection
The key mechanism: compute "signatures" of conversations based on Why-How-Who dimensions, then compare them to reference methodologies.
- Label dialogue acts with Why-How-Who tags
- Count frequencies of each tag type
- Create a vector representing conversation characteristics
- Compare to known methodology signatures (distance = similarity measure)
| Category | Metrics |
|---|---|
| Process | Intervention frequency/timing, question type distribution, speaking time balance, topic coverage |
| Outcome | Agreement level, idea quantity/quality, participant satisfaction |
| Signature | Distance to target methodology, consistency within session, appropriate adaptation |
A CLI tool that processes raw facilitation transcripts into anonymized, WHoW-annotated benchmarks.
npm install
npx tsx src/process-transcript.ts --input <file> --methodology <name> --date <YYYY-MM-DD> [--facilitator <names>] [--dry-run]Pipeline: parse (auto-detect format) → anonymize (names, PII) → annotate (gpt-4o-mini WHoW tagging) → signature (tag frequency vectors) → render (markdown + YAML frontmatter).
See src/ for implementation. Add your OpenAI key to .env.
Signatures are comparative, but some quality questions still reduce to a judgment ("is this a good
close?"). Where an LLM-as-judge scores such an axis, it needs grounding in expert taste — the
gap the weval methodology addresses. calibration/
holds real, anonymized facilitation turns + the rubric criterion in plain English, for expert
facilitators to label good / weak. Those labels recalibrate the judge so it scores like an expert,
not a model default. First pack: the closing-turn axis.
why-how-who-framework.md # Full framework specification
src/ # Transcript processor CLI
schemas/ # Data schemas for evaluation results
prompts/ # LLM prompts for automated tagging
benchmarks/ # Annotated benchmark transcripts
transcripts/ # Raw input transcripts
calibration/ # Human labels calibrating LLM-as-judge to expert taste
- Pattern development — Encode and compare facilitation methods using consistent dimensions
- AI training — Generate labeled datasets and feedback signals for reinforcement learning
- Quality assessment — Automated comparison of conversations to reference methodologies
Fora is a corpus of 262 facilitated dialogues (39,911 turns) with human annotations for 7 facilitation strategies and 2 personal sharing types. It's the closest academic dataset to what OFL evals produces. Key differences:
- Fora annotates facilitator strategies (follow-up questions, making connections, etc.) and participant sharing — narrower but human-validated
- WHoW annotates purpose (why), technique (how), and interaction direction (who) — broader, captures dimensions Fora doesn't (purpose, directionality)
- Complementary: running WHoW annotation on Fora transcripts would validate our taxonomy against human ground truth
Fora data is available by request from github.com/schropes/fora-corpus.
ConvoKit is a Python toolkit for conversational analysis with a standardized corpus format and built-in analysis transformers. Used by 30+ datasets including Fora.
Relevant analysis tools that would work on our data if we export in ConvoKit format:
- Politeness Strategies — lexical/parse-based politeness scoring (facilitator tone measurement)
- Linguistic Coordination — measures whether participants mirror facilitator language (rapport signal)
- CRAFT Forecasting — predicts conversation derailment (evaluates whether facilitation prevents breakdown)
- Redirection detection — built-in tool for what our
redirecttag captures - Linguistic Diversity — quantifies speaker variation within conversations
Our pipeline handles the upstream problem ConvoKit doesn't: raw transcript → structured annotated corpus. ConvoKit handles the downstream analysis. Adding ConvoKit-format export would make all these tools available on our benchmarks.
weval is an open, CC0 platform for public AI evals: evals are written as blueprints — YAML pairing prompts with should/should_not rubrics, scored by LLM-as-judge with multi-judge consensus and inter-rater reliability (Krippendorff's α). Two ways it connects to OFL evals:
- Format (interop target). A Why-How-Who facilitation eval can be expressed as a weval blueprint — making OFL evals runnable on weval-shaped infrastructure, not just this pipeline. Planned interop direction (the eval-side analog of a ConvoKit export).
- Judge calibration. weval's methodology — consensus judging + Krippendorff's α reliability bands + bias countermeasures — is prior art for grounding any LLM-as-judge scoring added here.
CIP run weval and have been a proposed validation partner for the OFL eval suite. See the weval wiki page and the blueprint format.
- Joseph Low, Cooperative AI Fellowship research (2026)
- Why-How-Who Framework — Full specification
- OFL Pattern Schema — Patterns include evaluation criteria
- WHoW Framework — Chen et al. 2024, cross-domain moderation analysis
- Discussion Quality in the LLM Era — Korre et al. 2025
MIT