-
Notifications
You must be signed in to change notification settings - Fork 498
feat: Add speech pacing and rhetorical evaluator #357
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
e104df8
cad0bb7
77e9ce3
4f736ca
62a879d
61ed711
2b706f4
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
| # Speech Pacing & Rhetorical Evaluator | ||
|
|
||
| A Lamatic AgentKit template for evaluating speeches before live delivery. | ||
|
|
||
| ## Problem | ||
|
|
||
| A speech can be grammatically polished and still fail in a timed presentation because it is too dense, difficult to articulate, or rhetorically front-loaded. Traditional word-count estimates do not account for linguistic complexity, jargon, acronyms, numbers, long sentences, or intentional pauses. | ||
|
|
||
| ## What this agent does | ||
|
|
||
| Give it a speech draft and a target window such as 5–7 minutes. The evaluator returns: | ||
|
|
||
| - Projected speaking-time range using word count, baseline WPM, and complexity signals. | ||
| - Timing verdict against the target window. | ||
| - Content-density and pacing risks. | ||
| - Jargon and articulation flags. | ||
| - Opening, structure, audience connection, transitions, and conclusion scores. | ||
| - Specific commendations tied to the submitted draft. | ||
| - Three prioritized recommendations. | ||
| - Suggested pause and emphasis points. | ||
| - A focused rehearsal plan. | ||
|
|
||
| The evaluator explicitly distinguishes a draft-based timing estimate from measured speaking performance. Audio/video analysis is outside the scope of this template. | ||
|
|
||
| ## Example input | ||
|
|
||
| ```json | ||
| { | ||
| "speech": "Good evening everyone... [complete speech draft]", | ||
| "targetMinMinutes": 5, | ||
| "targetMaxMinutes": 7, | ||
| "speakerWpm": 130 | ||
| } | ||
| ``` | ||
|
|
||
| If `speakerWpm` is omitted, the evaluator uses 130 WPM as its baseline assumption. | ||
|
|
||
| ## Example output structure | ||
|
|
||
| ```text | ||
| Executive verdict | ||
| Timing analysis | ||
| Pacing and density risks | ||
| Jargon and articulation flags | ||
| Rhetorical scorecard | ||
| Commendations | ||
| Priority recommendations | ||
| Pause and emphasis map | ||
| Rehearsal plan | ||
| ``` | ||
|
|
||
| ## Why this is different | ||
|
|
||
| AgentKit contains text-generation and grammar-oriented kits, but this contribution focuses specifically on the spoken word as a timed performance artifact. Its evaluation dimensions combine temporal constraints with rhetorical structure and articulation risk rather than treating the draft as ordinary prose. | ||
|
|
||
| ## Flow | ||
|
|
||
| `Speech Evaluation Request → Evaluate Speech → Evaluation Report` | ||
|
|
||
| The flow uses a dedicated constitution and externalized evaluator prompts so the evaluation criteria can evolve independently from the flow graph. | ||
|
|
||
| ## Setup | ||
|
|
||
| 1. Import or deploy the `speech-pacing-evaluator` flow in Lamatic Studio. | ||
| 2. Configure an LLM provider available to your Lamatic project. | ||
| 3. Invoke the GraphQL/API trigger with `speech`, `targetMinMinutes`, `targetMaxMinutes`, and optionally `speakerWpm`. | ||
| 4. Review the returned evaluation and rehearse against the recommended timing/pause plan. | ||
|
|
||
| ## Limitations | ||
|
|
||
| - Timing is an estimate, not a stopwatch measurement. | ||
| - The current flow evaluates text only; it cannot detect actual vocal rushing, filler words, pronunciation problems, volume, or pauses that differ from the draft. | ||
| - For highly technical speeches, domain terminology may be legitimate; jargon flags should therefore be treated as audience-dependent recommendations rather than automatic errors. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| # Speech Pacing & Rhetorical Evaluator | ||
|
|
||
| ## Overview | ||
|
|
||
| This AgentKit flow acts as a rigorous evaluator for spoken presentations. It accepts a speech draft and a target time window, estimates delivery time using both word count and linguistic complexity, identifies phrases likely to slow delivery, and evaluates rhetorical structure. | ||
|
|
||
| The kit is designed for high-stakes presentations, Toastmasters speeches, interviews, executive briefings, sales pitches, conference talks, and other timed speaking situations where a written draft can look strong but fail at delivery speed or audience comprehension. | ||
|
|
||
| ## Inputs | ||
|
|
||
| - `speech`: the complete speech draft as plain text. | ||
| - `targetMinMinutes`: minimum acceptable speaking time. | ||
| - `targetMaxMinutes`: maximum acceptable speaking time. | ||
| - `speakerWpm` (optional): baseline words-per-minute assumption. Defaults to 130 WPM. | ||
|
|
||
| ## Evaluation pipeline | ||
|
|
||
| 1. Normalize and measure the draft. | ||
| 2. Calculate word count, sentence count, average sentence length, lexical density, and a complexity-adjusted WPM estimate. | ||
| 3. Compare projected duration with the requested time window. | ||
| 4. Detect dense jargon, long phrases, overloaded sentences, repeated concepts, and likely breath/pause points. | ||
| 5. Assess rhetorical architecture: opening hook, thesis/promise, transitions, evidence/story progression, audience relevance, callback/cohesion, and conclusion. | ||
| 6. Produce a structured report with commendations, risks, prioritized recommendations, and a revised pacing plan. | ||
|
|
||
| ## Output | ||
|
|
||
| The response should contain: | ||
|
|
||
| - Timing verdict and projected duration. | ||
| - Timing range based on baseline and complexity-adjusted delivery. | ||
| - Pacing risk level. | ||
| - Content-density assessment. | ||
| - Jargon/complexity flags with suggested simplifications. | ||
| - Opening score and explanation. | ||
| - Rhetorical structure score. | ||
| - Conclusion score and explanation. | ||
| - Top strengths. | ||
| - Top three improvements, prioritized by impact. | ||
| - Suggested pause/emphasis map. | ||
| - A concise rehearsal plan. | ||
|
|
||
| ## Timing model | ||
|
|
||
| Do not claim that word count can predict an individual's exact speaking time. Use it as an estimate. Start from the speaker's supplied WPM or the 130 WPM default, then widen the estimate when sentence complexity, punctuation, parentheticals, numbers, acronyms, or dense terminology indicate additional cognitive and articulation load. | ||
|
|
||
| ## Guardrails | ||
|
|
||
| - Evaluate the speech, not the person. | ||
| - Do not infer protected or sensitive personal attributes from writing style. | ||
| - Do not invent delivery observations that require audio/video evidence. Label them as draft-based risks. | ||
| - Do not present estimated timing as measured performance. | ||
| - Preserve the speaker's intended meaning when recommending edits. | ||
| - Avoid generic praise; every commendation should point to an observable feature of the draft. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| # Default Constitution | ||
|
|
||
| ## Identity | ||
| You are a rigorous speech and presentation evaluator. You assess the submitted draft against the stated timing target and rhetorical objectives. | ||
|
|
||
| ## Accuracy | ||
| - Distinguish draft-based estimates from observed speaking performance. | ||
| - Never claim that the speaker actually paused, rushed, stumbled, or emphasized a phrase unless audio/video evidence is supplied. | ||
| - Explain assumptions behind timing estimates. | ||
|
|
||
| ## Evaluation | ||
| - Prefer specific, actionable observations over generic encouragement. | ||
| - Evaluate opening, structure, transitions, content density, audience relevance, and conclusion. | ||
| - Flag jargon when it may slow articulation or increase audience processing load. | ||
| - Recommend simplification only when it improves clarity without changing intended meaning. | ||
|
|
||
| ## Safety and privacy | ||
| - Do not infer protected or sensitive characteristics about the speaker. | ||
| - Do not expose secrets or credentials included accidentally in a draft. | ||
| - Treat personal names, contact details, and other personal information as unnecessary to the evaluation unless required for context. | ||
|
|
||
| ## Output discipline | ||
|
Comment on lines
+3
to
+22
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win Fix heading spacing at the constitution template source.
Based on learnings, 🧰 Tools🪛 markdownlint-cli2 (0.23.2)[warning] 3-3: Headings should be surrounded by blank lines (MD022, blanks-around-headings) [warning] 6-6: Headings should be surrounded by blank lines (MD022, blanks-around-headings) [warning] 11-11: Headings should be surrounded by blank lines (MD022, blanks-around-headings) [warning] 17-17: Headings should be surrounded by blank lines (MD022, blanks-around-headings) [warning] 22-22: Headings should be surrounded by blank lines (MD022, blanks-around-headings) 🤖 Prompt for AI AgentsSources: Learnings, Linters/SAST tools |
||
| - Use clear headings and concise tables where useful. | ||
| - Give a prioritized set of recommendations. | ||
| - Include a timing verdict and uncertainty caveat. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,103 @@ | ||
| // Flow: speech-pacing-evaluator | ||
|
|
||
| // ── Meta ────────────────────────────────────────────── | ||
| export const meta = { | ||
| "name": "Speech Pacing & Rhetorical Evaluator", | ||
| "description": "Evaluates a speech draft against a target speaking window, estimates delivery time, identifies pacing and jargon risks, and produces a structured rhetorical evaluation with actionable refinements.", | ||
| "tags": ["🎤 Speech", "⏱️ Pacing", "🧭 Rhetoric", "💼 Presentation"], | ||
| "testInput": null, | ||
| "githubUrl": "", | ||
| "documentationUrl": "", | ||
| "deployUrl": "", | ||
| "author": { "name": "Rahul Rajesh", "email": "rahullrajesh@users.noreply.github.com" } | ||
| }; | ||
|
|
||
| // ── Inputs ──────────────────────────────────────────── | ||
| export const inputs = {}; | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
rg -n -C 5 \
'export const inputs|advance_schema|targetMinMinutes|targetMaxMinutes|speakerWpm' \
kits --glob '*.ts' --glob '*.md' || trueRepository: Lamatic/AgentKit Length of output: 50373 🏁 Script executed: #!/bin/bash
set -euo pipefail
file='kits/speech-pacing-evaluator/flows/speech-pacing-evaluator.ts'
printf '%s\n' '--- target flow ---'
cat -n "$file"
printf '%s\n' '--- repository references to timing inputs ---'
rg -n -C 4 --glob '*.ts' --glob '*.tsx' --glob '*.js' --glob '*.json' --glob '*.md' \
'targetMinMinutes|targetMaxMinutes|speech-pacing-evaluator|speakerWpm' . \
| head -n 500
printf '%s\n' '--- kit files ---'
git ls-files 'kits/speech-pacing-evaluator/**'Repository: Lamatic/AgentKit Length of output: 13699 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '--- trigger-schema tooling and validation references ---'
rg -n -C 3 --glob '!kits/speech-pacing-evaluator/**' \
'advance_schema|trigger schema|GraphQL.*trigger|graphql.*input|input.*validation|JSON Schema' \
. | head -n 1000
printf '%s\n' '--- numeric trigger schemas in flow exports ---'
python3 - <<'PY'
import re
from pathlib import Path
for path in Path("kits").glob("*/flows/*.ts"):
text = path.read_text(errors="replace")
for match in re.finditer(r'"advance_schema":\s*"((?:\\.|[^"])*)"', text):
schema = bytes(match.group(1), "utf-8").decode("unicode_escape")
if re.search(r'"(?:number|integer|minimum|maximum|required)"', schema):
print(f"{path}: {schema}")
PY
printf '%s\n' '--- speech kit contract text ---'
cat -n kits/speech-pacing-evaluator/agent.md
cat -n kits/speech-pacing-evaluator/README.mdRepository: Lamatic/AgentKit Length of output: 50373 🌐 Web query:
💡 Result: In Lamatic Studio, input validation for workflows—including those triggered via GraphQL—is primarily managed by ensuring that your input payload matches the schema defined within the Workflow configuration [1][2]. While Lamatic allows for advanced schema customization in its GraphQL trigger nodes, it does not explicitly document a built-in syntax for applying numeric minimum and maximum constraints directly within the GraphQL trigger configuration block [3][4]. Instead, validation is generally handled as follows: 1. Workflow Schema Alignment: When you define an input schema for a workflow (often via the Studio interface), your external GraphQL request must provide a payload that conforms to these defined types and structures [1][2]. 2. Error Handling: If the payload sent in your GraphQL Citations:
🏁 Script executed: #!/bin/bash
set -euo pipefail
python3 - <<'PY'
import re
from pathlib import Path
flow = Path("kits/speech-pacing-evaluator/flows/speech-pacing-evaluator.ts").read_text()
prompt = Path(
"kits/speech-pacing-evaluator/prompts/"
"speech-pacing-evaluator_generate-text_user.md"
).read_text()
checks = {
"inputs export is empty": bool(re.search(r"export const inputs\s*=\s*\{\s*\};", flow)),
"advance_schema is empty": '"advance_schema": ""' in flow,
"targetMinMinutes is interpolated": "{{triggerNode_1.output.targetMinMinutes}}" in prompt,
"targetMaxMinutes is interpolated": "{{triggerNode_1.output.targetMaxMinutes}}" in prompt,
"speakerWpm has fallback": "speakerWpm as 130 when it is missing or invalid" in prompt,
"targetMinMinutes has fallback": bool(re.search(r"targetMinMinutes.{0,120}(?:default|fallback|missing|invalid)", prompt, re.I)),
"targetMaxMinutes has fallback": bool(re.search(r"targetMaxMinutes.{0,120}(?:default|fallback|missing|invalid)", prompt, re.I)),
}
for name, result in checks.items():
print(f"{name}: {'yes' if result else 'no'}")
PYRepository: Lamatic/AgentKit Length of output: 384 Enforce the timing-input contract. The flow leaves 🤖 Prompt for AI Agents |
||
|
|
||
| // ── References ──────────────────────────────────────── | ||
| export const references = { | ||
| "constitutions": { "default": "@constitutions/default.md" }, | ||
| "prompts": { | ||
| "speech_pacing_evaluator_generate_text_user": "@prompts/speech-pacing-evaluator_generate-text_user.md", | ||
| "speech_pacing_evaluator_generate_text_system": "@prompts/speech-pacing-evaluator_generate-text_system.md" | ||
| } | ||
| }; | ||
|
|
||
| // ── Nodes & Edges ───────────────────────────────────── | ||
| export const nodes = [ | ||
| { | ||
| "id": "triggerNode_1", | ||
| "type": "triggerNode", | ||
| "position": { "x": 0, "y": 0 }, | ||
| "data": { | ||
| "nodeId": "graphqlNode", | ||
| "trigger": true, | ||
| "values": { | ||
| "nodeName": "Speech Evaluation Request", | ||
| "responeType": "realtime", | ||
| "advance_schema": "" | ||
| } | ||
| } | ||
| }, | ||
| { | ||
| "id": "LLMNode_1", | ||
| "type": "dynamicNode", | ||
| "position": { "x": 0, "y": 0 }, | ||
| "data": { | ||
| "nodeId": "LLMNode", | ||
| "values": { | ||
| "nodeName": "Evaluate Speech", | ||
| "tools": [], | ||
| "prompts": [ | ||
| { "id": "speech-user", "role": "user", "content": "@prompts/speech-pacing-evaluator_generate-text_user.md" }, | ||
| { "id": "speech-system", "role": "system", "content": "@prompts/speech-pacing-evaluator_generate-text_system.md" } | ||
| ], | ||
| "memories": "", | ||
| "messages": "", | ||
| "generativeModelName": "gpt-4o-mini" | ||
| } | ||
| } | ||
| }, | ||
| { | ||
| "id": "graphqlResponseNode_1", | ||
| "type": "dynamicNode", | ||
| "position": { "x": 0, "y": 0 }, | ||
| "data": { | ||
| "nodeId": "graphqlResponseNode", | ||
| "values": { | ||
| "nodeName": "Evaluation Report", | ||
| "outputMapping": "{\n \"evaluation\": \"{{LLMNode_1.output.generatedResponse}}\"\n}" | ||
| } | ||
| } | ||
| } | ||
| ]; | ||
|
|
||
| export const edges = [ | ||
| { | ||
| "id": "triggerNode_1-LLMNode_1", | ||
| "source": "triggerNode_1", | ||
| "target": "LLMNode_1", | ||
| "sourceHandle": "bottom", | ||
| "targetHandle": "top", | ||
| "type": "defaultEdge" | ||
| }, | ||
| { | ||
| "id": "LLMNode_1-graphqlResponseNode_1", | ||
| "source": "LLMNode_1", | ||
| "target": "graphqlResponseNode_1", | ||
| "sourceHandle": "bottom", | ||
| "targetHandle": "top", | ||
| "type": "defaultEdge" | ||
| }, | ||
| { | ||
| "id": "response-graphqlResponseNode_1", | ||
| "source": "triggerNode_1", | ||
| "target": "graphqlResponseNode_1", | ||
| "sourceHandle": "to-response", | ||
| "targetHandle": "from-trigger", | ||
| "type": "responseEdge" | ||
| } | ||
| ]; | ||
|
|
||
| export default { meta, inputs, references, nodes, edges }; | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| export default { | ||
| name: "Speech Pacing & Rhetorical Evaluator", | ||
| description: "Evaluates a speech draft against a target speaking window, estimates delivery time, identifies pacing and jargon risks, and produces a structured rhetorical evaluation with actionable refinements.", | ||
| version: '1.0.0', | ||
| type: 'template' as const, | ||
| author: { name: "Rahul Rajesh", email: "rahullrajesh@users.noreply.github.com" }, | ||
| tags: ["speech", "presentation", "pacing", "rhetoric", "communication"], | ||
| steps: [ | ||
| { id: "speech-pacing-evaluator", type: 'mandatory' as const } | ||
| ], | ||
| links: { | ||
| "github": "https://github.com/Lamatic/AgentKit/tree/main/kits/speech-pacing-evaluator" | ||
| }, | ||
| }; |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| You are the Speech Pacing & Rhetorical Evaluator. | ||
|
|
||
| Your job is to give a demanding but constructive evaluation of a written speech draft intended for live delivery. Analyze timing, linguistic complexity, rhetorical structure, opening, transitions, audience relevance, and conclusion. | ||
|
|
||
| Timing requirements: | ||
| - Use the supplied speakerWpm when present; otherwise use 130 WPM. | ||
| - Calculate a baseline duration from word count. | ||
| - Adjust the estimate modestly upward when the draft contains unusually long sentences, acronyms, numbers, dense technical terminology, parentheticals, or punctuation that implies pauses. | ||
| - Provide an estimated range rather than false precision. | ||
| - Clearly state that this is a draft-based estimate and not a measurement of the speaker's actual delivery. | ||
|
|
||
| Rhetorical evaluation requirements: | ||
| - Score opening, structure, audience connection, transitions, content density, and conclusion on a 1–10 scale. | ||
| - Identify the strongest rhetorical device or structural choice actually present. | ||
| - Identify the three highest-impact weaknesses. | ||
| - Flag jargon or complex wording that could create articulation or comprehension friction. | ||
| - Recommend specific substitutions, cuts, moves, or pauses where justified. | ||
| - Distinguish content that should be cut from content that should simply be slowed, paused, or emphasized. | ||
|
|
||
| Return a professional report with these sections: | ||
| 1. Executive verdict | ||
| 2. Timing analysis | ||
| 3. Pacing and density risks | ||
| 4. Jargon and articulation flags | ||
| 5. Rhetorical scorecard | ||
| 6. Commendations | ||
| 7. Priority recommendations | ||
| 8. Pause and emphasis map | ||
| 9. Rehearsal plan | ||
|
|
||
| Do not invent facts about the speaker, audience, or delivery. Do not claim to have heard the speech. Be precise, candid, and actionable. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| Evaluate the following speech draft. | ||
|
|
||
| Speech draft: | ||
| {{triggerNode_1.output.speech}} | ||
|
Comment on lines
+3
to
+4
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win Mission requirement: isolate the speech draft from control instructions. The value at Proposed prompt boundary-Speech draft:
-{{triggerNode_1.output.speech}}
+BEGIN_SPEECH_DRAFT
+{{triggerNode_1.output.speech}}
+END_SPEECH_DRAFT🤖 Prompt for AI Agents |
||
|
|
||
| Target minimum time: {{triggerNode_1.output.targetMinMinutes}} minutes | ||
| Target maximum time: {{triggerNode_1.output.targetMaxMinutes}} minutes | ||
| Speaker baseline WPM: {{triggerNode_1.output.speakerWpm}} | ||
|
|
||
| Return the complete structured evaluation requested by the system instructions. Treat speakerWpm as 130 when it is missing or invalid. Use the draft itself as the evidence base. Make timing estimates explicit and distinguish them from actual measured delivery. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Keep the documented scorecard consistent with the system prompt.
The system prompt requires six numeric dimensions. The documentation omits fields differently in each file.
kits/speech-pacing-evaluator/agent.md#L27-L40: document scores for opening, structure, audience connection, transitions, content density, and conclusion.kits/speech-pacing-evaluator/README.md#L11-L20: add the content-density score to the documented output.📍 Affects 2 files
kits/speech-pacing-evaluator/agent.md#L27-L40(this comment)kits/speech-pacing-evaluator/README.md#L11-L20🤖 Prompt for AI Agents