Add a spoken-punctuation eval, and freeze its dataset in the repo by alexkroman · Pull Request #164 · AssemblyAI/blurt

alexkroman · 2026-08-26T00:38:58Z

A dictation user who wants a comma often says "comma", and the transcript comes back with
the word in it. Nothing in the harness posed that task: disfluency.py only inserts
hesitation, and both sides of every real paired corpus here are already punctuated.

--spoken-punctuation RATE layers the task onto the real corpus rather than replacing it,
so nyra's hand-annotated disfluencies stay and only the punctuation is synthetic:

input : Um we shipped it today period monday was really, quiet comma so nobody noticed period
target: We shipped it today. Monday was quiet, so nobody noticed.

How it stays honest

A mark is spoken only where the reference licenses it — the same word carries the same
mark in the target — so the correct answer never stops being the corpus's own target, and
the target is never edited. really, above survives as a comma, because the annotator
deleted the span it introduced, and asking for a mark that would then be scored as an error
teaches an instruction that commands are sometimes to be ignored.

ALL CAPS is the one operator that edits a target, and it has to be: you cannot pose
"uppercase this word" against a target with no uppercase in it.

Verified over 2,631 commands on 1,200 rows — none mis-scored, and a perfect cleanup
still scores exactly 1.0 on both axes. The whole reporting path was also exercised against
a stub server that echoes its input: it reproduces the no-cleanup floor to the digit
(test format 0.3104) at 0% converted / 100% left as words.

It selects on format, not blend. normalize() casefolds and strips marks, so a restored
comma and a missed one are the same string to the content axis, and a casing command is
invisible to it entirely.

Alongside the axes the run reports what became of the commands themselves — converted /
left as words / dropped. That is the number WER cannot give you, and it can only be
asked of a synthetic operator, because nothing else knows what was planted.

The instructions under test

Four candidates. punct-appended is the cheapest possible change to what ships:
PRIOR_WINNER plus one clause, at 2006 of the 2048 characters. That leaves 42 characters
of headroom, so GEPA cannot grow from it — punct-explicit at 1430 is the searchable seed.
Which one wins on dev and which one a search can improve are different questions.

BASELINE stays prior-winner. It has never heard of spoken punctuation, which is exactly
what makes it the right bar: the held-out comparison answers the product question — what
does teaching the shipped instruction this task buy?

A spoken run scores BASELINE plus the four, not the whole table. The six terse contrast
instructions rank framings of disfluency cleanup, none mentions punctuation, and at the
default 900-row dev split that ordering costs 5,400 model calls to re-discover.

Three things this turned up

The false-start classifier read question mark as an abandoned phrase. Rule 2 is "two
or more non-hesitation words that echo nothing", which is exactly the shape of a dictation
command, so it charged FALSE_START_WEIGHT per word — making a leftover question mark ten
errors and a leftover period one. An asymmetry nobody chose, and one that would have
pointed the search at the multi-word commands for arithmetic reasons. metrics.score now
takes a not_abandoned set that Utterance fills from the commands it planted. It defaults
to empty, and the false-start fraction over 300 nyra rows is 0.207 with spoken
punctuation and 0.207 without — so every number measured before it still holds.

A proper-noun heuristic was removed as harmful. It protected the capital on a name after
a spoken terminal mark. But that word is sentence-initial in the reference by construction,
so restoring its capital is the rule every candidate already states — name or not — and
protecting names would have handed the answer to 30% of the commands on nyra (measured:
the share whose following word appears capitalized mid-sentence somewhere in the corpus).
Only the pronoun I is exempt, because it is capitalized mid-sentence too, and
politics full stop i'm not sure is a transcript no service returns.

--help was broken. Since a help string first said ~42%: argparse %-expands help
text, so a bare % raises TypeError and the whole thing fails. Invisible to every test,
because nothing had ever asked for the help. Now pinned by one.

The frozen dataset

--dump-corpus PATH writes the loaded corpus as JSONL; --jsonl reads it back exactly,
commands included. The corpus is otherwise assembled at load time out of a dataset download
and two seeded injectors — reproducible in principle, unreviewable in practice, and a
change to an injector silently changes what every past number was measured on. A file in
the tree is diffable.

data/spoken-punctuation.jsonl is 76 rows generated from --source builtin, whose
sentences are written for this repo and so carry no third-party terms. nyra derives from
LDC-licensed Switchboard transcripts and is deliberately not committed — corpus.py
already had this stance, and this respects it. test_eval.py regenerates the committed file
and asserts it is byte-identical, so an injector change fails the suite instead of leaving a
stale dataset in the tree.

That bundled corpus grew from 12 sentences to 76 to give the task something to bite on:
internal commas, questions, exclamations, colons and semicolons that nyra never supplies
(over 400 rows it has 483 periods, 460 commas, 43 question marks and none of the other
three). The first twelve are unchanged and still first, so --limit 12 sees what it always
saw.

It also does the one thing nyra cannot. 15% of its references use a command word as
ordinary content — "one grace period, so plan accordingly", "add a comma after the second
clause" — against ~0–1% of nyra's. It is the only corpus here that can charge an
instruction for converting a word the speaker meant literally, and those rows compose with
the injector into the sharpest case on the task: one grace period comma so plan accordingly, where the word and the command sit side by side.

Everything downstream keys off what the corpus contains, not what flags were passed
(Corpus.has_commands), so the saved dataset is scored the same way as the corpus it came
from. Before that, reading it back selected on blend and printed no command outcomes.

What this does not establish

No paid run has been made yet — this is the harness and the dataset, not a result. The
README documents how to run it. Two limits worth reading before trusting a number:

Over-conversion is nearly unmeasured on nyra (~0–1% literal-use rows). The clause
forbidding it is in every candidate on product grounds, not scored ones, and unlike
REQUIRED_SAFEGUARDS it is not gated — a search is free to delete it. The run prints the
figure so the gap is visible rather than assumed away.
The input is the clean form of the task. A real speech-to-text pass applies its own
casing and punctuation model on top of a spoken command, so what it returns is probably
nearer Children period. than children period. --verify-live is the only thing here
that speaks to the real pipeline.

Testing

Python and docs only — no Swift touched. 188 tests pass (46 new), and
scripts/check.sh --portable is green: ruff format, ruff check, pytest, prettier,
markdownlint, shellcheck, shfmt, swiftlint, the settled-decision invariants. The
Swift build and tests were not run locally; CI on macos-26 is the authority there.

🤖 Generated with Claude Code

A dictation user who wants a comma often says "comma", and the transcript comes back with the word in it. Nothing in the harness posed that: the disfluency injector only inserts hesitation, and both sides of every real paired corpus here are already punctuated. `--spoken-punctuation RATE` layers the task onto the real corpus rather than replacing it, so nyra's hand-annotated disfluencies stay and only the punctuation is synthetic: input : Um we shipped it today period monday was really, quiet comma so nobody noticed period target: We shipped it today. Monday was quiet, so nobody noticed. A mark is spoken only where the reference licenses it -- the same word carries the same mark in the target -- so the correct answer never stops being the corpus's own target and the target is never edited. `really,` above survives as a comma because the annotator deleted the span it introduced, and asking for a mark that would then be scored as an error teaches an instruction that commands are sometimes to be ignored. ALL CAPS is the one exception, and has to be: a target with no uppercase in it cannot pose the task. Verified over 2,631 commands on 1,200 rows -- none mis-scored, and a perfect cleanup still scores exactly 1.0. It selects on `format`, not `blend`. normalize() casefolds and strips marks, so a restored comma and a missed one are the same string to the content axis and a casing command is invisible to it. Alongside the axes the run reports what became of the commands themselves -- converted / left as words / dropped -- which is the number WER cannot give and only a synthetic operator can be asked. Four candidate instructions come with it. `punct-appended` is the cheapest possible change to what ships: PRIOR_WINNER plus one clause, at 2006 of the 2048 characters. That leaves 42 characters of headroom, so GEPA cannot grow from it -- `punct-explicit` at 1430 is the searchable seed. BASELINE stays `prior-winner`, which has never heard of the task and is therefore the right bar: the held-out comparison answers what teaching it this buys. Three things this turned up. The false-start classifier read "question mark" as an abandoned two-word phrase -- rule 2 is "two or more non-hesitation words that echo nothing", which is exactly the shape of a dictation command -- and charged it FALSE_START_WEIGHT per word. That made a leftover "question mark" ten errors and a leftover "period" one, an asymmetry nobody chose and one that would have pointed the search at the multi-word commands for arithmetic reasons. metrics.score now takes a `not_abandoned` set that Utterance fills from the commands it planted; it defaults to empty, and the false-start fraction over 300 nyra rows is 0.207 with spoken punctuation and 0.207 without, so every number measured before it still holds. A proper-noun heuristic stood in the injector and was removed as harmful. The word after a spoken terminal mark is sentence-initial in the reference by construction, so restoring its capital is the rule every candidate already states -- name or not. Protecting names would have handed the answer to 30% of the commands on nyra. Only the pronoun I is exempt, because it is capitalized mid-sentence too. `--help` has been broken since a help string first said "~42%": argparse %-expands help text, so a bare % raises TypeError and the whole thing fails. Invisible to every test, because nothing had asked for the help. Now pinned. Also freezes a dataset. `--dump-corpus PATH` writes the loaded corpus as JSONL and `--jsonl` reads it back exactly, commands included, so a corpus otherwise assembled at load time from a download plus two seeded injectors becomes something diffable. data/spoken-punctuation.jsonl is generated from `--source builtin`, whose sentences are written for this repo and so carry no third-party terms; nyra derives from LDC-licensed Switchboard and stays out of the tree. test_eval.py regenerates the committed file and asserts it is byte-identical, so an injector change fails the suite instead of leaving a stale dataset behind. That corpus grew from 12 sentences to 76 to give the task something to bite on -- internal commas, questions, exclamations, colons and semicolons that nyra never supplies. It also does the one thing nyra cannot: 15% of its references use a command word as ordinary content ("one grace period, so plan accordingly"), against ~0-1% of nyra's, so it is the only corpus here that can charge an instruction for converting a word the speaker meant literally. Everything downstream keys off what the corpus contains rather than what flags were passed, so the saved dataset is scored the same way as the corpus it came from. Before that it selected on `blend` and reported no command outcomes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

aikido-pr-checks · 2026-08-26T00:39:25Z

+        if utterance.disfluent in table and table[utterance.disfluent] != utterance.commands:
+            raise RuntimeError(
+                "two rows share an input side but carry different planted commands: "
+                f"{utterance.disfluent!r}"


The duplicate-input error includes utterance.disfluent verbatim, potentially logging personal transcript data; report a redacted or derived identifier instead.

Details

✨ AI Reasoning
The duplicate-input error includes utterance.disfluent directly in the exception message. This is free-form transcript content originating from a corpus or user-provided JSONL and may contain personal data; raising the error can expose it through logs or error reporting.

🔧 How do I fix it?
Keep sensitive data such as emails, passwords, and tokens out of logs. When logging values tied to a user, prefer a safe identifier like a user ID over the raw input, and strip line breaks from any user-provided text you do log.

_{Reply @AikidoSec feedback: [FEEDBACK] to get better review comments in the future.}
_{Reply @AikidoSec ignore: [REASON] to ignore this issue.}
_{More info}

…after one Two gaps the "how is this weighted" question exposed. **A leftover command word cost 2, and nobody decided 2.** Plain WER charges one twice on the format axis -- a substitution for the mark that never appeared plus an insertion for the word that did -- so it fell out of the arithmetic. That is only twice what *dropping* the command costs, and dropping it is the cosmetic version: `children` where the reference wants `children.` is a missing mark, while `children period` puts a word in the user's document they never meant to write. Those are not two grades of the same mistake. metrics.COMMAND_WEIGHT charges it 3. Swept over 1114 command-carrying nyra rows, the gradient in the region rival instructions differ in (0% to 25% residue) is 0.056 unweighted and 0.104 at 3, so a weight is worth having -- and it keeps growing, so the gradient alone picks nothing. Two bounds meet at 3. FALSE_START_WEIGHT is the ceiling and it binds at 4, not 5: on the format axis a leftover command carries the mark's substitution too, so its total is the weight plus one, and at 3 that is 4 -- between a leftover filler (1) and a leftover abandoned word (5). At 4 it ties the abandoned word, which is the wrong ordering, since an abandoned span fabricates a clause the speaker never said while a command is one stray word that failed to disappear. The decaying tail bounds it from the same side: a row whose residue alone clears WER 2 stops being rankable, which is 1.0% of rows at 3, 4.6% at 5 and 20% at 8. Measured per occurrence, that leaves filler 1, command dropped 1, capital missed 1, ALL CAPS missed 1, command left in 3.97, abandoned word 5. `not_abandoned` is renamed `command_words`, because it now does two jobs and both follow from one fact -- that these words are commands rather than speech. A token charged as abandoned is never charged again as a command: rule 1 of _is_abandoned fires on a cut-off word whatever the vocabulary, so a command caught inside an abandoned run is charged at the higher weight only. Still empty for every corpus that plants none, so nothing measured before it exists moves: plain nyra's floor is unchanged at 0.6452/0.6011/0.6320 and its false-start fraction at 0.207. metrics.feedback now names leftover commands apart from disfluencies and says what they cost, and spoken_punctuation.feedback_note drops that half. Said in both places it was the same complaint twice in every reflection prompt; the split follows what each module can see -- metrics owns the leftover because that is where the weight lives, and the note owns a *dropped* command, which is one substitution among many on the format axis and nothing at all on content. **The fixture never exercised the capital after a spoken mark.** 0 of 61 terminal marks had a following word, because every bundled sentence ended at its own period -- and restoring the next word's capital is half of what a period command asks for. On nyra, where utterances run to several sentences, it is 26% of terminal marks. Fifteen multi-sentence rows put it back, at 13% and 14 concrete cases, including the two shapes the injector treats specially: a proper noun after the mark, and the pronoun I. Pinned by a test, as is the every-mark-in-the-vocabulary property. The bundled corpus is now 91 sentences and the frozen dataset 91 rows; the literal-use trap rate is 12%, still an order of magnitude above nyra's. Also fixes `--help`, which has been broken since a help string first said "~42%": argparse %-expands help text, so a bare % raises TypeError and the whole thing fails. Invisible to every test, because nothing had asked for the help. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Two changes, both from the same objection: a command has to actually do something, and nothing but the commands should be able to move the score. **--punctuation-only.** The input now differs from the target by the commands alone. No disfluencies are injected, and a paired source's verbatim side is discarded in favour of its clean one -- so on nyra the eval scores against the intended transcripts, real conversational English, and nothing a cleanup does about hesitation can move the number. Every row is guaranteed at least one command (spoken_punctuation.inject gained `require`), rows the injector cannot plant one in are dropped, and the rate defaults to 1.0 because the mode is meaningless without it. The property is asserted on the alignment rather than by reconstruction: on the content axis the input is the target plus command words, no word substituted and none dropped. **The utterance-final mark is never spoken.** This is the half that makes a command a test of anything. A dictated "period" on the last word asks for a mark any instruction produces unprompted -- every candidate here says "restore punctuation" -- so obeying it and ignoring it score identically and the row is padding. Those were 55% of everything BUILTIN_SAMPLE licensed and 37% of nyra's. What survives is discriminative by construction, and spoken_punctuation.effect prices it against the reference with that one command's effect undone: a mid-utterance terminal mark is worth 2 (the mark, and the capital behind it), an internal mark 1, ALL CAPS one per word. A test asserts nothing is worth 0 over the whole corpus -- a zero means an invariant broke, not that the row is unusable. That exclusion is why PUNCTUATION_SAMPLE exists as a second bundled source. A disfluency smoke corpus wants plain declaratives with room to inject hesitation into; a punctuation corpus wants internal marks, and 37 of BUILTIN_SAMPLE's 91 rows have none. All 80 of these carry at least one, most several, and 14% use a command word as ordinary content -- against ~0% of nyra's, which is what makes this the only corpus here that can charge an instruction for converting a word the speaker meant literally. The committed dataset is regenerated from it, punctuation-only. Selection stays on format WER rather than the raw conversion rate, and deliberately. "Fraction of commands obeyed" is the number the task is named for and it is blind to the damage done getting there: an instruction that converts every command and deletes half the sentence scores 1.0 on it, and GEPA would find that. On a punctuation-only corpus format WER is not a compromise either -- every error it can charge is a command failure or collateral damage to text the instruction was told to leave alone, and both belong in the objective. Conversion rate is reported. Two pure candidates join the table. punct-mapping and punct-literal-guard carry no disfluency rules at all, which matters here: there are no disfluencies to remove, so every deletion those rules invite is damage. The composites stay, because whether carrying them costs anything measurable is one of the things a run answers. Also fixes a real inconsistency the new test caught. `caps on ... caps off` emits the word "off", but the Command only knew ("caps", "on"), so "off" sat in the input outside command_words and a cleanup that left `caps off` standing was charged one ordinary insertion for it instead of COMMAND_WEIGHT -- the same command priced two ways depending on which half survived. Command.closing records it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Both changes come from working out how to run --punctuation-only over the whole corpus, which is 3442 nyra rows -> 900 dev. `--candidates baseline` was silently ignored on a spoken run: resolve_candidates returned the full punctuation set whenever the corpus carried commands, so the flag was a lie exactly where it earns its keep. At 900 dev rows the default sweep is 7 x 900 = 6300 calls, and having already run it once, paying for it again to reach the search is the one thing anyone would reach for the flag to avoid. `--start` now takes any candidate name as well as prior-winner and best-candidate, validated against the run's own table before anything is spent. Together the two mean a search picks up where a ranking left off: --start punct-explicit --candidates baseline. The README's punctuation section now carries the arithmetic for four run shapes, from the 420-call committed fixture to the full 4000-row load, with a note that ~14% of rows drop because their only mark is utterance-final and that HF_TOKEN is needed for a load that size. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`RuntimeError: datasets-server returned HTTP 500 ... {"error":"Unexpected error."}` killed a 4000-row load. Probing six offsets of the same dataset minutes later, every page came back 200 and num_rows_total was 4458 — so the dataset was fine and the failure was a blip. `_fetch_page` retried 429 only, on the reasoning that every other status is a fact about the request that waiting will not change, and that a load which silently took four minutes to work around a broken upstream is harder to diagnose than one that said so. That was wrong twice over. It contradicted itself — the same docstring called the 500s intermittent — and the diagnosis argument is answered by printing each retry as it happens, not by abandoning the load. What it produced instead was a 45-page load discarding 30 fetched pages over one blip, *before the first model call*, which is precisely the failure the 429 backoff was written to prevent: a run set up to take hours dying at second zero. So the 5xx family is retried on the same bounded backoff, and a 4xx still raises at once — 403 and 404 really are facts about the request. The give-up message now says which kind it was and what to do about it, since "authenticate to raise the limit" is useless advice about a 500 and the opposite of what a caller needs to hear. Also fixes a bug in the new test's own helper, which is worth naming because it silently passed: `(sleeps or []).append(...)` appends to a throwaway list whenever `sleeps` is empty, which is exactly its state on the first call. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`--candidates baseline --start NAME` is the resume path added one commit ago, and it crashed: the seed's dev score was read with `dict(dev_rows)[seed_name]`, so skipping the sweep -- the entire point of `--candidates baseline` -- raised KeyError before the search began. It is the exact command the README recommends for a large-corpus run. The seed is scored instead of the lookup being made lenient, because the seed is what the search's result has to be measured against: "did evolving this beat starting from it" is the question the run answers, and a missing comparison would leave only the delta against BASELINE, which on this task is an instruction that has never heard of the task. One dev sweep, against the seven the full table costs. The dev winner is recomputed once the seed is in, so a search that fails falls back to the better of the seed and the bar rather than to whichever was measured first. Tested by driving `main` through the path with a fake `program` module injected into sys.modules, which keeps the suite free of DSPy -- the property two other tests here exist to protect -- while still exercising the CLI's control flow, where this mistake lived. Reverting the fix reproduces the KeyError under that test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

+    class Optimized:
+        signature = types.SimpleNamespace(instructions=instruction)
+
+    module.ModelSpec = lambda **kwargs: types.SimpleNamespace(**kwargs)


`--baseline NAME` moves what a run compares against: the row scored on held-out test, what `--candidates baseline` scores, and the one instruction held to the safeguard requirement. Without it a second round spends a 900-call dev sweep re-confirming that prior-winner scores what the first round already said it scores -- measured on the first punctuation-only run, which learned 0.6124 for a second time. candidates.BASELINE does not move, deliberately. It names the string CleanupInstruction.text actually sends, and the whole "what does teaching this task buy over what ships" comparison depends on it continuing to mean that. A flag is the right place for "the bar for this run". Three details that follow. The promoted baseline is de-duplicated out of the punctuation sweep, since it is itself one of those candidates and would otherwise be scored twice. Whatever a run compares against is what it would fall back to shipping, so the safeguard check follows the flag rather than the constant. And `--start prior-winner` loses its special case: now that --start takes any candidate name, "prior-winner" is simply one of them, which removes a branch that only existed to spell BASELINE. Worth noting what was already true and is easy to misread: the seed is already the promotion bar. The dev winner is recomputed once the seed is scored, so an evolved instruction has to beat the instruction it evolved from. That is why a search returning 0.7706 against a 0.7781 seed was correctly refused, rather than being promoted for beating prior-winner's 0.6124. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

"What percent of statements had an error?" could not be answered from a run's own output, which is the product question and a bad thing to be missing. `--out` recorded means only -- format 0.8081, commands_converted 0.682 -- and a mean of 0.682 over 2.26 commands a row is consistent with failures sitting in a third of rows or in nearly all of them. Those are different products and a per-command average cannot distinguish them. So `evaluate` now also reports, per candidate: rows_with_command_failure any command missed in this utterance rows_with_command_literal a command word pasted into this utterance rows_exact_content/format/blend came back exactly right The last of those is the one a user would recognise. The first two get their own columns in the command table, labelled ROWS to keep them from being read as more per-command rates. For scale on why this matters: at the measured 31.8% per-command failure rate, and the actual command-count distribution of the 450-row nyra test split (149 rows with one command, 136 with two, 95 with three, up to one row with nine), independent failures would put >=1 error in 53.7% of utterances. Clustering would lower that -- which is exactly the thing worth measuring rather than assuming. Also worth recording: the corpus load that produced that distribution hit four rate limits and an HTTP 502 and completed anyway, which is the retry added two commits ago earning its place on its first real outing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

aikido-pr-checks Bot reviewed Aug 26, 2026

View reviewed changes

github-code-quality Bot found potential problems Aug 26, 2026

View reviewed changes

Comment thread evals/dictation-prompt/test_eval.py Fixed

Comment thread evals/dictation-prompt/test_eval.py Fixed

alexkroman-assembly and others added 4 commits August 25, 2026 18:40

github-code-quality Bot found potential problems Aug 26, 2026

View reviewed changes

Comment thread evals/dictation-prompt/test_eval.py

class Optimized:

signature = types.SimpleNamespace(instructions=instruction)

module.ModelSpec = lambda **kwargs: types.SimpleNamespace(**kwargs)

alexkroman-assembly and others added 2 commits August 25, 2026 20:33

Navigation Menu

Uh oh!

Add a spoken-punctuation eval, and freeze its dataset in the repo - #164

Add a spoken-punctuation eval, and freeze its dataset in the repo#164
alexkroman wants to merge 8 commits into
mainfrom
eval/spoken-punctuation

alexkroman commented Aug 26, 2026

Uh oh!

aikido-pr-checks Bot Aug 26, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

alexkroman commented Aug 26, 2026

How it stays honest

The instructions under test

Three things this turned up

The frozen dataset

What this does not establish

Testing

Uh oh!

aikido-pr-checks Bot Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants