PolyAudio

Run one model across many audio tasks and report per-task metrics on a shared harness — so a unified model can be compared honestly against single-task baselines.

Audio research keeps shipping "one model to rule them all": a single encoder or seq2seq model that claims to do keyword spotting and language ID and tagging and transcription. The claim is only as good as the comparison behind it. PolyAudio is the comparison: a small, dependency-light harness where the same model object is driven across heterogeneous tasks, each scored with the metric that task actually cares about, next to trivial baselines that show how much of the score is just guessing.

No neural network is bundled. PolyAudio scores predictions, so it stays tiny (numpy is the only runtime dependency) and works with whatever produced them — a 300M-parameter unified model or a majority-class baseline.

pip install polyaudio

Why per-task metrics on a shared harness

A single accuracy number across tasks is meaningless — WER and mAP don't average. PolyAudio keeps each task's metric intact and reports them side by side, plus one honest summary (mean of primary scores). The point is the table, not a leaderboard number:

model: unified-v2
task              kind        n    primary   score    time_s
----------------  ----------  ---  --------  -------  ------
keyword-spotting  multiclass  120  accuracy  0.9083   0.004
language-id       multiclass  200  accuracy  0.7350   0.006
audio-tagging     multilabel  180  map       0.4120   0.011
emotion-valence   regression  150  pearson   0.5810   0.003
speech-mini       sequence    100  wer       0.1840   0.009

mean primary score: 0.5641

Quick start

from polyaudio import Evaluator, load_suite
from polyaudio.models import PriorModel
from polyaudio.report import format_table

config = load_suite("suite.toml")
tasks = config.build_tasks()

result = Evaluator(seed=13).evaluate(
    PriorModel(seed=13),
    tasks,
    references=config.build_references(),
)
print(format_table(result))

Or from the command line:

# The suite names its own model (a universal `random` baseline here):
polyaudio run --suite examples/suites/mini.toml --report table

# Compare a universal baseline against the single-task specialists it competes
# with. `prior` only applies to the classification/tagging tasks, so the
# regression and ASR rows show `-` for it — which is the honest thing to show.
polyaudio run --suite examples/suites/mini.toml --model random --report json --output random.json
polyaudio run --suite examples/suites/mini.toml --model prior  --report json --output prior.json
polyaudio compare random.json prior.json

The built-in models split into universal baselines that run across every task kind (random, constant) and single-task baselines that are strong only on their own kind (majority, prior for classification/tagging, mean for regression). A real unified model is meant to slot in exactly where random does and be judged against both groups.

Core ideas

• One Model.predict(spec, samples) handles every task. The harness hands it a TaskSpec (output kind, label set) so it can branch — that's what makes a unified model directly comparable to per-task ones.
• Tasks are typed by output kind — multiclass, multilabel, sequence, regression — which determines the valid metrics.
• Metrics are registered by name and know which output kinds they apply to.
• Baselines are first-class: random, majority, prior, mean, constant. If your model can't beat prior, the table will say so.
• Reproducible by construction: seeds, a provenance block, and a declarative TOML suite you can commit next to your data.

Documentation

• docs/usage.md — writing suites, manifests and custom models
• docs/architecture.md — how the harness fits together
• docs/api-reference.md — the public surface
• docs/design-notes.md — why it's built this way

License

MIT — see LICENSE.