Model Validation Infrastructure, Script, and Workflow

[cwitkowitz]

Implements a model validation framework executed manually or as a daily GitHub Action. The main script headlessly drives the pyharp gradio endpoints (/controls, /process) across two tiers: local pyharp/examples/ apps, and every Hugging Face Space under teamup-tech.

Every model gets a default case with inputs synthesized from its own /controls spec; config.yml adds per-model and shared cases, declarative expect rules on outputs (audio/MIDI/label properties), and custom validators.

See model_validation/README.md for more details.

One current limitation is the lack of precision in tracking ZeroGPU quota usage. Since quota usage is dynamic and can be refunded, it might be worthwhile to retry ZeroGPU models before outright skipping the remaining ones after a single ZeroGPU quota exceeded error. Another workaround could be to set the number of workers to 1 to avoid eating up extra quota due to initial time reservations.

[2cylu2]

I ran the validation script locally against both local example servers (--local-examples) and remote Spaces:
--local-examples --load-only passes all 4 models.
Running full inference (--local-examples) fails for midi_synthesizer and ui_tester with:
AppError: The upstream Gradio app has raised an exception but has not enabled verbose error reporting. To enable, set show_error=True in launch().

It might be helpful to enable show_error=True either in pyharp's build_endpoint() or when launching the local example servers in harness.py. That way, validate_models.py can capture the actual traceback in report.json instead of the generic AppError, which should make debugging failures easier.

One thought on the ZeroGPU reservation issue mentioned in the PR description: instead of reducing --workers to 1 globally (which could slow down CPU model validation), we could run only the ZeroGPU models one at a time while keeping CPU models running in parallel. Since this script runs automatically, taking a little longer seems like a reasonable tradeoff if it helps avoid unnecessary quota usage.

One model (generalizable-accompaniment-generation) failed because it requires audio between 3 and 30 seconds long.
Would it make sense to increase the default synthesized audio duration to 3 or 5 seconds in assets.py, or should models with duration requirements always specify custom test files in config.yml under test_data/?

[ChughVansh]

Catching a \n error in results.py to help with report.md formatting.

Also, an edge case catch and suggestion in harness.py.

Rest everything looks nice! Left more in-depth comments in the review

[ChughVansh]

Also, another question, is there any specific reason teamup-tech isn't treated as the default in --exclude or --spaces or even config.yml? doing so would be a very short code change and wouldn't require us to write

--spaces teamup-tech/model1, teamup-tech/model2, teamup-tech/model3

and instead just have

--spaces model1, model2, model3

something like:
if opts.spaces:
spaces = [s if "/" in s else f"{opts.org}/{s}" for s in opts.spaces]
space_ids = [s for s in spaces if s not in excluded]

in line 203 @ validate_model.py. This would also allow us to input

--spaces model1, model2, model3, diff-org/model-name

[ChughVansh]

Looks great! Just one thing.

There are times when GPU exhaustion is read as a failure and not a skip; these were the two messages that triggered that:

1. The backend's ZeroGPU quota is exhausted for the identity making this call. Paste your own Hugging Face token in the token field (read scope) so usage is attributed to your account (OmniVoice, Melodyflow, etc.). Custom error handling from automatic model deployment agent.

2. You have exceeded your Pro ZeroGPU quota (180s requested vs. 111s left). Try again in 22:05:43. Add credits to keep using ZeroGPU. $1 for 10 minutes of ZeroGPU inference (DDSP Guitar). Regex matching for this format - 180s requested vs. 111s left - may be helpful bcs i've encountered this too

[2cylu2]

I also think this looks great! I got the same messages as @ChughVansh.

For the first one, this was intentional with the two-Space approach I used for those models. Without a token, requests from the frontend Space are treated as anonymous calls, so Hugging Face counts them against the anonymous ZeroGPU quota even though the backend is functioning correctly. I put our HF token as a secret in those Spaces, but if the token is exhausted, users can enter their own token in the space provided. I might need to redeploy some of the models to add the option to enter a token, though, since that was a later addition.

I made the agent show this error when a Space doesn't have a secret HF token or when the token is exhausted. I manually added my token to some of the models that showed this message, and they worked. So it seems like the deployment itself is working, but this could be inconvenient for model validation since these currently show up as failures. It would probably make sense to treat these cases as skips instead. The regex suggestion for the Pro ZeroGPU quota message also makes sense.

[cwitkowitz]

Ok, to keep the reviewing and changes manageable, I'm going to merge this PR and open issues for:

1. Secondary (downstream) ZeroGPU errors trigger our skipping mechanism
2. Customizing MIDI instrument number for synthesized files