Skip to content

Test samplerz()'s output distribution in CI - #9

Merged
tprest merged 1 commit into
masterfrom
saga-test-samplerz-distribution
Jul 10, 2026
Merged

Test samplerz()'s output distribution in CI#9
tprest merged 1 commit into
masterfrom
saga-test-samplerz-distribution

Conversation

@tprest

@tprest tprest commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

The gap

sampler.c gets its output chi²-tested against the folded PDT (test_sampler_c.py) — but that covers only the base sampler: half-Gaussian plus sign bit, no BerExp.

samplerz() — the full sampler proposed in [HPRR20], with the BerExp rejection step, and the artifact's headline deliverable — was covered only by test_sampler_determinism.py, which checks reproducibility and never the output distribution.

So the paper's own sampler had no distribution test at all, while its C base-sampler cousin did.

The test

Five (mu, sigma) points chosen for the boundaries that matter:

(μ, σ) why
(0, 1.55) typical
(0.5, 1.55) worst-case fractional center for the sign/BerExp recombination
(0, 1.2778) σ_min — discreteness strongest here
(0, 1.80) near σ₀ = 1.8205
(3.7, 1.55) nonzero center

Plus a negative control: the battery must reject samples drawn at the wrong σ, so test_samplerz_passes_battery cannot be vacuous.

Seeded, hence deterministic. Verified bit-identical samples and p-values on Python 3.10 and 3.12 (the CI matrix):

py3.10  first 12 = [-3, 0, -3, 0, 0, 0, 3, 1, -1, -1, 1, -2]  chi2 p=0.876550  Fisher p=0.656100
py3.12  first 12 = [-3, 0, -3, 0, 0, 0, 3, 1, -1, -1, 1, -2]  chi2 p=0.876550  Fisher p=0.656100

Observed Fisher p-values run 0.11 … 0.82 against α = 0.001 — two orders of magnitude of headroom. Cost: ~3.3 s. No workflow change needed; pytest code/tests/ already discovers it.

Scope, stated honestly

This guards the class "the output distribution is wrong" — the same class test_sampler_c.py guards for the C sampler. Measured against deliberately broken variants:

Injected bug Caught?
corrupted table entry (+2%) ✅ p = 0.0000
broken sign recombination ✅ p = 0.0000
ccs scaling (sf = σ/σ₀) no — a constant factor in z, so the output distribution is unchanged; only the rejection rate moves
berexp p = -1 edge case no — mass ~2⁻⁷², unobservable at any feasible N

Note that means it would not have caught either of the two samplerz bugs fixed earlier. It is a regression guard on the distribution, not a catch-all. (What would catch the ccs bug is a check that the mean iteration count is constant in σ — deliberately out of scope, as isochrony testing was descoped.)

Suite: 153 passed, 1 skipped.

🤖 Generated with Claude Code

sampler.c gets its output chi2-tested against the folded PDT
(test_sampler_c.py), but that covers only the *base* sampler -- half-Gaussian
plus sign bit, no BerExp. samplerz(), the full sampler proposed in [HPRR20] and
the artifact's headline deliverable, was covered only by
test_sampler_determinism.py, which checks reproducibility and never the output
distribution. So the paper's own sampler had no distribution test at all.

Add one, over five (mu, sigma) points chosen for the boundaries that matter:
centered, the worst-case fractional center for the sign/BerExp recombination,
both ends of the permitted sigma range, and a large center. Plus a negative
control asserting the battery rejects samples drawn at the wrong sigma, so the
test cannot be vacuous.

Seeded, hence deterministic: verified bit-identical samples and p-values on
Python 3.10 and 3.12, the CI matrix. Observed Fisher p-values run 0.11..0.82
against an alpha of 0.001, so there is ample margin. Cost: ~3.3 s.

Scope, honestly: this guards the class "the output distribution is wrong"
(table corruption, sign/recombination bugs, sigma/mu scale errors), the same
class test_sampler_c.py guards for the C sampler. Measured: it catches a
corrupted table entry and a broken sign recombination, and it does NOT catch
the two samplerz bugs fixed earlier -- the ccs scaling (a constant factor in z,
so the output distribution is unchanged; only the rejection rate moves) nor the
berexp p=-1 edge case (mass ~2^-72). It is a regression guard, not a catch-all.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@tprest
tprest merged commit 09fedd1 into master Jul 10, 2026
4 checks passed
@tprest
tprest deleted the saga-test-samplerz-distribution branch July 10, 2026 08:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant