Phase 6: power matrix and battery selection - #7
Open
jameshoweee wants to merge 5 commits into
Open
Conversation
Calibration layer (code/calibration.py): - Flaw-injection library: 16 parametrized broken samplers covering sigma shift (1-10%), mu shift, sign coupling (52-60%), tail truncation (3-4 sigma), table error (2-3x), contamination (1-5%), Markov coupling (rho 0.1-0.3) - Power matrix: detection rate for each (test, flaw) pair over configurable replicates. Run with: python calibration.py --reps 20 - p-value uniformity meta-test: KS test of chi-square p-values against U[0,1] under H0. Detects miscalibration of the suite itself. Passes (KS p=0.16, mean_p=0.45). - Default battery selection: tests that win on at least one flaw type. Result: chi2, tail_exceedance, sign_halfgauss, discrete_ad, higher_crit, ljung_box, runs_test (7/8 tests earn a spot; block_homog wins nowhere at n=5000 but may at higher n). Key power matrix findings (n=5000, reps=5 smoke): - tail_exceedance: sole detector of 3-sigma truncation - ljung_box + runs_test: sole detectors of Markov coupling - higher_crit: best at mu shift (catches at mu+0.1) - discrete_ad: tied-best on sign bias and table errors - chi2: strong on all diffuse alternatives Test suite: 127 passed, 0 failed
Owner
Author
|
consolidating into a single PR |
Bug 1 (blocking): sampler.c paired an ascending CDT with the RCDT
"v < row" count, sampling 18 - z0 (mass piled at |z| ~ 18). Flip the
count to "v >= row" so it matches the Python reference sampler0().
Add tests/test_sampler_c.py plus a CI step that compiles, runs, and
chi2-tests the output against the folded PDT -- the table lint checked
the table but never the comparison logic.
Bug 2 (serious): berexp accepted with prob ~255/256 when the quantized
acceptance probability floored to 0 (p = -1, whose byte is 0xff).
Guard with `if p < 0: return False`, matching certification.py's model.
Bug 3 (moderate): reported R_a - 1 as exp(ra*(a-1)) - 1, i.e.
sum p^a/q^(a-1) - 1 ~ (a-1)(R_a - 1) -- a factor a-1 too large vs the
[Pre17] convention the 2^-75 requirement is stated in. Use exp(ra) - 1.
Nits:
- max_offdiag_correlation: MC-calibrate (the log(dim) Gumbel centering
miscalibrated the max over dim(dim-1)/2 dependent pairs).
- fft_domain_battery: test the summed variance statistic against
chi2(2(n-1)) instead of the averaged form against chi2(n-1).
- mc_pvalue: (1 + #{null >= obs}) / (B + 1) so finite B stays valid.
- calibration: reproducible per-flaw seeds (drop PYTHONHASHSEED-salted
hash()); enumerate index instead.
- doornik_hansen: project onto the nonzero eigenvectors on the singular
path instead of dropping arbitrary data columns by eigenvalue index.
- _energy_distance: cdist-based, no subsampling.
Also: pyproject declared license = MIT, but the repo ships under the
PQShield non-commercial research licence (see LICENCE). Point at the
file instead of asserting a wrong SPDX id.
The battery verdict was an AND of 7 (uni) / 5 (multi) per-test passes,
each thresholded at alpha -- inflating the family false-alarm rate to
~1-(1-alpha)^k (about 0.7% at alpha=1e-3, k=7). Replace it with one
family verdict:
all_pass = (Fisher-combined global p > alpha) AND no test rejects at
the Bonferroni threshold alpha/k
Fisher controls the family-wise rate against diffuse alternatives; the
Bonferroni gate keeps a single strong localized flaw from being diluted.
Individual results and per-test p-values are still reported
(report-first); global_pvalue / fisher_stat / bonferroni_rejects are
exposed for downstream use.
The SAGA v2 suite runs on generated, seeded vectors, not the original HPRR20 reference sample files. Drop the ~530MB testdata/ directory from the working tree and gitignore it. The data is preserved verbatim in the pqcrypto2020-as-published tag; test_falcon() documents how to restore it (git checkout pqcrypto2020-as-published -- code/testdata). Note: removes it going forward only; the blobs remain in history. A full history purge is a separate, destructive step left for later.
- report_card.py: ANSI terminal "report card" that runs the full battery on a sample set and prints a per-section pass/fail summary with component p-values, the global verdict, and per-test "check:" hints plus a failure-fingerprint match (nearest calibration flaw family). Supports --flaw for demoing injected defects. - visualize_mockups.py: matplotlib figures (power-matrix heatmap, detection frontiers, p-value calibration, FFT spectrum, Manhattan, ACF, Renyi profile, QQ norms) for the paper. Illustrative layouts; swap in real run data before publication. - extensive_calibration.json: results of the 7-param x 65-flaw x 8-test x 100-rep power-matrix sweep (n=10000). Generated figures are gitignored (regenerate from the scripts).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
calibration.py— flaw-injection library with 16 parametrized broken samplers and the power matrix computation that measures each test's detection rate against each flaw type.Extensive calibration (65 flaw types x 7 param points x 100 reps) running separately — results to follow.