Validation matlab - #18
Draft
davidrpenas wants to merge 26 commits into
Draft
Conversation
Self-contained validation/ tree: layer-1 golden vectors (79 checks) plus the MATLAB-vs-C baseline for lv2, alpha-pinene (partobs), SIR and NF-kB. Deterministic layers 3+4 (fixed-theta ODE/sensitivities, UQ replayed on the injected MATLAB LOO ensemble) pass 71/71 checks; conformal bands are machine-exact. Includes the seeded MATLAB reference exports (R2024a), seed-1 C results, side-by-side figures, and the layer-5 tooling (run_c_seeds.sh, compare_baseline.py, drago sbatch). Touches nothing outside validation/: the AP-partobs example files and a full-precision-sigma NFKB config live here until promoted to example-files. To build the harness add 'add_subdirectory(validation)' to the top-level CMakeLists (see validation/baseline/README.md). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rewritten for clarity: the layered-validation rationale, up-to-date results (79/79 golden checks, 71/71 baseline checks across the 4 preprint models), what golden/ and baseline/ actually store, the one-line CMake integration step, where the per-model figures live, and how to read the tolerances. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…figures Layers renamed everywhere (dirs, tol keys, harness labels, docs): 1 algebra kernels (golden/), 2 integration & sensitivities, 3 UQ replay on the injected MATLAB ensemble, 4 statistical end-to-end. The validation README table now details what each layer actually does and what a failure means. New: plot_matlab_vs_c.m renders ONE figure per model with the MATLAB bands (left) and the C bands (right), both drawn by the same MATLAB code so the styling is identical by construction (compare_<model>_matlab_vs_c.png, all 4 models included). run_c_seeds.sh now supports ap and sir; nfkb uses the full-sigma config and the MATLAB-matched 2e4 budget. All 5 ctests re-verified green after the rename. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… search Implements the shared-randomness idea: all randomness stays on the MATLAB side. gen_cost_replay.m runs one seeded MEIGO fit with the cost handle wrapped in a recorder and freezes every evaluated theta with its MATLAB cost J; test_cost_replay.c re-evaluates the C cost (CVODES + cuqdyn_residual_weight) on the same sequence and diffs J point by point - fully deterministic, and it covers exactly the region a real search visits. This validates the cost function inside the optimisation loop, the one piece layers 2-3 could not see. Result (lv2, 20126 recorded evaluations): 0 beyond rel_tol 1e-3, mean rel 4.2e-5, max 5.2e-4, no CVODES failures. New ctest: cost_replay_lv2 (SKIPs without the recording). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… report C runs with SACESS_SEED=1..10 under validation/baseline/c/<model>/. AP MATLAB seeds are complete too, so its distribution report is included: every parameter ratio C/MATLAB = 1.000 - on alpha-pinene both optimisers converge to the same optimum on every seed. lv2/sir MATLAB seeds still running; their reports follow. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The plain-text exports are the reference; the .mat workspaces are regenerable byproducts and stay untracked. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Under sbatch, BASH_SOURCE points at the slurmd spool copy, so the cd never reached the repo and MATLAB could not find gen_baseline. Found on the first real run on drago.csic.es. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
SIR, 10 seeds per side: every C/MATLAB parameter ratio is 1.000. The report's agreement criterion now also accepts medians within 0.1% when both IQRs are degenerate - on strongly identified problems both optimisers collapse to the same point and strict interval intersection flagged 4-digit agreement as disagreement. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
10 seeds per side: parameter ratios 0.999-1.001; p2 differs by 0.13%, just over the 0.1% agreement threshold, with both sides fully collapsed - a budget-level offset, not a systematic bias. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
10 seeds per side. Observed states: conformal band widths agree to 0.81-1.27x. Hidden states: delta bands vary 1.2-6.1x, the documented signature of the near-singular FIM (weak directions differ run to run on either side). The operative metric holds everywhere: empirical coverage of the true trajectory is 1.000 on all 15 states on BOTH sides (nominal 0.90). Parameters: 14/29 medians differ beyond 0.1%, consistent with non-identifiability, not implementation bias. Seeds 11-20 run on drago.csic.es and will extend this report when the host is reachable again. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The SLURM array rode out a ~26h login-node outage and completed all 10 tasks. With 20 MATLAB seeds vs 10 C seeds the distributions tighten as expected for a weakly identifiable problem: disagreeing parameter medians drop from 14/29 to 11/29, hidden-state band-width ratios move toward 1 (y10 1.24->1.02, y8 1.55->1.33, y14 4.3->2.9, y6 6.1->5.0), and empirical coverage of the true trajectory stays 1.000 on all 15 states on both sides (nominal 0.90). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New plot_c_hybrid_uq.m parses cuqdyn-results.txt and draws it with the exact plot_hybrid_uq layout (min(3,nstates)-column grid, default colour cycle per state, legends, data markers, 0-clamped lower bands), so the MATLAB and C figures of every model are comparable panel by panel with no cosmetic differences. Regenerates the four c_*_seed1 figures - the previous Python rendering laid NF-kB out as a single 15-panel row with a two-colour scheme. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The last validation stage: MATLAB's MEIGO runs twice with the same rng seed, once evaluating the MATLAB cost (ode object, cvodesstiff, same tolerances as the C config) and once evaluating the REAL C cost served over TCP by cost_server (solve_ode + cuqdyn_residual_weight, the exact CLI code path, returning J plus the residual vector so lsqnonlin works too). All randomness is generated once, on the MATLAB side. Result (lv2, seed 20260904, maxeval 1e4): costs agree to median 1.1e-10 on the common prefix; the two searches stay in lock-step for 123 evaluations, then one rounding-level difference flips a discrete eSS decision and the paths separate - and still both land on the same optimum: final thetas within 6e-8..3e-7 relative, J within 5e-9. Matching the integrator family tightens the cost agreement by five orders of magnitude over the ode15s-based cost replay (1e-10 vs 4e-5). hybrid_report.txt carries the numbers; the per-evaluation logs are regenerable and stay untracked.
The MPI implementation of the algorithm, validated where it belongs - as one more noise-free stage of this suite. sacess reseeds its RNG on every solver invocation, so with a fixed SACESS_SEED each LOO refit is deterministic no matter which rank runs it, and the MPI build's result must not depend on the process count. validation/mpi/test-mpi.sh runs the partially observed lv2 case at np=1/2/5 requiring byte-identical results, checks that np=4 aborts cleanly (4 does not divide the 30 LOO points) and wraps every run in a timeout so a deadlock fails instead of hanging. Runs use the serial sacess config: under -DMPI each rank runs its own serial eSS over its shard. Verified on this branch's tree (WSL, gcc 11.4, OpenMPI): 4/4 checks, np=2 and np=5 byte-identical to np=1 - first green evidence for the 'MPI untested with partial observability' TODO line. The README table gains the corresponding row. The ci.yml wiring for the new main layout lives separately on feat/mpi-ci-validation.
hybrid_meigo_cvodes.m is now parameterised by model (lv2/ap/sir/nfkb, same problem definitions as gen_baseline) and one cost_server per model serves the C cost. Same seed and budget everywhere (20260904, 1e4). Cost agreement on the lock-step prefix: 1.1e-10 (lv2), 1.5e-9 (ap), 1.0e-6 (sir), 3.7e-6 (nfkb); lock-step lengths 123/148/74/656 evaluations. After the rounding-triggered divergence, lv2/ap/sir land on the same optimum (final thetas within 1e-7..3e-4); on nfkb the two paths end in different basins of the non-identifiable landscape (J 914 vs 518, the C-cost run finding the better one) - a measurement of the problem's multimodality, not of the code: on the common prefix the two costs agree to 3.7e-6. The per-model report replaces the single-model hybrid_report.txt.
solve_ode takes a fifth argument now, and sensitivity.h is gone with its declarations folded into ode_solver.h. Paths follow example-files/<model>/ and build/<type>-<variant>. write_expected_output.m is dropped: its only consumer was a comparison that is not being proposed.
compare_baseline.py stamped the absolute directory it was given, so the committed reports carried the path of the laptop that made them. The layer number they still called 5 is fixed, and the raw hybrid traces are gitignored instead of showing up as untracked.
Nothing reads them; everything the comparison consumes is in cuqdyn-results.txt. They were 39.9 MB of the 43 the C results weighed.
It called baseline/ the MATLAB reference and then listed only matlab/, so the C runs, the per-seed campaign, the cost replay, the hybrid experiment and the reports were all invisible.
baseline/ is the MATLAB reference and the tools that read it; the seeded C runs are neither. They move to c/layer4/<model>/, with the stage in the path.
With the C runs already out, baseline/ still held the very reference it is supposed to read. It now holds only the harness.
golden/ sat loose at the top, baseline/ had become a catch-all, and two stages appeared in no layer table. validation/ is now layer1..layer5 plus common/ for what belongs to no single layer; the cost replay, called 2.5 in the code and nothing in the docs, becomes layer 3. test_baseline takes the validation directory and a model name and composes its own paths. Verified on FT3: ctest 6/6, same numbers as before the move.
…cumented ones The renumbering left names behind: gen_baseline.m kept l3 and l4 for what are now layers 4 and 5, test_baseline.c had a layer 3 banner over run_layer4, and both cost-replay files still called themselves layer 2.5. The ctests were named after a baseline/ directory that no longer exists. Variables, banners and headers now carry the number of the layer they belong to, and the tests are layer1_golden, layer2_4_<model> and layer3_lv2. Verified on FT3: ctest 6/6.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.