Record the 129x129 Ghia cavity results and report steps run per backend

CHANGELOG.md

@@ -9,6 +9,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **129×129 lid-driven cavity validation recorded** — the AVX2, OpenMP and CUDA projection
+  backends match Ghia et al. (1982) at 129×129 with RMS_u / RMS_v of 0.0017 / 0.0024
+  (Re=100), 0.0096 / 0.0328 (Re=400) and 0.0299 / 0.0300 (Re=1000), identical across
+  backends to four decimals and across three EC2 workflow runs. Each backend line of
+  `test_cavity_backends` now also prints the steps actually run, whether the kinetic-energy
+  residual stopped the run early, and that residual
+  (`docs/validation/cavity-backends-validation.md`, `tests/validation/test_cavity_backends.c`).
 - **Multigrid pressure solve on the OpenMP projection solver** — `projection_omp`
   now accepts `NS_PRESSURE_SOLVER_MULTIGRID` (OpenMP multigrid V-cycles, RHS
   interior mean subtracted for Neumann compatibility) and
@@ -108,6 +115,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- `scripts/ec2-validate.sh` runs every `CavityBackend_*` ctest entry. It ran the test binary
+  without a filter, which skips the Re=400 and Re=1000 cases, and under `set -e` a failing run
+  ended the script before its FAILED summary.
 - `poisson_solve_3d()` now checks `poisson_solver_init()`'s return status: a failed init
   (e.g. multigrid on non-2^k+1 dims) no longer leaves a broken solver in the convenience
   cache; the call returns -1 and later valid calls re-create the solver.

ROADMAP.md

@@ -455,10 +455,15 @@ headers. See CHANGELOG.
 Lid-driven cavity (33×33, Re=100, all backends), Taylor-Green vortex, and Poiseuille flow are
 all validated. See CHANGELOG and `docs/validation/`.
 
-**Still needed — 129×129 release validation** (too slow for CI; uses `CAVITY_FULL_VALIDATION=1`):
+**129×129 release validation** (`-DCAVITY_FULL_VALIDATION=ON`; runs in the EC2 GPU workflow on
+every push to master, about 50 minutes):
 
-- [ ] Full Ghia validation at 129×129 for Re=100, 400, 1000 (record RMS on EC2)
+- [x] Full Ghia validation at 129×129 for Re=100, 400, 1000 — AVX2/OMP/GPU projection RMS
+  ≤ 0.033, recorded in `docs/validation/cavity-backends-validation.md`
 - [ ] Extended cavity convergence to true steady-state (residual < 1e-8)
+- [ ] Explicit Euler cavity cases stop at 11,300–11,800 steps (t ≈ 1.1–1.2) through the harness's
+  kinetic-energy exit, on the 33×33 CI grid (of 25,000) and at 129×129 (of 250,000) alike; make
+  them run to a developed flow or drop them from Ghia validation
 - [ ] Multi-Reynolds grid-convergence study (Richardson extrapolation)
 - [ ] Extended-time Taylor-Green decay-rate verification
 - [ ] Cross-architecture consistency (all backends identical within 0.1%)
@@ -473,7 +478,7 @@ all validated. See CHANGELOG and `docs/validation/`.
 
 | Test | CI Mode | Release Mode |
 |------|---------|--------------|
-| Cavity Ghia Validation | 33×33, 5000 steps | 129×129, 50000 steps |
+| Cavity Ghia Validation | 33×33, 5000 steps | 129×129; 50000 steps (Re=100), 60000 (Re=400), 100000 (Re=1000) |
 | Cavity Re=400 Stability | 25×25, 500 steps | 65×65, 20000 steps |
 | Grid Convergence | 17→25→33 | 33→65→129 |
 | Taylor-Green Vortex | 32×32, 200 steps | 128×128, 10000 steps |

docs/reference/solvers.md

@@ -641,11 +641,13 @@ Classic benchmark problem validated against Ghia et al. (1982):
 - All other walls no-slip
 - Reynolds number Re = ρUL/μ
 
-**Results (Re=100, 129×129 grid):**
-- Centerline velocity profiles match published data
-- RMS error < 0.01
+**Results (129×129 grid, AVX2/OpenMP/CUDA projection):**
+- Centerline velocity profiles match published data at Re = 100, 400 and 1000
+- RMS error vs Ghia (u / v): 0.0017 / 0.0024 at Re=100, 0.0096 / 0.0328 at Re=400,
+  0.0299 / 0.0300 at Re=1000
 
-See [validation/lid-driven-cavity.md](validation/lid-driven-cavity.md) for details.
+See [lid-driven-cavity.md](../validation/lid-driven-cavity.md) and
+[cavity-backends-validation.md](../validation/cavity-backends-validation.md) for details.
 
 ### Taylor-Green Vortex
 

docs/validation/cavity-backends-validation.md

@@ -16,7 +16,7 @@ This document describes the comprehensive backend validation system for the lid-
 - **CPU Scalar** (`projection`)
 - **AVX2/SIMD** (`projection_optimized`)
 - **OpenMP** (`projection_omp`)
-- **CUDA GPU** (`projection_jacobi_gpu`)
+- **CUDA GPU** (`projection_gpu`)
 
 ### Explicit Euler Backends
 - **CPU** (`explicit_euler`)
@@ -82,18 +82,23 @@ dt: 0.0005
 ```
 
 ### Full Validation Mode
-```c
-#define CAVITY_FULL_VALIDATION 1
 
+Enabled with the CMake option `-DCAVITY_FULL_VALIDATION=ON`:
+
+```text
 Re=100:   Grid 129×129, 50000 steps  (250000 for Explicit Euler), dt 0.0002
 Re=400:   Grid 129×129, 60000 steps  (projection only),           dt 0.0005
 Re=1000:  Grid 129×129, 100000 steps (projection only),           dt 0.0005
 ```
 
+Step counts are budgets: the harness stops a run early once the relative change in kinetic
+energy per step drops below 1e-8 (after step 100), and each backend prints a `Steps run:` line
+with the steps it actually ran.
+
 Higher-Reynolds cases run the **projection backends only** (the production solver)
 and require the finer 129×129 grid to resolve the stronger primary vortex and the
-secondary corner vortices. The Re=400/1000 `dt`/step counts are starting points —
-refine them after recording the first full-validation RMS values (see below).
+secondary corner vortices. Every projection case passes with at least a 3× margin on the
+RMS target (see Full Validation Results below), so the step budgets are unchanged.
 
 To run full validation:
 ```bash
@@ -136,8 +141,8 @@ All available backends produce consistent results (within 0.1%)
 
 ## Key Findings
 
-1. **Projection method meets scientific target** (RMS < 0.10) ✅
-2. **Explicit Euler meets relaxed target** (RMS < 0.15) ✅
+1. **Projection method meets scientific target** (RMS < 0.10) ✅ — at 33×33, and at 129×129 for Re=100, 400 and 1000
+2. **Explicit Euler meets relaxed target** (RMS < 0.15) ✅ — but its 129×129 runs stop early (see Known Issue below)
 3. **CPU and OpenMP backends are consistent** ✅
 4. **Test correctly fails when RMS >= target** ✅
 
@@ -176,33 +181,68 @@ To exclude validation tests (e.g., sanitizer builds):
 ctest -LE validation
 ```
 
+Full-validation builds (`-DCAVITY_FULL_VALIDATION=ON`) replace `CavityBackendsTest` with one
+ctest entry per backend and Reynolds number (`CavityBackend_*`, registered in `CMakeLists.txt`).
+The Cross-Architecture Validation (EC2) workflow, `.github/workflows/gpu-validation.yml`, builds
+that configuration on every push to master and on pull requests labelled `gpu-test`. It runs the
+`validation` label on a g4dn.4xlarge with an NVIDIA T4 and writes a per-(Re, backend) RMS table
+to the job summary. `scripts/ec2-validate.sh` runs the same `CavityBackend_*` entries on a
+manually started instance.
+
 ## Full Validation Results (129×129)
 
-Record the RMS values from the `-DCAVITY_FULL_VALIDATION=ON` run here. All entries
-must satisfy RMS < 0.10 (projection). Fill in once the EC2/GPU run completes.
-
-| Re   | Backend       | RMS_u  | RMS_v  | Pass? |
-|------|---------------|--------|--------|-------|
-| 100  | AVX2/SIMD     | _TBD_  | _TBD_  | _TBD_ |
-| 100  | OpenMP        | _TBD_  | _TBD_  | _TBD_ |
-| 100  | CUDA GPU      | _TBD_  | _TBD_  | _TBD_ |
-| 400  | AVX2/SIMD     | _TBD_  | _TBD_  | _TBD_ |
-| 400  | OpenMP        | _TBD_  | _TBD_  | _TBD_ |
-| 400  | CUDA GPU      | _TBD_  | _TBD_  | _TBD_ |
-| 1000 | AVX2/SIMD     | _TBD_  | _TBD_  | _TBD_ |
-| 1000 | OpenMP        | _TBD_  | _TBD_  | _TBD_ |
-| 1000 | CUDA GPU      | _TBD_  | _TBD_  | _TBD_ |
-
-> If Re=1000 does not converge to RMS < 0.10 within the step budget, increase
-> `VALIDATION_STEPS_RE1000` (more physical time) before relaxing the tolerance —
-> the target is a scientific standard, not a tunable.
+All nine projection cases pass the RMS < 0.10 target with at least a 3× margin, and the AVX2,
+OpenMP and CUDA backends give the same RMS to four decimals at every Reynolds number.
+
+| Re   | Backend   | RMS_u  | RMS_v  | Pass? | Step budget × dt (t) | Wall time |
+|------|-----------|--------|--------|-------|----------------------|-----------|
+| 100  | AVX2/SIMD | 0.0017 | 0.0024 | Yes   | 50000 × 0.0002 (10)  | 293 s     |
+| 100  | OpenMP    | 0.0017 | 0.0024 | Yes   | 50000 × 0.0002 (10)  | 428 s     |
+| 100  | CUDA GPU  | 0.0017 | 0.0024 | Yes   | 50000 × 0.0002 (10)  | 516 s     |
+| 400  | AVX2/SIMD | 0.0096 | 0.0328 | Yes   | 60000 × 0.0005 (30)  | 336 s     |
+| 400  | OpenMP    | 0.0096 | 0.0328 | Yes   | 60000 × 0.0005 (30)  | 487 s     |
+| 400  | CUDA GPU  | 0.0096 | 0.0328 | Yes   | 60000 × 0.0005 (30)  | 481 s     |
+| 1000 | AVX2/SIMD | 0.0299 | 0.0300 | Yes   | 100000 × 0.0005 (50) | 516 s     |
+| 1000 | OpenMP    | 0.0299 | 0.0300 | Yes   | 100000 × 0.0005 (50) | 801 s     |
+| 1000 | CUDA GPU  | 0.0299 | 0.0300 | Yes   | 100000 × 0.0005 (50) | 805 s     |
+
+**Source:** GitHub Actions run 34853615452 of the Cross-Architecture Validation (EC2) workflow,
+commit `6e1b104` (2026-09-14). Runs 34815324268 (`23b196a`) and 30116961115 (2026-07-24, #201)
+produced the same RMS values. Configuration: g4dn.4xlarge (16 vCPUs, NVIDIA T4), gcc Release
+with AVX2 and CUDA enabled, `ctest -L validation -j 4` with `OMP_NUM_THREADS=4`. Wall times come
+from that shared run, with several tests running at once, so they are not performance
+benchmarks. These runs predate the `Steps run:` line, so the table lists step budgets; the next
+workflow run reports the steps each case ran.
+
+> If a later change pushes a case above RMS 0.10, increase its step budget (more physical
+> time) before relaxing the tolerance — the target is a scientific standard, not a tunable.
+
+### Known Issue: Explicit Euler at 129×129
+
+The Explicit Euler cases are not evidence of 129×129 accuracy. Both solvers cap dt at 1e-4, and
+the harness's kinetic-energy exit (relative change per step below 1e-8) ends the runs long before
+their 250,000-step budget:
+
+| Backend   | Steps run | Simulated time | KE residual | RMS_u  | RMS_v  |
+|-----------|-----------|----------------|-------------|--------|--------|
+| AVX2/SIMD | 11,323    | ≈ 1.1          | 8.3e-9      | 0.0957 | 0.1293 |
+| OpenMP    | 11,775    | ≈ 1.2          | 9.8e-9      | 0.0957 | 0.1277 |
+
+Measured on a local Windows (MSVC Release) build with 4 OpenMP threads; the RMS values match the
+EC2 runs. The 33×33 CI case stops the same way: the scalar Euler solver ends at 11,300 of 25,000
+steps with RMS_u 0.0957 and RMS_v 0.1284. The flow needs about 10–20 time units to reach steady
+state at Re=100 (see [lid-driven-cavity.md](lid-driven-cavity.md)), so these runs pass the 0.15
+target without a developed solution on either grid. The test logic is unchanged; the follow-up is
+tracked in ROADMAP §6.1.
 
 ## Next Steps
 
 ### Immediate (for v1.0)
-1. Run full 129×129 validation (release mode) and fill in the results table above
-2. Tune Re=400/1000 `dt`/step counts if convergence is incomplete
-3. Tick the ROADMAP 6.1.1 / 6.1.5 boxes once recorded
+1. ROADMAP §6.1 cross-architecture consistency stays open: `CavityBackend_Consistency` compares
+   CPU, AVX2 and OpenMP center values at 33×33 without the GPU, and matching RMS values at
+   129×129 are not a field-level 0.1% comparison.
+2. Make the Explicit Euler 129×129 cases run to a developed flow, or drop them from full
+   validation (see Known Issue above).
 
 ### Future Enhancements
 1. Add grid convergence study (33→65→129→257)

docs/validation/lid-driven-cavity.md

@@ -132,35 +132,45 @@ Computed by:
 Two modes are available:
 
 **Fast Mode (CI):**
-- Iterations: 1500-4000 steps
+- Steps: 5000 for the 33×33 Ghia comparisons (25000 for the Explicit Euler backend cases)
 - Time step: 0.0005
 - Purpose: Quick regression testing
 
 **Full Validation Mode:**
-- Iterations: 3000-10000 steps
-- Time step: 0.0005
-- Purpose: Comprehensive validation
+- Grid: 129×129 at Re = 100, 400 and 1000
+- Steps: 50000 at Re=100 (dt 0.0002), 60000 at Re=400 and 100000 at Re=1000 (dt 0.0005)
+- Purpose: Release validation; runs in the EC2 GPU workflow on every push to master
 
-Enable full mode with:
-```c
-#define CAVITY_FULL_VALIDATION 1
+Enable full mode with the CMake option:
+```bash
+cmake -B build -DCAVITY_FULL_VALIDATION=ON
 ```
 
+See [cavity-backends-validation.md](cavity-backends-validation.md) for the per-backend
+parameters and ctest entries.
+
 ## Current Solver Performance
 
-### Status: ACCEPTABLE (Engineering Quality)
+### 129×129 Release Validation
+
+The AVX2, OpenMP and CUDA projection backends match Ghia et al. at 129×129, all under the
+0.10 target (RMS_u / RMS_v): Re=100 0.0017 / 0.0024, Re=400 0.0096 / 0.0328, Re=1000
+0.0299 / 0.0300. The full table and run details are in
+[cavity-backends-validation.md](cavity-backends-validation.md).
+
+### 33×33 CI Validation
 
-The current projection solver with CG Poisson achieves:
-- **u-centerline RMS: ~0.10** (target: < 0.10 ✅)
-- **v-centerline RMS: ~0.08** (target: < 0.10 ✅)
-- **Convergence:** Reaches steady state within tolerance
-- **Poisson Solver:** Conjugate Gradient (CG) with tolerance 1e-6
+`CavityBackendsTest` runs each projection backend at 33×33 for 5000 steps with dt = 0.0005
+(t = 2.5), each with a CG pressure solve on its own backend. The CPU scalar backend gives:
+- **u-centerline RMS: 0.0382** (target: < 0.10 ✅)
+- **v-centerline RMS: 0.0440** (target: < 0.10 ✅)
+- **Not a steady state:** the run uses its whole 5000-step budget and ends with a kinetic-energy
+  change of about 2.7e-5 per step, far above the 1e-8 stop threshold. t = 2.5 is also well short
+  of the 10–20 time units the flow needs (see Insufficient Time Stepping below), so this is a
+  quick regression check, not a converged comparison.
 
-**Configuration:**
-- Grid: 33×33 (current CI tests)
-- Time steps: ~3000-4000 iterations
-- dt = 0.0005
-- Poisson: CG method (`POISSON_METHOD_CG`) with scalar or SIMD backend (`POISSON_BACKEND_SCALAR` / `POISSON_BACKEND_SIMD`)
+See [cavity-backends-validation.md](cavity-backends-validation.md) for the other backends' CI
+results.
 
 ### Path to Excellence (RMS < 0.05)
 
@@ -197,10 +207,13 @@ The lid-driven cavity flow at Re = 100 requires ~10-20 time units to reach stead
 - Even 1% pressure error → 5-10% velocity error after many time steps
 - Insufficient Poisson convergence → spurious divergence → incorrect vortex structure
 
-**Current solution:** All backends now use robust solvers:
-- CPU: Conjugate Gradient (CG) - reliable convergence in ~150 iterations
-- SIMD/OMP: Red-Black SOR or CG-SIMD
-- GPU: Jacobi with sufficient iterations for parallelism
+**Current solution:** Every projection backend solves the pressure equation with Conjugate
+Gradient (CG) on its own backend (the scalar and OpenMP solvers can switch to multigrid through
+`ns_solver_params_t.pressure_solver`):
+- CPU: scalar CG - reliable convergence in ~150 iterations
+- SIMD: CG with AVX2/NEON kernels
+- OMP: OpenMP CG
+- GPU: device-resident CUDA CG
 
 #### 3. Time Step Selection
 

scripts/ec2-validate.sh

@@ -1,20 +1,23 @@
 #!/usr/bin/env bash
-# ec2-validate.sh — Build and run 129x129 cavity validation on EC2
+# ec2-validate.sh — Build and run the 129x129 cavity validation on EC2
+#
+# Runs every CavityBackend_* ctest entry (Re=100, 400 and 1000), the same cases the
+# Cross-Architecture Validation (EC2) workflow runs on each push to master.
 #
 # Quick-start (on a fresh g4dn.4xlarge with NVIDIA Deep Learning AMI):
 #
 #   # First time:
 #   git clone <your-repo-url> cfd && cd cfd
 #   ./scripts/ec2-validate.sh --setup
-#   ./scripts/ec2-validate.sh --build --run
+#   ./scripts/ec2-validate.sh --build --release --run
 #
 #   # Subsequent runs (after code changes):
 #   git pull  # or: rsync from local machine
-#   ./scripts/ec2-validate.sh --build --run
-#
-#   # Release mode for benchmarking:
 #   ./scripts/ec2-validate.sh --build --release --run
 #
+#   # Debug build (much slower; for stepping through with gdb):
+#   ./scripts/ec2-validate.sh --build --run
+#
 # VSCode Remote-SSH debugging:
 #   1. Install "Remote - SSH" extension in VSCode
 #   2. Ctrl+Shift+P > "Remote-SSH: Connect to Host..." > ubuntu@<ip>
@@ -56,10 +59,10 @@ while [[ $# -gt 0 ]]; do
             echo ""
             echo "  --setup    Install build dependencies (first time only)"
             echo "  --build    Build the project (Debug mode by default)"
-            echo "  --release  Use Release mode instead of Debug"
-            echo "  --run      Run the 129x129 cavity validation test"
+            echo "  --release  Use Release mode instead of Debug (matches CI)"
+            echo "  --run      Run all 129x129 cavity validation cases (Re=100/400/1000)"
             echo "  --all      Do setup + build + run (default if no flags given)"
-            echo "  --threads  Set OMP_NUM_THREADS (default: auto-detect)"
+            echo "  --threads  Set OMP_NUM_THREADS per test (default: nproc / 4)"
             exit 0
             ;;
         *) echo "Unknown option: $1"; exit 1 ;;
@@ -158,30 +161,40 @@ fi
 # ---------- Run ----------
 if [[ "${DO_RUN}" == "true" ]]; then
     echo ""
-    echo "=== Running 129x129 Cavity Validation ==="
-
-    if [[ -n "${OMP_THREADS}" ]]; then
-        export OMP_NUM_THREADS="${OMP_THREADS}"
-        echo "OMP_NUM_THREADS=${OMP_NUM_THREADS}"
-    else
-        echo "OMP_NUM_THREADS=auto ($(nproc) cores available)"
+    echo "=== Running 129x129 Cavity Validation (Re=100/400/1000) ==="
+
+    # Same scheduling as CI: 4 ctest slots (each CavityBackend_ entry reserves 2)
+    # and nproc/4 OpenMP threads per test
+    CTEST_JOBS=4
+    if [[ -z "${OMP_THREADS}" ]]; then
+        OMP_THREADS=$(( $(nproc) / CTEST_JOBS ))
+        OMP_THREADS=$(( OMP_THREADS > 0 ? OMP_THREADS : 1 ))
     fi
+    export OMP_NUM_THREADS="${OMP_THREADS}"
+    echo "ctest -j ${CTEST_JOBS}, OMP_NUM_THREADS=${OMP_NUM_THREADS} ($(nproc) cores available)"
 
     BINARY="${BUILD_DIR}/test_cavity_backends"
     if [[ ! -f "${BINARY}" ]]; then
         echo "ERROR: ${BINARY} not found. Run with --build first."
         exit 1
     fi
 
+    LOG="${BUILD_DIR}/validation_output.txt"
     echo ""
     START_TIME=$(date +%s)
-    "${BINARY}"
-    EXIT_CODE=$?
+    # Capture the status instead of letting set -e end the script before the summary
+    EXIT_CODE=0
+    ctest --test-dir "${BUILD_DIR}" -R '^CavityBackend_' --no-tests=error \
+        --output-on-failure -V -j "${CTEST_JOBS}" 2>&1 | tee "${LOG}" || EXIT_CODE=$?
     END_TIME=$(date +%s)
     ELAPSED=$((END_TIME - START_TIME))
     MINUTES=$((ELAPSED / 60))
     SECONDS=$((ELAPSED % 60))
 
+    echo ""
+    echo "=== Results (full log: ${LOG}) ==="
+    grep -E 'Testing backend|Steps run:|RMS_u:|SKIPPED|\[FAILED\]' "${LOG}" || true
+
     echo ""
     echo "============================================"
     if [[ ${EXIT_CODE} -eq 0 ]]; then

tests/validation/test_cavity_backends.c

@@ -229,6 +229,11 @@ static void test_backend_validation(const char* solver_type,
 
     TEST_ASSERT_NOT_NULL(ctx);
 
+    /* Steps actually run: the harness stops early once the KE residual drops below 1e-8 */
+    printf("      Steps run: %d/%d  stopped by KE threshold: %s  KE residual: %.2e\n",
+           result.steps_completed, max_steps, result.converged ? "yes" : "no",
+           result.final_residual);
+
     /* Extract centerline profiles */
     profile_data_t profiles = extract_profiles_from_ctx(ctx);
     TEST_ASSERT_NOT_NULL_MESSAGE(profiles.y_coords, "Failed to extract profiles");